RailswayCon 2010 - Command Your Domain


Published on

Published in: Technology

RailswayCon 2010 - Command Your Domain

  1. <ul>Command your Domain </ul><ul>Lourens Naude, WildfireApp.com </ul>
  2. <ul>Background </ul><ul><li>Independent Contractor </li></ul><ul><ul><li>Ruby / C / integrations
  3. Well versed full stack
  4. Architecture </li></ul></ul><ul><li>WildfireApp.com </li></ul><ul><ul><li>Social Marketing platform
  5. Large whitelabel clients
  6. Bursty traffic – Lady Gaga, EA, Gatorade </li></ul></ul>
  8. <ul>Audience </ul><ul><li>Monolithic applications </li></ul><ul><ul><li>Scale together, fail together
  9. Often have an anemic domain model, scattered logic
  10. Hockey stick function – fast initial growth, increasingly difficult to extend over time
  11. Hacks always have the longest lifecycle in a codebase </li></ul></ul><ul><li>Target platforms </li></ul><ul><ul><li>Projects with significant complexity
  12. Existing monolithic applications
  13. Environments with excessive coupling or dependencies </li></ul></ul>
  15. <ul>Anemic Domain Models </ul><ul><li>Cost to business </li></ul><ul><ul><li>No competitive advantage
  16. Long dev -> production cycle </li></ul></ul><ul><li>Identified by ... </li></ul><ul><ul><li>Tech oriented interfaces
  17. High coupling to neighboring layers
  18. Coupled persistence
  19. Dependencies on host frameworks </li></ul></ul>
  20. <ul>Layered Architecture </ul><ul><li>Layers </li></ul><ul><ul><li>UI
  21. Application
  22. Domain
  23. Infrastructure </li></ul></ul><ul><li>Rails && other MVC ? </li></ul><ul><ul><li>All of the above, monolithic tree
  24. Modeling is often data driven, not behavior driven
  25. Models / entities are often coupled to conventions and dependencies of the host framework </li></ul></ul>
  26. <ul>Problems in practice </ul><ul><li>Rails as a niche framework </li></ul><ul><ul><li>Initial conventions scale for small to medium size applications
  27. Fast time to market
  28. Request / response + rendering
  29. Beyond this, generally not a Rails application anymore </li></ul></ul><ul><li>Development and growth </li></ul><ul><ul><li>Your application, over time, becomes your framework for future development
  30. Conventions, practices and dependencies should grow with and adapt to that also </li></ul></ul>
  32. <ul>Custom policies and governance </ul><ul><li>Cater for both developers + production runtime </li></ul><ul><ul><li>Environment support with sanity checks
  33. Configuration management
  34. Dependency management etc. </li></ul></ul><ul><li>How ? </li></ul><ul><ul><li>Thin framework agnostic layer that provides tooling for developers
  35. Piggyback off known contracts eg. cache['something']
  36. Deprecate framework specific references in code
  37. Prefer SomeApp.cache to Rails.cache etc. </li></ul></ul>
  38. <ul>Moving off monilithic apps </ul><ul><li>How features used to roll out </li></ul><ul><ul><li>Slapped onto the existing codebase OR
  39. Spawned a new project, with yet another set of conventions
  40. Topic / feature development branches </li></ul></ul><ul><li>Challenges </li></ul><ul><ul><li>Team likely not structured to support distributed / decoupled systems
  41. Extreme self-discipline, from everyone
  42. Business Process Model skills is required </li></ul></ul>
  44. <ul>Digesting a domain </ul><ul><li>Language </li></ul><ul><ul><li>Ban acronyms
  45. Push for a common language – dev, product + biz
  46. Be childlike.Why, why, why ... </li></ul></ul><ul><li>Open ended </li></ul><ul><ul><li>A domain representation is always in flux during development, especially if it's new to developers
  47. It'll be enhanced with new concepts
  48. Weed out vague / deprecated elements
  49. Refactorings at this level is often overlooked </li></ul></ul>
  50. <ul>Refactoring strategies </ul><ul><li>Continuously throw usage scenarios at your system
  51. Refactor to deeper insights
  52. Extract hidden concepts – look for edge cases, policies etc.
  53. In cooperation with domain experts
  54. Refactor your domain first, code second </li></ul>
  55. <ul>Behavior and responsibility </ul><ul><li>Ignore schema and attributes </li></ul><ul><ul><li>Should not be the core of your design process
  56. Assigning data to entities prior to knowing that they do ? </li></ul></ul><ul><li>Behavior </li></ul><ul><ul><li>Partition objects and entities by behavior
  57. Distribute responsibilities first
  58. Then decide what each entity needs to know </li></ul></ul>
  60. <ul>Domain leaks - People </ul><ul><li>Developers </li></ul><ul><ul><li>Code ownership
  61. Some developers don't communicate well
  62. Snapshot thoughts to wiki != fun
  63. People come, people go </li></ul></ul><ul><li>Product and business </li></ul><ul><ul><li>Not willing to get into domain details
  64. Concerned with features
  65. Concerned with cash flow </li></ul></ul>
  66. <ul>Domain leaks – Process </ul><ul><li>Reorganization </li></ul><ul><ul><li>Migrates knowledge excessively
  67. Fine balance between code ownership and moving developers around too much </li></ul></ul><ul><li>Practices and culture </li></ul><ul><ul><li>Lack of Documentation culture
  68. Not factoring clued up domain experts into projects
  69. Shielding departments from each other – us VS them
  70. Visual communication.Whiteboards ? Diagrams ?
  71. Domain discussions encouraged ? Shot down ? </li></ul></ul>
  72. <ul>Domain leaks - concepts </ul><ul><li>Implicit concepts </li></ul><ul><ul><li>Instrumental in understanding a model
  73. But never referenced in an implementation
  74. Frequently would be hinted at, but may not seem important to pull in </li></ul></ul><ul><li>Explicit concepts </li></ul><ul><ul><li>Merging implicit concepts back into the domain
  75. Over time, this leads to breakthroughs
  76. Less edge cases
  77. Ability to handle offline process programatically </li></ul></ul>
  78. <ul>Notes on errors </ul><ul><li>Common error types </li></ul><ul><ul><li>Runtime errors
  79. Exceptions
  80. Errors in business logic </li></ul></ul><ul><li>Explicit domain errors </li></ul><ul><ul><li>Don't inline raise
  81. Do include business errors in your domain
  82. Define strategies for offline or programatic handling </li></ul></ul>
  83. <ul>STATE OF YOUR DOMAIN </ul>
  84. <ul>Loosely coupled domain </ul>-------------------------------------- X <-- X <-- X <-- X --------------------------------------
  85. <ul>Sparse domain </ul>-------------------------------------- X <-- <-- <-- X --------------------------------------
  86. <ul>Highly coupled domain </ul>-------------------------------------- X X <-- X X <-- X X <-- XXXX --------------------------------------
  87. <ul>COMMUNICATION </ul>
  88. <ul>Value of speech </ul><ul><li>Talking out loud </li></ul><ul><ul><li>Embrace senses
  89. Colorful speech
  90. “ A valid tax invoice for taxable sales that totals x”
  91. Fail faster – quicker feedback loop </li></ul></ul><ul><li>Translation costs </li></ul><ul><ul><li>Statement ...
  92. Refine statement ...
  93. Agree on statement ...
  94. repeat </li></ul></ul>
  95. <ul>A Common Language </ul><ul><li>Participants </li></ul><ul><ul><li>Domain experts
  96. Stakeholders
  97. Developers </li></ul></ul><ul><li>Referenced in </li></ul><ul><ul><li>Code
  98. Test code
  99. Discussions and meetings
  100. Specs / diagrams
  101. Documentation </li></ul></ul>
  102. <ul>Common Language Elements </ul><ul><li>Nouns </li></ul><ul><ul><li>Entities / classes if it has identity
  103. Invoice, transaction, business, user, expense etc.
  104. Value object if there's no identity.Reusable and immutable
  105. Address, Money etc. (load Address via User) </li></ul></ul><ul><li>Verbs </li></ul><ul><ul><li>Services, methods etc. - behavior
  106. “ A transaction between businesses generates an invoice.”
  107. @business.sells(2, product, other_biz) </li></ul></ul>
  108. <ul><li>VERBS AS COMMANDS </li></ul>
  110. <ul>Command -> event stream </ul>Sell(:seller => 'Acme', :quantity => 2, :product => 'XYZ', :buyer => 'MegaCorp') @business.sells(2, product, other_biz) [PaymentProcessed, TransactionCreated, InventoryUpdated, InvoiceCreated, ShippingLabelsGenerated ]
  111. <ul><li>EXPLICIT STATE CHANGES </li></ul>
  112. <ul>Pushing changes </ul><ul><li>Rethink persistence </li></ul><ul><ul><li>Remove persistance behavior from entities
  113. Focus on the domain, not data structure
  114. Delegate storage to a repository
  115. Easy to swap out repository implementations for testing etc. </li></ul></ul><ul><li>Domain events </li></ul><ul><ul><li>Instead of saving to the database, generate a domain event with the relevant changes as a payload
  116. Published to your whole stack
  117. Eventual consistency </li></ul></ul>
  118. <ul>ActiveRecord implications </ul><ul><li>Reporting contexts </li></ul><ul><ul><li>ActiveRecord::Base.read_only!
  119. Bypass persistence entirely – force ready-only mode
  120. @record.attributes #=> value object
  121. @record.changed #=> value object diff </li></ul></ul><ul><li>Repository use cases </li></ul><ul><ul><li>A Generic Repository implementation (CRUD)
  122. Invoice.find(1) #=> entity by identity
  123. Invoice.find_by_refr(x) #=> dynamic and explicit query methods </li></ul></ul>
  125. <ul>Commands </ul><ul><li>Drive change in your system </li></ul><ul><ul><li>Time driven : batched, cron etc.
  126. Request / Response : remote producedure calls </li></ul></ul><ul><li>Definition </li></ul><ul><ul><li>A well defined task / behavior with parameters
  127. Not expected to return a result
  128. Should be idempotent
  129. Should expire itself when not processed in a timeframe
  130. Yields an event stream through interaction with the domain when processed instead </li></ul></ul>
  131. <ul>Domain Events </ul><ul><li>Granularity </li></ul><ul><ul><li>InvoiceChanged, InvoiceStateChanged, InvoicePaid
  132. Try to signal the intent of a change
  133. Wrap InvoiceChanged in a module
  134. InvoicePaid.include InvoiceChanged </li></ul></ul><ul><li>Immutability </li></ul><ul><ul><li>Represents something that has happened
  135. Thus properties should never be changed </li></ul></ul>
  136. <ul>Event Driven Architecture (EDA) </ul><ul><li>Nervous system anology – hand on stove </li></ul><ul><ul><li>Hand is the event producer
  137. Spinal cord is the ESB
  138. Brain is the event listener
  139. Brain is the event processor
  140. Hand is the event reaction – we pull away </li></ul></ul><ul><li>Common event patterns </li></ul><ul><ul><li>Generation
  141. Consumption
  142. Processing </li></ul></ul>
  143. <ul>EDA attributes </ul><ul><li>Definition </li></ul><ul><ul><li>Loosely coupled / decoupled
  144. Async messaging
  145. Granular events
  146. Notification of state changes </li></ul></ul><ul><li>Transport </li></ul><ul><ul><li>ESB backbone
  147. No centralized control </li></ul></ul>
  150. <ul>* NOT BOTH * </ul>
  152. <ul>Major differences </ul><ul><li>Queries </li></ul><ul><ul><li>Synchronous and blocking
  153. Idempotent by default </li></ul></ul><ul><li>Commands </li></ul><ul><ul><li>Asyncronous – we expect no results …
  154. Other than a guarantee it's been received
  155. Should be idempotent / not reprocessed </li></ul></ul>
  156. <ul>Read and write seperation </ul><ul><li>Write service </li></ul><ul><ul><li>POST /invoices
  157. PUT /invoice/1
  158. DESTROY /invoice/1
  159. Append to a FIFO command queue </li></ul></ul><ul><li>Read service </li></ul><ul><ul><li>GET /invoice/1
  160. GET /invoices
  161. Serve from an eventually consistent reporting DB </li></ul></ul>
  162. <ul>“ Hot” upgrades </ul><ul><li>Scenario </li></ul><ul><ul><li>Read heavy web application
  163. 95% read, 5% write
  164. System upgrade required
  165. Ability to disable features is explicit in domain </li></ul></ul><ul><li>Solution </li></ul><ul><ul><li>Mirror all data
  166. Disable all write services + related features
  167. Continue serving requests </li></ul></ul>
  168. <ul>CRUD FAIL </ul>
  169. <ul>Why ? </ul><ul><li>Audits
  170. Can't be easily audited </li></ul><ul><ul><li>You can, but pointless if your system isn't dependent on this history – no way to trust it </li></ul></ul><ul><li>Accountant workflow </li></ul><ul><ul><li>Accountants don't change objects
  171. They keep histories
  172. No updates, no deletes
  173. Work is always auditable
  174. Totals is always based on history </li></ul></ul>
  175. <ul>Change oriented updates </ul><ul><li>CRUD API </li></ul><ul><ul><li>Create
  176. UPDATE
  177. Delete
  178. What's changed on UPDATE ? </li></ul></ul><ul><li>A better API </li></ul><ul><ul><li>#rename_invoice
  179. #update_line_item_quantity </li></ul></ul>
  181. <ul>Audit workflow </ul><ul><li>Commands </li></ul><ul><ul><li>Stored in a sequential FIFO fashion
  182. Allows for compensation on multiple updates
  183. Entity state is driven exclusively from this
  184. Guaranteed – as there's no other persistence </li></ul></ul><ul><li>Audit logs </li></ul><ul><ul><li>Command -> Domain -> Domain event
  185. Aggregates domain events
  186. Feeds into a reporting database </li></ul></ul>
  187. <ul>What's really happening here ? </ul><ul><li>Isolated contexts </li></ul><ul><ul><li>Command store
  188. Audit log store
  189. Reporting database
  190. Agree through communication </li></ul></ul><ul><li>Eventual consistency </li></ul><ul><ul><li>We don't guarantee consistenty accross these contexts
  191. at all times
  192. We do guarantee that it'll be consistent eventually
  193. Relaxed consistency is often acceptable </li></ul></ul>
  194. <ul>Relaxed Consistency </ul><ul><li>Viability ? </li></ul><ul><ul><li>Discuss with domain experts
  195. How long is too long ? </li></ul></ul><ul><li>Solutions </li></ul><ul><ul><li>Explicit SLA's at the command level
  196. Measure propogation and flag SLA violations </li></ul></ul>
  198. <ul>Performance benefits </ul><ul><li>Throughput </li></ul><ul><ul><li>High transactions/sec rate
  199. Total throughput is always governed by the weakest link </li></ul></ul><ul><li>Read / write specific optimizations </li></ul><ul><ul><li>Latency: data being far from where it's needed
  200. Never block a write on waiting for data – it's given
  201. Can use fast disks for write oriented services
  202. Cache read tier ftw!
  203. Formal state changes == better cache invalidation </li></ul></ul>
  204. <ul>Testing </ul><ul><li>Service / daemon structure </li></ul><ul><ul><li>Command -> Result
  205. Result being either an Event or Error </li></ul></ul><ul><li>Workflow </li></ul><ul><ul><li>test/unit, domain elements + your service
  206. Layered architectured == easy to swap out layers
  207. Arrays to mock queue infrastructure
  208. Behavior driven testing
  209. Assert resulting events or errors </li></ul></ul>
  210. <ul>Asserting events </ul>process MyApp::Payments do command :PayInvoice, :id => 1 assert_events :InvoicePaid, :RegenInvoice end
  211. <ul>Asserting errors </ul>process MyApp::Payments do command :PayInvoice, :id => 200 assert_errors :InvoiceAlreadyPaid end
  212. <ul>Commands in test cases </ul>def command(cmd, attrs) cmd = App::Commands.const_get(cmd).new attrs.each{|k,v| cmd[k] = v } @process << cmd end alias input command
  213. <ul>Asserting event streams </ul>def assert_events(*events) event_types = @output.map{|e| e.class } expected = events.map{|e| App::Events.const_get(e) } assert_equal expected, event_types, &quot;Expected events #{expected.inspect}, got #{event_types.inspect}&quot; end
  214. <ul>MESSAGING </ul>
  215. <ul>Messages - Communication </ul><ul><li>Communication styles </li></ul><ul><ul><li>Fat consumer : service call to determine destination, then send, slow
  216. Thin consumer : fire and forget, ESB routes, fast </li></ul></ul><ul><li>ESB </li></ul><ul><ul><li>Take time to research this
  217. Understand the protocol
  218. Client support
  219. Durability and failover requirements
  220. Publish / subscribe is preferred </li></ul></ul>
  221. <ul>Messages - Types </ul><ul><li>Loose coupling requirements </li></ul><ul><ul><li>Weak typing, loose data model - hashes
  222. Data types not always harmonized – accept that
  223. Also account for foreign / API messages </li></ul></ul><ul><li>Translation support </li></ul><ul><ul><li>Requires mapping between systems
  224. Individual systems can modify their internal structures, providing we update our mapping layer
  225. Useful if we have foreign messages (via API) to push in our stack as well – just coerce to the internal representation </li></ul></ul>
  226. <ul>Messages - Versioning </ul><ul><li>Embracing change </li></ul><ul><ul><li>Structures will change
  227. Version backward incompatible changes
  228. Intermediate sensible defaults for backward compatible changes </li></ul></ul><ul><li>Strategies </li></ul><ul><ul><li>Let queues be a part of version strategy also
  229. Allows for keeping old + new messages – no loss
  230. Thin and temp. translator to pop messages off the old queue, translate / coerce them, push to the new one </li></ul></ul>
  231. <ul>Messages - Payload </ul><ul><li>Minimal </li></ul><ul><ul><li>Just enough information and context for consumers to determine if updates is relevant to them
  232. Consumer required to lookup any additional information required
  233. Couples to services + point of failure </li></ul></ul><ul><li>Self contained </li></ul><ul><ul><li>Includes all information a consumer would require to process
  234. Loosely coupled – no additional service interactions
  235. Watch out for size – Amazon SQS is limited to 8k </li></ul></ul>
  236. <ul>Messages - Correlation </ul><ul><li>Why it's required </li></ul><ul><ul><li>Troubleshooting – backtraces is always app local
  237. Enforcing SLAs
  238. Idempotency </li></ul></ul><ul><li>Helpful correlation id elements </li></ul><ul><ul><li>Source system
  239. Destination system
  240. /rails/user/234/session_dssasdadas/controller/action
  241. Consider a “touch” mecanism – history of systems the
  242. message has passed through </li></ul></ul>
  243. <ul>QUESTIONS ? </ul>