CQRS why, what, how


Published on

This talk covers WHY we might want to look at CQRS, by first looking at the kinds of problems we see with the current architectural approaches. We will look at WHAT CQRS is, how it works well with DDD, look at staleness of data in collaborative domains and how we can use this to our advantage. We will look at the building blocks for CQRS, how to test it and how to build a bigger solution the involves communication between different parts of the solution/bounded contexts. We will look at how we can use CQRS to scale the solution (by looking at the CAP theorem and the BASE properties of distributed computing). Finally we will look at how we can approach legacy system migration by looking at 4 different alternatives to rewriting them from scratch. Key words are for this is published language, Anti Corruption layer and an Event Driven Architecture (which might utilize CQRS internally).

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • We’realwaysending up with the same old model for architecture . The layered approach, which is fine in and of it self, but it needs to held up againstour domain and solution complexityData are still vertical
  • Scale up vertically
  • There is nothing new happening – it’s all the same old just a little bit differentPeople do the same every timePeople dare not pursue new ways of doingthingsPeople dare not tryanything newDo the same eachyear, although all the participants aredead
  • Are you open to new ideas?
  • It is not possible to create a context-free modelDoing so youwillonlygetsomethingthatdoes not work in anycontextToo manyunintended side effectsExample. Life insurance with lifecoverage. My companyassuresme, mywife and childrenarebeneficiaries. Who is the customer? It depends on what point in time I look at it.
  • Aggregates reference each other by Id only in a scalable systemAn Aggregate Root is more likely to match a Use Case than any model structure
  • How did the business handle race conditions in the old days?
  • Spaghetti
  • Lasagna
  • [OrderService|registerOrder(...);closeOrder(...);updateAvailablility(...);addToOrder(...)]->[Order|OrderDate;Amount;Customer|get/setDate();get/setDate();get/setCustomer();get/setOrderLines()], [Order]++-*>[OrderLine|ProductId;UnitPrice;NoOfUnits|get/setProductId();get/setUnitPrice();get/setNoOfUnits()]
  • [OrderService|registerOrder(...);closeOrder(...);updateAvailablility(...);addToOrder(...)]->[Order|OrderDate;Amount;Customer|update();close();totalAmount()], [Order]++-*>[OrderLine|ProductId;UnitPrice;NoOfUnits|calculateAmount()]
  • From http://www.codeagle.com/blog/2010/03/10/275/
  • Ourread model is structurallyoriented (the usersperspective on the data in the given context)The write model is behaviouraloriented (transaction and consistencyboundaryoriented)
  • Why is this the bestwecan offer – Whyshould theuserbeinterrupted by a technicalconstraint?
  • Commands and Events areimmutable
  • [Order|OrderDate;Amount;Customer|addProduct();removeProduct();ship();...|onProductAdded(..);onProductRemoved(..);onOrderClosed(..);onOrderShipped(..)],[Order]++-*>[OrderLine|ProductId;UnitPrice;NoOfUnits|calculateAmount()]
  • EventStore: Why store only the currentstate of the system.Matureindustries store transactions and calculate the currentstate
  • Wecanassertthatbothwhatweexpectedhappened and nothingelsehappened (the unexpectedbecomesexplicit)
  • This report is generatedbased on the previous test code
  • (but youcanchosewhichare most important to you in the different cases)
  • ACID:Atomic: The transaction is indivisible - either all the statements in the transactionareapplied to the database, or none are.Consistent: The database remains in a consistentstatebefore and aftertransactionexecution.Isolated: While multiple transactionscanbeexecuted by one or more userssimultaneously, onetransactionshould not see the effects of otherconcurrenttransactions.Durable: Once a transaction is saved to the database (an action referred to in database programmingcircles as a commit), itschangesareexpected to persist.
  • From http://www.codeagle.com/blog/2010/03/10/275/
  • Use Cases: There is a difference between the static and the dynamic part of a systemThe dynamic part contributes to provides an precisecontext
  • Use Cases: There is a difference between the static and the dynamic part of a systemThe dynamic part contributes to provides an precisecontext
  • Loose coupling - data and temporalWeb shop can run without the otherbeing online
  • This is what a typicalenterprise system landscape looks likeIt has evolved over manyyears and itscurrentstate is the result of good decisions, bad decisions, time preassure, system pruchases, mergers, management, politics
  • Reimplementationcanbeextremelyexpensive.Years go by and we’re still just reimplementingwhatwe have today.Year 3 wecanfinally do whatwewantedoriginally…And perhaps whatwewanted to do, isn’t relevant anymore..
  • ETL = Electronic LandRegistration (Elektronisk TingLysning in Danish)
  • New strategic Business App is needed and LIMITED data is needed from legacyappsChangingexisting Systems (X + Y) model/interfaces canbedaunting. Rippleeffects due to coupling to other systems/modules/componentsChanging the model mightalso not work with the current system functionality, requiering a rewritewhich is expensiveAdding data/model withoutchanging the old onewilllead to duplication/conflictingrules/concepts.The bubblecontextallows for incrementalimprovementwithout the need for bigchanges.Note: The ACL risksstealingtoomuch business logicduring TransformationSee http://www.domainlanguage.com/newsletter/2011-06/ for details
  • A betterway to control side effectscompared to a repositoryoritented approach (handles logicmuchbetter)
  • Legacy systems modified to create and consume eventsEvents areeasier to translatethan normal messages/objectssincetheycontaincontext information
  • CQRS why, what, how

    3. 3. Complexity of code and solution Typical causes: • Too many layersComplexity • Big data models • Anemic domain model • Focus on frameworks instead of on the domain • Scalability not considered at the core of the design (scalability gets hacked in too late) Time
    4. 4. Typical One size fits all ”architecture” Client DTO DTO Remote Facade Application Service Domain Domain Object Object Data Access Layer Data Storage
    5. 5. Many perspectives on data Online Ordering System Product Pricing Unit Price Promotional Price Promotion End Date Stock Keeping Unit (SKU) Quantity On Hand (QOH) Location Code InventoryCustomer Price Quantity Ordered Name Sales What about the lifecycles for the data?
    6. 6. One big model to capture it ALL …. ….. …..….…..….. …. ….. ….. …. ….. …..
    7. 7. Result: Our queries get messy
    9. 9. The database determines our scalability
    10. 10. And databases often scale poorly To solve the performance problems for a few, you have to upgrade it all
    11. 11. Alternatively we can use Read replicas Slave Slave Master Slave Slave You’re now eventually consistent
    12. 12. Or introduce caching… ?What’s your synchronization mechanism?
    13. 13. SOA / Integration is often an after thought
    14. 14. THE Solution?
    15. 15. Same procedure as last year?Same procedure as every year!!!
    16. 16. Open for new ideas? Poul, you idiot, whats that!?You should only bring things we can eat!
    17. 17. IT exists to support the business by support various usecasesUsecases can be categorized as either• ”User” intent to manipulate information• ”User” intent to find and read informationWe already support this in OO at a Property level:• Setter (or mutator) methods Manipulate data• Getter (or accessor) methods Read data Let’s raise it to a higher Level
    18. 18. CQS Separation of functions that write & functions that readFunctions that write are called Command methodsand must not return a valueFunctions that read are called Query methods andmust have no side effects
    19. 19. CQSCommands – Change data UI Application Domain DataQueries – Ask for data (no side effects) UI Application Data
    20. 20. CQS from a code perspectivepublic class CustomerService{ // Commands void MakeCustomerPreferred(CustomerId) void ChangeCustomerLocale(CustomerId, NewLocale) void CreateCustomer(Customer) void EditCustomerDetails(CustomerDetails) // Queries Customer GetCustomer(CustomerId) CustomerSet GetCustomersWithName(Name) CustomerSet GetPreferredCustomers()} From: https://gist.github.com/1964094
    21. 21. SO WHAT IS CQRS?CQRS is simply the creation of two objects where there was previously only one Greg Young
    22. 22. CQRS In its simplest formpublic class CustomerWriteService{ // Commands void MakeCustomerPreferred(CustomerId) void ChangeCustomerLocale(CustomerId, NewLocale) void CreateCustomer(CreateCustomer) void EditCustomerDetails(CustomerDetails)}public class CustomerReadService{ // Queries Customer GetCustomer(CustomerId) CustomerSet GetCustomersWithName(Name) CustomerSet GetPreferredCustomers()} From: https://gist.github.com/1964094
    23. 23. Commands A Command’s primary goal is to capture USER INTENT A Command supports a single usecase and targets a single AggregateCommands are stated in the imperative: – DoStuff – CreateCustomer – ShipOrder – AddComment – DeleteComment
    24. 24. That’s all you need to know Thanks for your attention 
    25. 25. There’s always more than onemodel at play in any big project But when code based on multiple models is combined
    26. 26. … then the code becomes filled with errors, unreliable and difficult to understand Communication between team members becomes confusing. It is often unclear in what context a model should NOT be used!
    27. 27. Context is lacking
    28. 28. CONTEXTThe setting in which a word or a statement appears that determines its meaning
    29. 29. Smaller models & clear data ownership Online Ordering System Pricing Product DDD: SKU Bounded Unit Price Promotional Price Context … Sales Product SOA: SKU Name Entity identity Pricing Service Description Quantity Ordered … Inventory Product SKU QOH Location Code … Inventory Sales
    30. 30. Core DDD patternsBOUNDEDCONTEXT Repositories Factories Layered Architecture ENTITIES VALUE OBJECTS Modules Services AGGREGATES
    31. 31. Let’s first get the vocabulary right So we all speak the same UBIQUITOUS LANGUAGE
    32. 32. Entities Customer - ID/Social Security Number : NumberMotivators:Objects are defined by identity and NOT by their attributes.May have lifecycle, can change form by time and context (A Personthat becomes a father, who becomes a grand father, etc.) Focus is on Who and not on what!
    33. 33. Value Objects Customer - ID/Social Security Number : Number 1 AddressMotivators:Objects which describe things and are - street : Stringdefined by their attributes (not by their - zipCode : Stringidentity) and doesn’t have a lifecycle. Focus is on What and not on who!
    34. 34. Aggregates Account Invoice * RootWhat:• Cluster coherent Entities and Value Objects, with complex associations into Aggregates with well defined boundaries. *• Choose one entity to be root and control access to objects inside the boundary through the root. InvoiceLine• External objects hold references to the root• Aggregates only refer to other aggregates by identity (their id)Motivation:Control invariants and consistency through the aggregate root.Enables: Loading schemes, coarse grained locking, consistency/transactional boundariesfor DDDD
    35. 35. How have our ”architectures” evolved
    36. 36. Thin Layers of complexity Language of the database UI DataPro: Easy and simpleCon: Scales poorly for complex domains
    37. 37. Next step – more layers Language of Language of Language of getters and DTO’s ORM magic setters UI Application Domain Data ComplexityPro: Handles complex domains better by TimeSeparating usecase and domain logicCon: Increasing complexity. Risk of AnemicDomain model combined withtransaction-script.
    38. 38. Anemic Domain ModelThe Good parts The bad parts• All Order logic is called in ONE • Hard to test class • Hard to maintain• Lower entry level for junior • Code dupliction developers • An exposed domain model is like walking around without clothes – everyone can access your private parts and violate your invariants
    39. 39. We need a rich domain model Aggregate with Order as Aggregate Root• Order forms our Unit of consistency, which controls our invariants• Order and OrderLines are created, updated and deleted together
    41. 41. Circular architecture Task based UI DomainRead Model Model
    42. 42. 1. Realization Commands and Queriessupport very different usecases
    43. 43. A single model cannot beappropriate for reporting, searching and transactional behavior Greg Young, 2008
    44. 44. 2. realizationQuery results contain only data – no logic• Give me the customer with his last 10 orders• Give me the customer with total sum of orders for the last year• Give me the complete order• Give me the shipping status for the order• Give me the customers last reviewSo why go through the domain layer? UI Application Data
    45. 45. Things to think about in relation to Queries• Why take the pain of performing JOINS and/or UNIONS of our nicely normalized Relational Model?• Why not having a completely denormalized data model (or View) to support our Query?• When we completely denormalize our data, do we really need a relational database or could we use a Key/Value- or Document Store?• Do we need our Read Data to up to the microsecond consistent with our Write data?
    46. 46. So how is this relevant in the real world? Let’s look at a real life scenario
    47. 47. Collaborative domains 1. Read 2. Read4. Update 3. Coffee time 5. Stale data
    48. 48. We’re working with stale data
    49. 49. We’re always working with stale data! + thinking time (1000-1500 ms) 20 ms 20 ms At this point our data is at least 120 ms stale 1 ms 1 ms + Thinking time 100 ms 100 ms 100 ms
    50. 50. LET’S USE STALE DATA TO OUR ADVANTAGE BY OFFLOADING THEDATABASE BY USING OUR READ MODELSCommands – Change data UI Application Domain Write modelQueries – Ask for data (no side effects) UI Application Read models
    51. 51. ALL WE NEED IS A GOOD WAY TO SYNCHRONIZE OUR WRITE AND READ MODELSCommands – Change data UI Application Domain Write modelQueries – Ask for data (no side effects) UI Application Read models
    52. 52. Let’s make the implicit explicit!The original DDD book left out a very important concept: Domain EventsWhich:• Signal that something happened• Closely aligned to the Domain Model• Are handled by a messaging system• They are in the past tense: – CustomerBilled – ParcelShipped – CustomerCreated – ReviewCreated – CommentAdded – CommentDeleted
    53. 53. Commands & Events• Commands mutate Aggregate state which results in one or more Events being issued/published. Command Event(s) AcceptOrder OrderAccepted ShipOrder OrderShipped AddComment CommentAdded QuarantineReview ReviewQuarantined UnquarantineReview ReviewUnquarantined
    54. 54. With Domain Events we now have amechanism to support our denormalized View/Query models
    55. 55. Commands, Events and Query Models UI ”Find all Accepted Orders” ”AcceptOrder” Query command Events Query model Domain model Read model Read model ”OrderAccepted” eventCommands are Imperative: DoStuff Events are Past tense: StuffDone
    56. 56. Messaging Architectures Most CQRS implementations see Commands and Events as (asynchronous) Messages public class CreateCustomer {Command public final Guid CustomerId; public final Name CustomerName; … } Difference: public class CustomerCreated INTENT { Event public final Guid CustomerId; public final DateTime CreationDateTime; public final Name CustomerName; }
    57. 57. CQRS Building blocks Command Command Modify Commands Domain Bus Handlers Command Services Read Write Sends Data store Repositories EventsClient Event Events Event Handlers Bus Queries Query Services Read store Query Results Query Handlers
    59. 59. Event Sourcing Aggregates track their own Domain Events and derive state from themOctober October October October October October 5 6 6 7 7 9 OrderCreated ProductAdded ProductAdded ProductRemoved ProductAdded OrderShipped
    60. 60. Full CQRS With EventSourcing Our singleCommands – Change data source of truth Commands Events UI Domain EventQueries – Ask for data Store Events UI Query Data Build SQL DB Document DB Graph DB
    61. 61. public class CustomerCommandHandler { private Repository<Customer> customerRepository; public CustomerCommandHandler(Repository<Customer> customerRepository) { this.customerRepository = customerRepository; } @CommandHandler public void handle(UnsignCustomer cmd) { Customer customer = repository.load(cmd.getCustomerId()); customer.unsign(); }}public class Customer { private boolean signedUp; public void unsign() { if (signedUp) { apply(new CustomerUnsignedEvent()); } } @EventHandler private void handle(CustomerUnsignedEvent event) { signedUp = false; }}
    62. 62. Testing CQRS BDD style@Testpublic void a_signedup_customer_can_unsign() { UUID customerId = UUID.randomUuid().toString(); FixtureConfiguration fixture = Fixtures.newGivenWhenThenFixture(); fixture.registerAnnotatedCommandHandler( new CustomerCommandHandler(fixture.createGenericRepository(Customer.class)) ); fixture.setAggregateIdentifier(customerId); fixture .given( TestFactory.customerCreatedEvent(customerId), TestFactory.customerSignedUpEvent(customerId) ) .when( TestFactory.unsignCustomerCommand(customerId) ) .expectEvents( TestFactory.customerUnsignedEvent(customerId) );}
    63. 63. CQRS / BDD Report generated based on the previous testScenario: A signedup customer can unsignGiven a Customer with id ”abc1234” thathas been created and a customer with id”abc1234” that is signedupWhen a request for Customer with id”abcd1234” to be unsigned is receivedThen the Customer with id ”abcd1234” isunsigned
    64. 64. SOA / Integration Commands, Queries and Events form natural integration interfacesSo integration is backed in from the beginning!
    65. 65. What about Scalability?CAP Theorem• Consistency: All nodes in the cluster see exactly the same data at any point in time• Availability: Failure of a node does not render the entire system inoperative• Partition tolerance: Nodes can still function when communication with other groups of nodes is lost You can’t have all three!
    66. 66. CAP theorem ACID systems are hard and expensive to scaleUnless you use Latency Pessimistic concerns Locking – all data is stale(and eventual consistentwhen delivered to the user) Strong  Weak  Eventual Consistency Source: http://bit.ly/QUurnY
    67. 67. BASE Basically Available Soft-state Eventually consistent Eventually Consistency levels: • Causal consistency: This involves a signal being sent from between application session indicating that a change has occurred. From that point on the receiving session will always see the updated value. • Read your own writes: In this mode of consistency, a session that performs a change to the database will immediately see that change, even if other sessions experience a delay. • Monotonic consistency: In this mode, A session will never see data revert to an earlier point in time. Once we read a value, we will never see an earlier value.See http://www.allthingsdistributed.com/2008/12/eventually_consistent.html for more
    68. 68. CQRS can give us BASE at the architectural level throughasynchronous Commands and Events This gives the business and IT control over where to spend money to scale our solution - instead of trying to buy a bigger database server.
    69. 69. CQRS requires a New UI paradigme Task-based/Inductive UI• A depart from the traditional WHAT UI (CRUD)• Task based UI’s focuses on HOW the user wants to use the application• Guides users through the work process• The UI becomes an intrinsic part of the design• The UI design directly affects our commands and thereby our transactional boundaries• Should be preferably use asynchronous Commands
    70. 70. What about Validation• Happens in the UI before Commands are sent• Typically validated using Read models• Helps to ensure that commands don’t often fail• Validates simple things (values ranges, unique keys/values)• Doesn’t enforce business logic rules
    71. 71. Circular architecture Task based UI DomainRead Model Event Event Event Model
    72. 72. Separation of Concerns• Different developer skills and roles for command and query parts• New Business Requirements ripples controllably between areas• No need for Getters & Setters in your domain model• Testability – Event based testing is very easy
    73. 73. CQRS is not top level architecture Somethings missing How do we collaborate?
    74. 74. Composite UI’s Online Ordering System Sales Pricing Inventory Composite UIWeb/Application UI tier Layered Data Access Layer Application Write model Read modelBackground server tier Storage tier
    75. 75. Composite UI - example
    76. 76. That covers data presentation What about transactions?
    77. 77. Next Step – Event Driven Architecture Online Ordering System Pricing Service Product SKU New SKU Unit Price Promotional Price Event … Web Shop Who (Composite UI) coordinates the sales process? PricingCustomers Sales Inventory Service (SAP) Product Product SKU SKU Name Order QOH Price Accepted Location Code Quantity Ordered … … Event Inventory Message Bus New SKU New SKU Event Event Sales
    78. 78. Introducing Sagas Online Ordering System Billing Service Order Accepted Customer BilledSales Service Order Shipping Accepted Order Process Manager Accepted (Saga) Customer Billed Ship Order Message Bus Shipping Service Ship Order
    79. 79. So what do we do? We have this dream
    80. 80. ”Nirvana” Sales Pricing Inventory Composite UIWeb/Application tier UI Layered Application Data Access Layer Write model Read modelBackground server tier Storage tier Billing Service Order Accepte d Customer Billed Sales Service Order Shipping Accepte Order Process Manager d Accepte d Customer Billed Ship Order Message Bus Shipping Service Ship Order
    81. 81. …and this realitySystem X System Y System Z SAP
    82. 82. Finding strategic value
    83. 83. Reimplementation Still relevant?
    84. 84. Perhaps a more viable solution? Big Ball Of Mud
    85. 85. What path to take?
    86. 86. 1. Bubble context
    87. 87. New Context Z Business Application Repository Repository Anti Corruption Layer Transformation Transformation X↔Z Coordinator Y↔ZSystem X Service Repository System Y Data
    88. 88. Bubbles burst• Functionality gets absorbed into the legacy systems• Maintenance becomes too expensive• Expansion becomes too cumbersome• People start bypassing the ACL (encapsulation breaks)• …• Performance becomes a problem
    89. 89. 2. ACL Synchronization New Context Z Business Application Repository Repository Data Anti Corruption Layer Asynchronous updates Transformation Transformation X↔Z Coordinator Y↔Z System X Service Repository System Y Data
    90. 90. 3. ACL backed Open Host service New Context B Service Anti Corruption Layer Transformation A↔B Coordinator System A
    91. 91. 4. Event Channels New Context ZRepository Repository System X Service Repository System Y Data Query Update Anti Corruption Layer Transformation Events X↔Z Events Coordinator Events Messagebus Event Channel Published Language
    92. 92. Questions?Email: jeppe@tigerteam.dk @TigerTeamDK on TwitterWeb: http://www.tigerteam.dk @jeppec on Twitter
    93. 93. Useful linksCQRS/DDD:• Bounded Contexts and Context Maps: http://www.infoq.com/articles/ddd-contextmapping• Rinat Abdullings Getting Started with CQRS: http://abdullin.com/cqrs/• CQRS journey: https://github.com/mspnp/cqrs-journey-doc/• Greg Youngs old blog: http://codebetter.com/gregyoung/• Greg Youngs new blog: http://goodenoughsoftware.net• Greg Youngs CQRS info site: http://cqrs.wordpress.com• Sagas: http://www.udidahan.com/2009/04/20/saga-persistence-and-event-driven-architectures/• Clarified CQRS: http://www.udidahan.com/2009/12/09/clarified-cqrs/• When to avoid CQRS: http://www.udidahan.com/2011/04/22/when-to-avoid-cqrs/• Why you should be using CQRS almost everywhere: http://www.udidahan.com/2011/10/02/why- you-should-be-using-cqrs-almost-everywhere…/• Videos from DDD exchange 2011: http://skillsmatter.com/event/design-architecture/ddd-exchange- 2011• Fohjin DDD example: https://github.com/MarkNijhof/Fohjin/tree/master/Fohjin.DDD.Example• Axon Framework: http://www.axonframework.org/• Implementing Domain Driven Design (IDDD): http://my.safaribooksonline.com/book/project- management/9780133039900• CQRS Info: www.cqrs.nu• Greg Youngs EventStore: http://geteventstore.com