Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Similar to Intro to event sourcing and CQRS
Similar to Intro to event sourcing and CQRS (20)
Recently uploaded
Recently uploaded (20)
Intro to event sourcing and CQRS
- 1. EVENT SOURCING, CQRS, AND EVENT DRIVEN SYSTEMS EVENT SOURCING AND CQRS: A PATTERN FOR MICROSERVICES
- 2. WHO AM I? • Savvas Kleanthous Head of Engineering @ ParcelVision Ltd • Experience with DDD & ES & CQRS • Twitter account: @skleanthous
- 3. INTRO TO EVENT SOURCING INTRO TO EVENT SOURCING AND COMPARISON TO OTHER STORAGE FORMS
- 4. WHAT IS EVENT SOURCING * Only simple to explain - a lot of hidden complexity in practice!
- 5. BANK ACCOUNT EXAMPLE • Your bank account • Classic example • “Accountants don’t use pencils”
- 6. BATTLE OF THE STORE MECHANISMS USING A NON-CONTRIVED EXAMPLE
- 7. RELATIONAL STORE NORMAL FORM • The shipment is our entity • Low duplication • Joins are necessary
- 8. DOCUMENT STORE • Loading means to deserialize • Load by Id better performance • Context is lost since document form is a static model • Complex versioning due to implicit schema
- 9. EVENT SOURCE • Business logic creates contextful domain events instead of editing state • Loaded by id • Cannot serve complex queries directly
- 10. INTRO TO CQRS WHAT IS CQRS, AND ITS VARIATIONS
- 11. CQRS – WHAT IS IT AND WHY • Command Query Responsibility Segregation • Split paths for write and read (possibly separate stores) • Often needed in order to support different use cases
- 14. WHAT ARE THEY? • (Distributed) systems that communicate through events • Reactive systems • Easy to scale • Lead to loosely coupled services
- 15. CQRSES a very good option for implementing event driven systems
- 16. BENEFITS OF CQRSES REASONS THAT SHOULD MAKE YOU CONSIDER CQRSES FOR YOUR NEXT SOLUTION
- 17. 1. You have a built-in audit log •The ultimate source of truth
- 18. 2. Domain events speak the language of your SME’s •Analysis of domain events leads to breakthroughs
- 19. 3. Much easier deploymentrollback •No downtime deployments
- 20. 4. Lowest possible latencies
- 21. 5. Separation of responsibilities between readwrite models •Independent evolution of contract and domain
- 22. 6. Temporal and Bitemporal queries not only feasible, but actually easy too
- 23. 7. Debugging live is very easy •Easy as taking a snapshot and pushing command in
- 24. And many more
- 25. DISADVANTAGES OF CQRSES WHAT MAY BE THE PROBLEMS THAT YOU MAY BE FACING?
- 26. 1. Unfamiliar way of thinking
- 27. 2. Increased accidental complexity
- 28. 3. In async the best consistency you can get: •In theory eventual serializability •In practice causal
- 29. 4. No 2PC in async CQRS or across aggregate boundaries
- 30. LESSONS FROM THE TRENCHES SOME FIRST TIPS WHEN YOU START ON YOUR JOURNEY
- 31. 1. INVOLVE YOUR SME’S EARLY AND A LOT • Do event storming • Let the SME’s do your work for you • If you don’t, you will still learn the domain on your own but at a great cost
- 32. 2. MAKE EVENTS SPECIFIC AND CONTEXTFUL • Rule of thumb: Fat enough so that consumers are consistent, but not any more than that. • Events should be specific about what happened (contextful) • Events that are too fat or too thin may lead to duplication of logic
- 33. 3. EVENT VERSIONING 1. When possible, you should rely on weak (de)serializationdynamic nature of language ! Explicitly version your event regardless 2. Upconvert event inline when loading 3. Final option: event migration (although a nuclear one)
- 34. 4. ADD METADATA TO YOUR MESSAGES • Message id • Correlation id • Causation id • Type – prefer a property even in OO • Version
- 35. 5. CQRS: START WITH SYNC WRITE AND READ MODEL UPDATES • Still decouple writes and reads by splitting the two paths • This limits eventual consistency issues significantly – causal consistency “built-in”
- 36. 6. DISTINGUISH BETWEEN DOMAIN AND PUBLIC EVENTS DOMAIN • Used for storage and domain communication • Meaningful internally to domain • Needs to be fat enough to guarantee consistency in domain listeners PUBLIC • Used to inform public of a change • Meaningful as a contract – Can change independently of changes in domain • Needs to be fat enough to guarantee consistency of external listeners
- 37. 7. YOU DON’T NEED A CQRS OR ES FRAMEWORK • Current state is just a left-fold of events • Just make sure you don’t have side-effects on load – only change entity state when replaying events
- 38. 8. DON’T BE AFRAID OF MULTIPLE DATA STORES • For example: • ES domain entities • Mongo Db for view models • SQL db for BI and reporting
- 39. QA I’LL BE AVAILABLE FOR A CHAT AFTERWARDS. PLEASE APPROACH ME AND LETS HAVE A CHAT.
- 40. COMMON QUESTIONSCOMMENTS • “Unique emails for users in CQRS” – most asked question ever • “How do you delete” – “How do you fix an error in an event” • “Event sourcing is complex” • “Testing is more complicated when doing ES””How do you test” • “Event sourcing is slow” • “Data must be huge”
- 41. THANK YOU! Twitter account: @SKleanthous
Editor's Notes
- Still the easier way to explain. Open: set name, no initial value Deposit: add 5 Deposit add 10 AmountFromTransferReceived add 5 AmountWithdrawn subtract 15 ==> have 5
- And I haven’t event touched on legs.
- When doing a change, all context is lost. All you have is THIS IS THE STATE. Problem requirement: Find all shipments that are to be delivered to Manchester
- No info lost: From instrumentation around processing processing time = event timestamp – command timestamp Just reading names of event you can see what happened you can react on changes
- There are many MANY ways to implement CQRS. Log shipping from db to another system that processes it asynchronously to create another projections _is_ a form of CQRS. Different use cases needs of customers\users like need to query by destination with autocomplete of values Equivalent of CQS (method can either change state and return void or return value and not mutate object) on a larger scale Business logic\model and read projection generation separated Very simple architectural pattern to explain but it has hidden complexity Hitting read store should have no side effect
- Loosely coupled because of: reference autonomy, time autonomy Scaling: sharding based on events Inherently eventually consistent Reactive by nature “Tell don’t ask”
- Very high cohesion with ES and DDD Conceptually event driven systems are an extension of ES principles to systems Conceptual extension => In ES domain events encapsulate immutable facts and aggregates\observers react to it Events are central and highly important in both cases and all design and function is around them
- No information is lost ever A lot of info can also be picked up from metadata Causation analysis Correlation across external stimuli Security trail (who done it)
- You can easily validate your understanding of the domain by validating the events with your SME’s SME’s can be utilized to do analysis for you :) Event analysis often leads to breakthroughs Immutability of events are sometimes forced by domain Immutability of events are often in understanding of SME
- Separate read models and different db’s allows you to choose whichever db provides the best experience at the cost of eventual consistency
- Loosely coupled services Model-purpose mismatch
- Bitemporal queries answer: - Events that apply after a certain date. This is built-in! What was the state of X a month ago (with correction events) What did we know about X a month ago (without correction events)
- ES often implicitly required in some industries (usually for auditing purposes) Very low coupling of deployed services High synergy with stream processing. Same logic\process
- Mental shift is needed to start thinking of persistence in events Mental shift is needed to start considering split models, potentially multiple models for reading Both of these are a journey with multiple steps, but eventually surpass-able
- Read your own writes is doable but painful
- (you can cheat in some event stores, but that is violating the transactional guarantee of the aggregates)
- Event storming from Alberto Brandolini SME’s will do your work for you. SME’s can help you with scope of your aggregates. If events make sense for them they can tell you the info they need to contain If events don’t make sense to SME’s you’re potentially adding artificial complexity to your system. Also, you may be exposing implementation details to outside consumers increasing coupling Wrongly scoped events can lead to big problems. Wrong aggregate boundaries can lead to big problems. If SME’s aren’t directly involved at least make sure that events make sense to SME’s and\or Product owner If SME’s aren’t involved you end up with wrongly sized events and wrong aggregate boundaries. Both are huge technical debts that you need to avoid
- Fatter events duplication of business logic to discover what happened and react on it Thinner events duplication of state machine to discover when a certain transition actually happened When events are smaller they are far easier to maintain and version in the future When they are contectful it is easy to know how to apply them. Additional information is passed through the name Also think of this: if it makes sense for the domain expert, your domain model matches their mental model and this alignment is extremely valuable. Their type should be meaningful to the relevant domain expert or SME Prefer more granular events over “god” events Do not go overboard with granularity – consult your SME
- Additive changes to your events Load an event with it’s version, Translate it manually to new version and then apply it to entity Same as 2, but for all events in the stream before entity is loaded\hydrated. Usually done during deployments
- Message id is nothing special just an id that uniquely identifies message Correlation id gets assigned once and then any and all messages get their correlation id assigned from the message that caused it Causation id gets assigned from message id of previous message that caused it. Can trace everything that happened in the system Type: what command or event is this? PaymentMade, TransferAmount, UpdateFromNewScans etc. Version of current message-> useful during deserialization
- This is simpler to understand, debug and… … easier to start with and easy to migrate to async once needed If you have different stores for read & write eventual consistency will be there anyway but possibly very limited since read stores can be updated before publishing of event happens Since read stores get updated before event is published then you have causal consistency for sure Easy to understand since it’s a synchronous model without causal consistency or ordering issues. Decouple writes and reads so that you can scale reads independently and you have an easier path to asynd cqrs if you find out you need it further down the line
- This is mostly important for communication and service versioning. Consuming domain events in other microservices may result in coupling to another services internal implementations (since it’s used for persistence) Domain events should be fat they contain all necessary info to change the aggregate state\be applied but also to update read model without the need to make remote calls Public events can be thinner if you’re concerned with versioning, but can be fat to achieve decoupling Idempotency important in both cases, maybe slightly more so in domain case to make denormalizers easier
- Projections can be as simple as mapping of events You can store events in any db Storing events in: SQL server: Id column should be Aggregate Id + Version (count of events) Document store: A document with a collection of events Event store -> easier option Kafka -> may not be the best default, but if it is there and works for your needs use it When loading only apply on state! You don’t want to re-send emails etc whenever you load entity
- It is fine to use any combination of SQL stores, document stores, graph dbs, in-memory caches etc. ALWAYS listen to your Domain experts and requirements. Meet the requirement first Almost always, a couple of ms of inconsistency between read-write models is perfectly fine Queries may need to be fast -> use document stores to load full view model by id Complex queries may be better suited with Graph db’s. Populate them asynchronously through events: still fast to process, querying won’t impact rest of the system SQL stores can have complex indexes useful for BI In memory caches also a good option to decrease latency
- Unique XXX -> Do you need immediate consistency? If yes: you don’t need to ES everything! Use SQL or whatever makes sense for your requirements. Also: you can synchronously hit the ES if you need\makes sense. Deleting is more convoluted. Use “reset events” that rest the state to start. Update – use correction event. You mustn’t really edit the stream directly. Event migration is a safer option Event sourcing is not more complex. CQRS may introduce a complexity but you could not use CQRS. Also it is a tradeoff (easier migrations etc) Testing is very simple: command in, events out. Events check both behaviour and persistence It is very fast for small number of events. Definitely faster than SQL with joins (so always) or graph dbs. As events grow you could use snapshotting Huge data? How many Gb per second are we speaking about here?