Developing event-driven microservices with event sourcing and CQRS (Shanghai)

905 views

Published on

This is a talk I gave in Shanghai on July 4th 2016

In a microservices architecture, each service has its own database. While this ensures that services are loosely coupled it creates a problem: how do you maintain consistency across services without using 2PC? In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.

Published in: Software
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
905
On SlideShare
0
From Embeds
0
Number of Embeds
67
Actions
Shares
0
Downloads
37
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Developing event-driven microservices with event sourcing and CQRS (Shanghai)

  1. 1. @crichardson Event-driven microservices with event sourcing and CQRS Chris Richardson Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action @crichardson chris@chrisrichardson.net http://eventuate.io July 2016
  2. 2. @crichardson Presentation goal Show how Event Sourcing and Command Query Responsibility Segregation (CQRS) are a great way to implement microservices
  3. 3. @crichardson About Chris
  4. 4. @crichardson About Chris Consultant and trainer focusing on modern application architectures including microservices (http://www.chrisrichardson.net/)
  5. 5. @crichardson About Chris Founder of a startup that is creating a platform that makes it easier for developers to write transactional microservices (http://eventuate.io)
  6. 6. @crichardson For more information http://learnmicroservices.io
  7. 7. @crichardson Agenda Monolith vs. microservices Event-driven microservices Overview of event sourcing Implementing queries in an event sourced application
  8. 8. @crichardson Let’s imagine you are building a large, complex application, e.g. an online store
  9. 9. @crichardson Successful software development Architecture Process Organization Agile Continuous delivery … Small, autonomous, teams
  10. 10. @crichardson ?Architecture
  11. 11. @crichardson
  12. 12. @crichardson The monolithic architecture Tomcat Browser WAR SQL database HTML REST/JSON Client App Simple to …. Develop Test Deploy Scale Catalog Module Reviews Module Orders Module StoreFront UI Module
  13. 13. @crichardson But successful applications keep growing ….
  14. 14. @crichardson Monolithic architecture Process Organization Agile Continuous delivery … Small, autonomous, teams
  15. 15. @crichardson Apply functional decomposition X axis - horizontal duplication Z axis -data partitioning Y axis - functional decomposition Scale by splitting sim ilar things Scale by splitting different things
  16. 16. @crichardson Microservice architecture Browser Mobile Device Store Front UI API Gateway Catalog Service Review Service Order Service … Service Catalog Database Review Database Order Database … Database HTML REST REST REST
  17. 17. @crichardson Microservice architecture Process Organization Agile Continuous delivery … Small, autonomous, teams✔ ✔
  18. 18. @crichardson Drawbacks Complexity
  19. 19. @crichardson Drawbacks Complexity of developing a distributed system Implementing inter-process communication Handling partial failures Complexity of implementing business transactions that span multiple databases (without 2PC) Complexity of testing a distributed system Complexity of deploying and operating a distributed system Managing the development and deployment of features that span multiple services Fortunately solutions exists
  20. 20. @crichardson The benefits typically outweigh the drawbacks for large, complex applications
  21. 21. @crichardson Issues to address How to deploy the services? How do the services communicate? How do clients of the application communicate with the services? How to partition the system into services? How to deal with distributed data management problems? ….
  22. 22. @crichardson Microservices pattern language http://microservices.io/
  23. 23. @crichardson Agenda Monolith vs. microservices Event-driven microservices Overview of event sourcing Implementing queries in an event sourced application
  24. 24. Data management patterns
  25. 25. @crichardson The Database Shared database Order Service Customer Service … Service Order table Customer table … orderTotal creditLimit Tight coupling Simple and ACID
  26. 26. @crichardson Database per service Order Service Customer Service Order Database Customer Database Order table Customer table orderTotal creditLimit Loose coupling 😀 but more complex 😓 and….
  27. 27. @crichardson 2PC (aka. distributed transactions) is not viable choice for most modern applications
  28. 28. @crichardson Customer management How to maintain data consistency without 2PC? Order management Order Service placeOrder() Customer Service updateCreditLimit() Customer creditLimit ... has ordersbelongs toOrder total Invariant: sum(open order.total) <= customer.creditLimit ?
  29. 29. @crichardson Event-driven architecture
  30. 30. @crichardson Use event-driven, eventually consistent order processing Order Service Customer Service Order created Credit Reserved Credit Check Failed Place Order OR
  31. 31. @crichardson How atomically update database and publish an event Order Service Order Database Message Broker insert Order publish OrderCreatedEvent dual write problem ?
  32. 32. @crichardson Failure = inconsistent system Order Service Order Database Message Broker insert Order publish OrderCreatedEvent X
  33. 33. @crichardson 2PC is not an option
  34. 34. @crichardson Reliably publish events when state changes
  35. 35. @crichardson Agenda Monolith vs. microservices Event-driven microservices Overview of event sourcing Implementing queries in an event sourced application
  36. 36. @crichardson Just publish events Application Database Message Broker update publish X
  37. 37. @crichardson Event sourcing For each domain object (i.e. DDD aggregate): Identify (state changing) domain events, e.g. use Event Storming Define Event classes For example, ShoppingCart: ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent Order: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
  38. 38. @crichardson Persists events NOT current state Order status …. 101 ACCEPTED Order table X
  39. 39. @crichardson Persists events NOT current state Event table Entity type Event id Entity id Event data Order 902101 …OrderApproved Order 903101 …OrderShipped Event type Order 901101 …OrderCreated
  40. 40. @crichardson Replay events to recreate state Order state OrderCreated(…) OrderAccepted(…) OrderShipped(…) Events Periodically snapshot to avoid loading all events
  41. 41. @crichardson The present is a fold over history currentState = foldl(applyEvent, initialState, events)
  42. 42. @crichardson Familiar concepts restructured class Customer { public void reserveCredit( orderId : String, amount : Money) { // verify // update state this.xyz = … } public List<Event> process( ReserveCreditCommand cmd) { // verify … return singletonList( new CreditReservedEvent(…); ) } public void apply( CreditReservedEvent event) { // update state this.xyz = event.xyz }
  43. 43. @crichardson Event Aggregates: Command Events AggregateCommand Event
  44. 44. @crichardson Aggregates: Event Updated aggregate AggregateEvent Updated Aggregate
  45. 45. @crichardson Request handling in an event sourced application HTTP Handler Event Store pastEvents = findEvents(entityId) Order new() applyEvents(pastEvents) newEvents = processCmd(someCmd) saveEvents(newEvents) (optimistic locking) Order Service applyEvents(newEvents)
  46. 46. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) Customer update() Customer Service
  47. 47. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) CQRS View update() Service Xyz send notifications …
  48. 48. Event store = database + message broker Hybrid database and message broker Implementations: Home grown/DIY geteventstore.com by Greg Young http://eventuate.io (mine) Event Store Save aggregate events Get aggregate events Subscribe to events
  49. 49. @crichardson Benefits of event sourcing Solves data consistency issues in a Microservice/NoSQL based architecture Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,… Eliminates O/R mapping problem (mostly) Reifies state changes: Built in, reliable audit log temporal queries Preserved history More easily implement future requirements
  50. 50. @crichardson Drawbacks of event sourcing Requires application rewrite Weird and unfamiliar style of programming Events = a historical record of your bad design decisions Must detect and ignore duplicate events Idempotent event handlers Track most recent event and ignore older ones … Querying the event store can be challenging Some queries might be complex/inefficient, e.g. accounts with a balance > X Event store might only support lookup of events by entity id Must use Command Query Responsibility Segregation (CQRS) to handle queries application must handle eventually consistent data
  51. 51. @crichardson Agenda Monolith vs. microservices Event-driven microservices Overview of event sourcing Implementing queries in an event sourced application
  52. 52. @crichardson Find recent, valuable customers SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ? Customer Service Order Service What if event sourcing is used?…. is no longer easy
  53. 53. @crichardson Use Command Query Responsibility Segregation (CQRS)
  54. 54. @crichardson Command Query Responsibility Segregation (CQRS) Application logic Commands Queries X
  55. 55. @crichardson Command Query Responsibility Segregation (CQRS) Command side Commands Aggregate Event Store Events Query side Queries (Materialized) View Events
  56. 56. @crichardson Query side design Event Store Updater View Updater Service Events Reader HTTP GET Request View Query Service View Store e.g. MongoDB ElasticSearch Neo4J update query
  57. 57. @crichardson Persisting a customer and order history in MongoDB { "_id" : "0000014f9a45004b 0a00270000000000", "_class" : "net.chrisrichardson…..views.orderhistory.CustomerView", "version" : NumberLong(5), "orders" : { "0000014f9a450063 0a00270000000000" : { "state" : "APPROVED", "orderId" : "0000014f9a450063 0a00270000000000", "orderTotal" : { "amount" : "1234" } }, "0000014f9a450063 0a00270000000001" : { "state" : "REJECTED", "orderId" : "0000014f9a450063 0a00270000000001", "orderTotal" : { "amount" : "3000" } } }, "name" : "Fred", "creditLimit" : { "amount" : "2000" } } Denormalized = efficient lookup
  58. 58. @crichardson Persisting customers and order info using Spring Data for MongoDB...
  59. 59. @crichardson Persisting customers and order using Spring Data for MongoDB...
  60. 60. Benefits and drawbacks of CQRS Benefits Necessary in an event sourced architecture Separation of concerns = simpler command and query models Supports multiple denormalized views Improved scalability and performance Drawbacks Complexity Potential code duplication Replication lag/eventually consistent views
  61. 61. @crichardson Summary Use microservices to accelerate development Use an event-driven architecture to maintain data consistency Implement an event-driven architecture using event sourcing Use CQRS to implement materialized views for queries
  62. 62. @crichardson @crichardson chris@chrisrichardson.net http://learnmicroservices.io Questions?

×