Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QCon 2019 - Opportunities and Pitfalls of Event-Driven Utopia

558 views

Published on

Slides from my talk at QCon NYC on 25th of June 2019 in the Microservices and Serverless track

Published in: Technology
  • Be the first to comment

  • Be the first to like this

QCon 2019 - Opportunities and Pitfalls of Event-Driven Utopia

  1. 1. Opportunities & Pitfalls of Event-Driven Utopia @berndruecker
  2. 2. Why this talk
  3. 3. Why this talk
  4. 4. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3
  5. 5. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3 Events inside out2
  6. 6. Service 1 Agenda Service 2 Events on the inside1 Events on the outside3 Events inside out2
  7. 7. Once upon a time… RDMS Application Client BBC architecture (box - arrow – box – arrow - cylinder) Every architecture diagram you'll ever need
  8. 8. The great thing about this architecture RDMS Application DB gurantees (e.g. ACID)
  9. 9. The problem not webscale resiliency is expensive RDMS Application RDMS
  10. 10. Pat Helland https://vimeo.com/52831373
  11. 11. Append-only Log … bank account created +2,500 $ transfered -14.99$ paid by credit card … Persistent change RDMS Account # Balance 12345 2,500$ Persistent state Current Balance = 2,485.01 $
  12. 12. Append-only Log … bank account created +2,500 $ transfered -14.99$ paid by credit card … Persistent change Event Bank Account Created 2019/04/16 11:00 # 12345 Event Money Transfer Received 2,500$ 2019/04/16 11:00 # 12345
  13. 13. Event Sourcing in a nutshell Customer Business Logic Save event 2. Read events Replay … Internal State 3. Build internal state 5. Add Customer Created Event 1. Create Customer 4. Validation and Invariant Checks e.g. Customer Created, Customer Credit Limit Approved, …Customer Event Store 6. Async publish Customer Created Domain Event
  14. 14. Working without distributed transactions Customer Business Logic Save event 2. Read events Replay … Internal State 3. Build internal state 5. Add Customer Created Event 1. Create Customer 4. Validation and Invariant Checks e.g. Customer Created, Customer Credit Limit Approved, …Customer Event Store 6. Async publish Customer Created Domain Event This is the only atomic operation required
  15. 15. Traditional Architecture Customer 1. Create Customer Account RDMS Business Logic Open Account 2. Persist state changes 3. Remote Communication
  16. 16. Pat Helland “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  17. 17. Pat Helland Grown-Ups Don’t Use Distributed Transactions “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
  18. 18. Open Account Outbox pattern in traditional architectures Customer 1. Create Customer Account RDMS Business Logic Job: „Open Account“ 2. Persist state changes 3. Remote CommunicationExecute: „Open Account“ TX 1 TX 2 Async after first transaction!
  19. 19. Outbox pattern – Implementation Approaches Scheduler Database Transaction Log Workflow Automation
  20. 20. Idempotency Customer 1. Create Customer Account RDMS Business Logic Job: „Open Account“ 2. Persist state changes 3. Remote Communication Async after transaction! Execute: „Open Account“ Capture Request TX 1 TX 2
  21. 21. Idempotency Customer 1. Create Customer Account RDMS Business Logic Job: „Open Account“ 2. Persist state changes 3. Remote Communication Async after transaction! Execute: „Open Account“ Capture Request TX 1 TX 3TX 2
  22. 22. Working without distributed transactions Customer Business Logic Save event 2. Read events Replay … Internal State 3. Build internal state 5. Add Customer Created Event 1. Create Customer 4. Validation and Invariant Checks e.g. Customer Created, Customer Credit Limit Approved, …Customer Event Store 6. Async publish Customer Created Domain Event This is the only atomic operation required
  23. 23. Events on the inside. An example from my world
  24. 24. mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda
  25. 25. We offer two different workflow engines. Why? Camunda ZeebePersistent State Persistent change
  26. 26. Workflow Instance Id Current Activity State 2 RetrievePayment running Workflow Instance Id Current Activity State 2 ShipGoods running Workflow Instance Id Current Activity State 2 OrderDelivered ended 2. 3. UPDATE 2. UPDATE 1. INSERT 3.1. RDMS Workflow Engine
  27. 27. Append-only Log1. create workflow instance workflow instance created start event occured sequence flow taken activity activated task created lock created task locked complete task task completed activity completed sequence flow taken … … 2. 2. 1. Workflow Engine
  28. 28. Event Handling, Replication & Single Writer Follower Follower complete task command task completed event 1 send 2 append command Broker (Leader) Stream Processor 4 process 7 store & replicate event 6 append event 3 store & replicate command 5 respond Single Writer (single thread)
  29. 29. What we do different Follower Follower complete task command task completed event 1 send 2 append command Leader Stream Processor 4 process 7 store & replicate event 6 append event 3 store & replicate command 5 respond Single Writer (single thread) Store and replay commands Delete records that are fully processed Persist & replicate internal state
  30. 30. Consistency Availability Partition
  31. 31. Zeebe is CP Follower Follower complete task command task completed event 1 send 2 append command Leader Stream Processor 4 process 7 store & replicate event 6 append event 3 store & replicate command 5 respond Single Writer (single thread)
  32. 32. Horizontal scalability by partitioning Partition 1 Partition 2 Partition 3 Partition 4 Every workflow instance is exactly handled by one partition instance id: 2-42 instance id: 3-66 Stream Processor Single Writer (single thread)
  33. 33. Queries and read models Zeebe Broker Zeebe Broker Streaming Exporter ask ask
  34. 34. Recap 1 – Events on the inside # Natural mechanism to build scalable services in distributed systems (with Outbox & co included) But # You have to think about reads, queries & eventual consistency # Much less industry experience available @berndruecker
  35. 35. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3 Events inside out2
  36. 36. Event Store and Messaging Customer … 1. Create Customer Customer Event Store
  37. 37. Merge Messaging and Event Store Customer … 1. Create Customer Customer Event Store
  38. 38. Merge messaging and event store Customer … 1. Create Customer Shared Event Store
  39. 39. Enter the world of Kafka…
  40. 40. Merge messaging and event store Customer … 1. Create Customer Shared Event Store
  41. 41. Kafka as transport Customer … 1. Create Customer Used as queue (but persistent!)
  42. 42. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3 Events inside out2
  43. 43. Once upon a time Billing Customer Change Address
  44. 44. Event Notification Address changed Billing Customer
  45. 45. Event Notification Address changed Billing Customer Billing Customer Reverse direction of dependency direction of dependency Change Address
  46. 46. Event Notification Address changed Billing Customer AdressChanged { customerId: 42 } Ask for details
  47. 47. Event-carried State Transfer Address changed Billing Customer AddressChanged { customerId: 42, address: ... } CustomerChanged { customerId: 42, status: A, address: ..., } AddressChanged { customerId: 42, oldAddress: ... newAddress: ... } CustomerMoved { ...,
  48. 48. This decision is complex Address changed Billing Customer Billing Customer Reverse direction of dependency direction of dependency Change Address
  49. 49. Example Change Address Address Submit From bla@company.com Date 2019-04-23 09.05 To confirm your address change please click on this link: http://company.com/confirm?id=82e97d49-166c-4862- 9973-4db348e6225d Incoming Email
  50. 50. Example Customer Notification Address change confirmed Change Adress Address change requested http://company.com/confirm?id=82e97 d49-166c-4862-9973-4db348e6225d direction of dependency
  51. 51. Example Customer Notification ‚Confirmation‘ approved Change Adress http://company.com/confirm?id=82e97 d49-166c-4862-9973-4db348e6225d Send mail ‚Confirmation‘ Address changed direction of dependency
  52. 52. Challenge: Command vs. Event Event Command vs
  53. 53. It is NOTabout communication protocols Address changed Billing Customer Billing Customer Change Address It can be messaging, REST, whatever, ….
  54. 54. Manifold ways of transport …
  55. 55. Manifold ways of transport …
  56. 56. ? Event Command Query Message Record Event Fact, happened in the past, immutable Intend, Want s.th. to happen, The intention itself is a fact
  57. 57. ? Event Command Query Message Record Event
  58. 58. Commands in disguise The Customer Needs To Be Sent A Message To Confirm Address Change Event Send Message Wording of Sender Wording of recipient
  59. 59. Examples PaymentOrder Subscription Retrieve Payment More general, does not need to know who is retrieving payments Customer Order Address Changed Billing More general, does not need to care about who is interessted in address changes Notification Send Mail Global service Order Notification Order Placed Payment Received Goods Shipped Service that can handle notifications for orders autonomously
  60. 60. Distributed Monoliths Authorization Service Document Context Page Context Document attached Page created Document moved Page moved …
  61. 61. Define stable contract/API instead Authorization Service Document Context Page Context Add auth …
  62. 62. Next challenge: Event chains
  63. 63. Event Chains Adress Check Credit Check Registration @berndruecker Customer Event Bus Registration requested Credit checked Address checked Customer registered
  64. 64. Event Chains Adress Check Credit Check Registration @berndruecker Customer Event Bus Registration requested Credit checked Address checked Customer registered How does customer registration work?
  65. 65. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  66. 66. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  67. 67. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  68. 68. Monitoring Workflows Across Microservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices @berndruecker
  69. 69. Typical approachesDistributed Tracing Data Lake / Event Monitoring Process Mining Process Tracking @berndruecker
  70. 70. What we currently build with customers… Camunda Optimize Elastic Registration requested Credit checked Address checked Customer registered @berndruecker
  71. 71. All great – until you have to move…
  72. 72. Changes required for an additional check Adress Check Credit Check Registration Criminal Check @berndruecker Customer Event Bus Registration requested Credit checked Customer registered Address checked
  73. 73. Changes required for an additional check Adress Check Credit Check Registration Criminal Check @berndruecker Customer Event Bus Registration requested Credit checked Customer registered Address checked Criminal checked
  74. 74. Alternative flow Adress Check Credit Check Registration Criminal Check @berndruecker Customer Kafka Customer registered Registration requested Address checked Credit checked Criminal checked
  75. 75. Alternative flow Adress Check Credit Check Registration Criminal Check @berndruecker Customer Kafka Customer registered Registration requested Address checked Credit checked Criminal checked „Credit checks got more expensive, do that only if all other checks succeed“
  76. 76. Keep it stable, just move sticks with yellow color to the top. How hard can it be?
  77. 77. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0 @berndruecker
  78. 78. Orchestration Adress CheckCredit Check Registration Customer Kafka Registration requested Credit checked Address checked Customer registeredCheck credit Check address Customer checked Customer On-boarding Of course these two services could be merged
  79. 79. Changes Adress CheckCredit Check Registration Customer Kafka Registration requested Credit checked Address checked Customer registeredCheck credit Check address Customer checked Customer On-boarding Criminal Check Crimes checked Check crimes
  80. 80. Comparison 2 changes criminal check can be deployed first 2 changes, criminal check can be deployed first See also https://www.infoworld.com/article/3391592/ how-to-tame-event-driven-microservices.html
  81. 81. In my world… Customer On-boarding Leverage Workflow Engine & BPMN within Service
  82. 82. Customer On-boarding Local Orchestration Central Orchestration Service
  83. 83. Recap 2 # Commands vs. Events: Decide about the direction of dependencies # Beware of event-chains and avoid losing sight # Balance choreography and orchestration @berndruecker
  84. 84. Service 2Service 1 Recap Events on the inside Events on the outside 1 3 Persistent state vs persistent change Event sourcing & Event Store Consistency & CAP Read Models & CQRS Events as API Event vs Command Event chains & visibility Orchestration vs Choreography Shared Event Store Events inside out2
  85. 85. Want to see code?
  86. 86. Nothing for the faint of heart… Events on the inside Events on the outside
  87. 87. Nothing for the faint of heart… …but doable… …and worth it
  88. 88. Thank you! @berndruecker
  89. 89. mail@berndruecker.io @berndruecker https://berndruecker.io https://medium.com/berndruecker https://github.com/berndruecker https://www.infoq.com/articles/events- workflow-automation Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/

×