Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Complex event flows in distributed systems (QCon London 2019)

938 views

Published on

Slides from my talk at QCon London on 5th of March 2019. More information can be found here: https://berndruecker.io/complex-event-flows-in-distributed-systems/

Abstract: Event-driven architectures enable nicely decoupled microservices and are fundamental for decentral data management. However, using peer-to-peer event chains to implement complex end-to-end logic crossing service boundaries can accidentally increase coupling. Extracting such business logic into dedicated services reduces coupling and allows to keep sight of larger-scale flows - without violating bounded contexts, harming service autonomy or introducing god services. Service boundaries get clearer and service APIs get smarter by focusing on their potentially long running nature. I will demonstrate how the new generation of lightweight and highly-scalable state machines ease the implementation of long running services. Based on my real-life experiences, I will share how to handle complex logic and flows which require proper reactions on failures, timeouts and compensating actions and provide guidance backed by code examples to illustrate alternative approaches.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Complex event flows in distributed systems (QCon London 2019)

  1. 1. Complex event flows in distributed systems @berndruecker
  2. 2. 3 common hypotheses I check today: # Events decrease coupling # Orchestration needs to be avoided # Workflow engines are painful @berndruecker
  3. 3. Warning: Contains Opinion
  4. 4. Berlin, Germany mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda
  5. 5. Simplified example: dash button Photo by 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/ @berndruecker
  6. 6. Three steps… @berndruecker
  7. 7. Who is involved? Some bounded contexts… Checkout Payment Inventory Shipment @berndruecker
  8. 8. (Micro-)services Checkout Payment Inventory Shipment @berndruecker
  9. 9. Autonomous (micro-)services Checkout Payment Inventory Shipment Dedicated Application Processes Dedicated infrastructure Dedicated Development Teams @berndruecker
  10. 10. Events decrease coupling @berndruecker
  11. 11. Example Checkout Payment Inventory Shipment The button blinks if we can ship within 24 hours @berndruecker
  12. 12. Request/response: temporal coupling Checkout Payment Inventory Shipment Request Response The button blinks if we can ship within 24 hours @berndruecker
  13. 13. Temporal decoupling with events and read models Checkout Payment Inventory Shipment Good Stored Read Model Good Fetched The button blinks if we can ship within 24 hours *Events are facts about what happened (in the past) @berndruecker
  14. 14. Order Placed Payment Received Goods Fetched Notification Checkout Payment Inventory Shipment Event-driven architecture @berndruecker
  15. 15. Events can decrease coupling* *e.g. decentral data-management, read models, extract cross-cutting aspects @berndruecker
  16. 16. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched @berndruecker
  17. 17. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched @berndruecker
  18. 18. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  19. 19. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  20. 20. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  21. 21. Monitoring Workflows Across Microservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices @berndruecker
  22. 22. Typical approachesDistributed Tracing Data Lake / Event Monitoring Process Mining Process Tracking @berndruecker
  23. 23. Stefan Tilkov: Microservice Patterns & Antipatterns - MicroXchg 2018 @berndruecker
  24. 24. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment @berndruecker
  25. 25. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Goods fetched Order placed Payment received Goods shipped @berndruecker
  26. 26. Photo by born1945, available under Creative Commons BY 2.0 license. @berndruecker
  27. 27. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0 @berndruecker
  28. 28. „Challenges?“ Source: Microservices orchestration survey, July 2018, 354 responses https://camunda.com/microservices-orchestration-survey-results-2018/ @berndruecker
  29. 29. Order Extract the end-to-end responsibility Checkout Payment Inventory Shipment *Commands have an intent about what needs to happen in the future Payment received Order placed Retrieve payment @berndruecker
  30. 30. Order It is about where to decide about the coupling! Checkout Payment Inventory Shipment Order placed Retrieve payment Order decides . to listen to the event . to issue the command @berndruecker
  31. 31. Order It is about where to decide about the coupling! Checkout Payment Inventory Shipment Order placed Retrieve payment It can still be messaging! @berndruecker
  32. 32. Commands help to avoid (complex) peer-to-peer event chains @berndruecker
  33. 33. Orchestration needs to be avoided @berndruecker
  34. 34. Smart ESB-like middleware Checkout Payment Inventory Shipment Order Order placed Payment received Good fetched Good shipped @berndruecker
  35. 35. Dumb pipes Checkout Payment Inventory Shipment Order Smart endpoints and dumb pipes Martin Fowler @berndruecker
  36. 36. Danger of god services? Checkout Order A few smart god services tell anemic CRUD services what to do Sam Newmann Payment Inventory Shipment @berndruecker
  37. 37. Danger of god services? Checkout Payment Inventory Shipment Order A few smart god services tell anemic CRUD services what to do Sam Newmann @berndruecker
  38. 38. A god service is only created by bad API design! @berndruecker
  39. 39. Example Order Payment Retrieve Payment @berndruecker
  40. 40. Example Order Payment Credit Card Retrieve Payment @berndruecker
  41. 41. Example Order Payment Credit Card Retrieve Payment Rejected @berndruecker
  42. 42. Example Order Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker
  43. 43. Example Order Payment Client of dumb endpoints easily become a god services. If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker
  44. 44. Payment failed Who is responsible to deal with problems? Order Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received @berndruecker
  45. 45. Payment failed Long running services Order Payment Credit Card Retrieve Payment Rejected Payment received Smart endpoints are potentially long-running @berndruecker
  46. 46. Persist thing (Entity, Actor, …) State machine or workflow engine Typical concerns DIY = effort, accidental complexity Scheduling, Versioning, operating, visibility, scalability, … Handling State @berndruecker
  47. 47. Workflow engines are painful Complex, proprietary, heavyweight, central, developer adverse, … @berndruecker
  48. 48. Avoid the wrong tools! Death by properties panel Low-code is great! (You can get rid of your developers!) Complex, proprietary, heavyweight, central, developer adverse, … @berndruecker
  49. 49. Workflow engines, state machines It is relevant in modern architectures @berndruecker
  50. 50. CADENCE Silicon valley has recognized Workflow engines, state machines @berndruecker
  51. 51. CADENCE Workflow engines, state machines @berndruecker
  52. 52. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } What do I mean by „leightweight?“ @berndruecker
  53. 53. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Build engine in one line of code (using in- memory H2) @berndruecker
  54. 54. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL @berndruecker
  55. 55. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL @berndruecker
  56. 56. BPMN Business Process Model and Notation ISO Standard @berndruecker
  57. 57. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } We can attach code… @berndruecker
  58. 58. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } …that is called when workflow instances pass through @berndruecker
  59. 59. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Start instances @berndruecker
  60. 60. Payment Now you have a state machine! @berndruecker
  61. 61. Payment Easy to handle time @berndruecker
  62. 62. Distributed systems @berndruecker
  63. 63. Example with synchronous communication REST Order Payment Credit Card @berndruecker
  64. 64. Example with synchronous communication REST Order Payment Credit Card @berndruecker
  65. 65. Example with synchronous communication REST Order Payment Credit Card @berndruecker
  66. 66. Example with synchronous communication REST Order Payment Credit Card Stateful Retry @berndruecker
  67. 67. Works also for asynchronous communication Order Payment Credit Card Monitor Timeouts @berndruecker
  68. 68. Distributed systems @berndruecker
  69. 69. It is impossible to differentiate certain failure scenarios: Independant of communication style! Service Provider Client @berndruecker
  70. 70. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST @berndruecker
  71. 71. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST The service can be long running. You get a better API and less gods @berndruecker
  72. 72. Workflows live inside service boundaries @berndruecker
  73. 73. No BPM(N) monoliths https://blog.bernd-ruecker.com/avoiding-the-bpm-monolith-when-using-bounded-contexts-d86be6308d8 @berndruecker
  74. 74. Pat Helland “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  75. 75. Pat Helland Grown-Ups Don’t Use Distributed Transactions “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  76. 76. Distributed transactions using compensation * Compensation @berndruecker
  77. 77. Homework: Try to do this purely event-driven! Send to: mail@berndruecker.io @berndruecker
  78. 78. Biz Dev Leverage state machine & workflow engine Living documentation Visibility in testing improve communication improve communication Ops @berndruecker
  79. 79. Visual HTML reports for test cases @berndruecker
  80. 80. Living documentation for long-running behaviour @berndruecker
  81. 81. Proper Operations Visibility + Context @berndruecker
  82. 82. Biz Dev Leverage state machine & workflow engine Living documentation Visibility in testing Operate with visibility and context Understand and discuss business processes Evaluate optimizations in-sync with implementation improve communication improve communication Ops @berndruecker
  83. 83. Monitoring Workflows Across Microservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices @berndruecker
  84. 84. Tracking Checkout Inventory Payment Shipment Event Bus Workflow Engine https://www.confluent.io/kafka-summit-sf18/the_big_picture @berndruecker
  85. 85. Journey towards more orchestration @berndruecker
  86. 86. Vodafone, Liongate & WDW Presented at CamundaCOn Berlin 2018
  87. 87. Before mapping processes explicitly with BPMN, the truth was buried in the code and nobody knew what was going on. Jimmy Floyd, 24 Hour Fitnesse „ @berndruecker
  88. 88. … It addresses one of the core issues in a distributed microservices architecture—where is the source of truth for the coordinated interaction of the entire system? … the system we are replacing uses a complex peer- to-peer choreography that requires reasoning across multiple codebases to understand. https://medium.com/@sitapati/node-js-client-for-zeebe-microservices-orchestration-engine-72287e4c7d94 Josh Wulf Credit Sense @berndruecker
  89. 89. Lightweight workflow engines are great – don‘t DIY* *e.g. enabling potentially long-running services, solving hard developer problems, can run decentralized @berndruecker
  90. 90. Sales-Order & Order-Fulfillment via Camunda for every order worldwide (Q2 2017: 22,2 Mio) @berndruecker
  91. 91. Code, code, code… @berndruecker
  92. 92. Event-driven example InventoryPaymentOrder ShippingCheckout Monitor https://github.com/berndruecker/flowing-retail/ Human Tasks H2 H2 @berndruecker
  93. 93. # Events decrease coupling: sometimes read-models, but no complex peer-to-peer event chains! # Orchestration needs to be avoided: sometimes no ESB, smart endpoints/dumb pipes, balance orchestration and choreography # Workflow engines are painful: some of them lightweight engines are easy to use and can run decentralized, they solve hard developer problems, don‘t DIY @berndruecker
  94. 94. Thank you! @berndruecker
  95. 95. mail@berndruecker.io @berndruecker https://berndruecker.io https://medium.com/berndruecker https://github.com/berndruecker https://www.infoq.com/articles/events- workflow-automation Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/

×