Complex event flows in distributed systems

1. Complex event flows in distributed systems @berndruecker With thoughts from http://flowing.io @berndruecker | @martinschimak

2. 3 common hypotheses I check today: # Events decrease coupling # Orchestration needs to be avoided # Workflow engines are painful

3. Berlin, Germany bernd.ruecker@camunda.com @berndruecker Bernd Ruecker Co-founder and Developer Advocate of Camunda

4. Simplified example: dash button Photo by 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/

5. Three steps…

6. Who is involved? Some bounded contexts… Checkout Payment Inventory Shipment

7. (Micro-)services Checkout Payment Inventory Shipment

8. Autonomous (micro-)services Checkout Payment Inventory Shipment Dedicated Application Processes Dedicated infrastructure Dedicated Development Teams

9. Events decrease coupling

10. Example Checkout Payment Inventory Shipment The button blinks if we can ship within 24 hours

11. Request/response: temporal coupling Checkout Payment Inventory Shipment Request Response The button blinks if we can ship within 24 hours

12. Temporal decoupling with events and read models Checkout Payment Inventory Shipment Good Stored Read Model Good Fetched The button blinks if we can ship within 24 hours *Events are facts about what happened (in the past)

13. Events can decrease coupling* *e.g. decentral data-management, read models, extract cross-cutting aspects

14. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched

15. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched

16. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html

19. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment

20. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment Customers can pay via invoice …

21. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment

22. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Goods fetched Order placed Payment received Goods shipped

23. Photo by born1945, available under Creative Commons BY 2.0 license.

24. Order Extract the end-to-end responsibility Checkout Payment Inventory Shipment *Commands have an intent about what needs to happen in the future Order placed Payment received Retrieve payment

25. Commands help to avoid (complex) peer-to-peer event chains

26. Orchestration needs to be avoided

27. Smart ESB-like middleware Checkout Payment Inventory Shipment Order Order placed Payment received Good fetched Good shipped

28. Dumb pipes Checkout Payment Inventory Shipment Order Smart endpoints and dumb pipes Martin Fowler

29. Danger of god services? Checkout Order A few smart god services tell anemic CRUD services what to do Sam Newmann Payment Inventory Shipment

30. Danger of god services? Checkout Payment Inventory Shipment Order A few smart god services tell anemic CRUD services what to do Sam Newmann

31. A god service is only created by bad API design!

32. Example Order Payment Retrieve Payment

33. Example Order Payment Credit Card Retrieve Payment

34. Example Order Payment Credit Card Retrieve Payment Rejected

35. Example Order Payment Client of dumb endpoints easily become a god services. If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected

36. Payment failed Who is responsible to deal with problems? Order Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received

37. Payment failed Long-running execution Order Payment Clients of smart endpoints remains lean. If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received Smart endpoints are potentially long-running

38. Persist thing (Entity, Document, Actor, …) State machine or workflow engine Typical concerns DIY = effort, accidental complexity Scheduling, Versioning, operating, visibility, scalability, … Handling State

39. Workflow engines are painful Complex, proprietary, heavyweight, central, developer adverse, …

40. Avoid the wrong tools! Death by properties panel Low-code is great! (You can get rid of your developers!) Complex, proprietary, heavyweight, central, developer adverse, …

41. Workflow engines, state machines It is relevant in modern architectures

42. CADENCE Silicon valley has recognized Workflow engines, state machines

43. CADENCE Workflow engines, state machines

44. CADENCE also at scale Workflow engines, state machines

45. Zeebe.io

46. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } What do I mean by „leightweight?“

47. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Build engine in one line of code (using in- memory H2)

48. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL

49. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL

50. BPMN Business Process Model and Notation ISO Standard

51. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } We can attach code…

52. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } …that is called when workflow instances pass through

53. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Start instances

54. Payment Now you have a state machine!

55. Payment Easy to handle time

56. martin.schimak@plexiti.com @martinschimak

57. Synchronous communication REST Order Payment Credit Card

60. Synchronous communication REST Order Payment Credit Card Stateful Retry

61. Distributed systems

62. It is impossible to differentiate certain failure scenarios. Independant of communication style! Service Provider Client

63. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST

64. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST

65. Distributed systems

66. Distributed transactions using compensation * Compensation

67. Relaxed consistency Temporarily inconsistent state But eventually consistent No Isolation (as in ACID)

68. https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/

69. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Orchestration Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation

70. Biz Dev improve communication improve communication Ops

71. Biz Dev Leverage state machine & workflow engine Living documentation Visibility in testing improve communication improve communication Ops

72. Visual HTML reports for test cases

73. Biz Dev Leverage state machine & workflow engine Living documentation Visibility in testing Understand and discuss business processes Evaluate optimizations in-sync with implementation improve communication improve communication Ops

74. Living documentation for long-running behaviour

75. Biz Dev Leverage state machine & workflow engine Living documentation Visibility in testing Understand and discuss business processes Evaluate optimizations in-sync with implementation improve communication improve communication Ops

76. Biz Dev Leverage state machine & workflow engine Living documentation Visibility in testing Operate with visibility and context Understand and discuss business processes Evaluate optimizations in-sync with implementation improve communication improve communication Ops

77. Proper Operations Visibility + Context

78. Before mapping processes explicitly with BPMN, the truth was buried in the code and nobody knew what was going on. Jimmy Floyd, 24 Hour Fitnesse „

79. Workflows live inside service boundaries

80. Manigfold architecture options https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

84. … Manigfold architecture options https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

85. Lightweight workflow engines are great* – don‘t DIY *e.g. enabling potentially long-running services, solving hard developer problems, can run decentralized

86. Sales-Order & Order-Fulfillment via Camunda for every order worldwide (Q2 2017: 22,2 Mio)

87. Code example & live demo InventoryPaymentOrder ShippingCheckout Monitor https://github.com/flowing/flowing-retail/ Human Tasks H2 H2

88. # Events decrease coupling: sometimes read-models, but no complex peer-to-peer event chains! # Orchestration needs to be avoided: sometimes no ESB, smart endpoints/dumb pipes, important capabilities need a home # Workflow engines are painful: some of them lightweight engines are easy to use and can run decentralized, they solve hard developer problems, don‘t DIY

89. Thank you!

90. bernd.ruecker@camunda.com @berndruecker https://bernd-ruecker.com https://blog.bernd-ruecker.com https://github.com/flowing https://www.infoq.com/articles/events- workflow-automation With thoughts from http://flowing.io @berndruecker | @martinschimak Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/