Download as PDF, PPTX
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
What do I mean by
„leightweight?“
@berndruecker](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-qconlondon-190306091653/85/Complex-event-flows-in-distributed-systems-QCon-London-2019-53-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Build engine
in one line of
code
(using in-
memory H2)
@berndruecker](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-qconlondon-190306091653/85/Complex-event-flows-in-distributed-systems-QCon-London-2019-54-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL
@berndruecker](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-qconlondon-190306091653/85/Complex-event-flows-in-distributed-systems-QCon-London-2019-55-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL
@berndruecker](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-qconlondon-190306091653/85/Complex-event-flows-in-distributed-systems-QCon-London-2019-56-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
We can attach
code…
@berndruecker](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-qconlondon-190306091653/85/Complex-event-flows-in-distributed-systems-QCon-London-2019-58-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
…that is
called when
workflow
instances pass
through
@berndruecker](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-qconlondon-190306091653/85/Complex-event-flows-in-distributed-systems-QCon-London-2019-59-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Start
instances
@berndruecker](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-qconlondon-190306091653/85/Complex-event-flows-in-distributed-systems-QCon-London-2019-60-320.jpg)
Uploaded byBernd Ruecker
Complex event flows in distributed systems (QCon London 2019)
The document discusses three main hypotheses regarding event-driven architecture in distributed systems: events can reduce coupling, orchestration should be avoided, and workflow engines can be problematic. It outlines the importance of managing event flows and the challenges faced in ensuring system coherence without losing sight of overall architecture. The speaker emphasizes the need for lightweight workflow engines and balancing orchestration with choreography while avoiding complex peer-to-peer event chains.
More Related Content
Similar to Complex event flows in distributed systems (QCon London 2019)
Complex event flows in distributed systems (QCon London 2019)
- 1. Complex event flowsin distributed systems @berndruecker
- 2. 3 common hypothesesI check today: # Events decrease coupling # Orchestration needs to be avoided # Workflow engines are painful @berndruecker
- 3.
- 4. Berlin, Germany mail@berndruecker.io @berndruecker Bernd Ruecker Co-founderand Chief Technologist of Camunda
- 6. Simplified example: dash button Photoby 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/ @berndruecker
- 7.
- 8. Who is involved?Some bounded contexts… Checkout Payment Inventory Shipment @berndruecker
- 9.
- 10. Autonomous (micro-)services Checkout Payment Inventory Shipment Dedicated ApplicationProcesses Dedicated infrastructure Dedicated Development Teams @berndruecker
- 11.
- 12. Example Checkout Payment Inventory Shipment The button blinksif we can ship within 24 hours @berndruecker
- 13. Request/response: temporal coupling Checkout Payment Inventory Shipment Request Response Thebutton blinks if we can ship within 24 hours @berndruecker
- 14. Temporal decoupling withevents and read models Checkout Payment Inventory Shipment Good Stored Read Model Good Fetched The button blinks if we can ship within 24 hours *Events are facts about what happened (in the past) @berndruecker
- 15.
- 16. Events can decreasecoupling* *e.g. decentral data-management, read models, extract cross-cutting aspects @berndruecker
- 17.
- 18.
- 19. The danger isthat it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 20. The danger isthat it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 21. The danger isthat it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 22. Monitoring Workflows AcrossMicroservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices @berndruecker
- 23. Typical approachesDistributed Tracing DataLake / Event Monitoring Process Mining Process Tracking @berndruecker
- 24. Stefan Tilkov: MicroservicePatterns & Antipatterns - MicroXchg 2018 @berndruecker
- 25. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetchthe goods before the payment @berndruecker
- 26. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetchthe goods before the payment Goods fetched Order placed Payment received Goods shipped @berndruecker
- 27. Photo by born1945,available under Creative Commons BY 2.0 license. @berndruecker
- 28. What we wanted Photoby Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0 @berndruecker
- 29. „Challenges?“ Source: Microservices orchestration survey, July2018, 354 responses https://camunda.com/microservices-orchestration-survey-results-2018/ @berndruecker
- 30. Order Extract the end-to-endresponsibility Checkout Payment Inventory Shipment *Commands have an intent about what needs to happen in the future Payment received Order placed Retrieve payment @berndruecker
- 31. Order It is aboutwhere to decide about the coupling! Checkout Payment Inventory Shipment Order placed Retrieve payment Order decides . to listen to the event . to issue the command @berndruecker
- 32. Order It is aboutwhere to decide about the coupling! Checkout Payment Inventory Shipment Order placed Retrieve payment It can still be messaging! @berndruecker
- 33. Commands help toavoid (complex) peer-to-peer event chains @berndruecker
- 34. Orchestration needs tobe avoided @berndruecker
- 35.
- 36. Dumb pipes Checkout Payment Inventory Shipment Order Smart endpoints anddumb pipes Martin Fowler @berndruecker
- 37. Danger of godservices? Checkout Order A few smart god services tell anemic CRUD services what to do Sam Newmann Payment Inventory Shipment @berndruecker
- 38. Danger of godservices? Checkout Payment Inventory Shipment Order A few smart god services tell anemic CRUD services what to do Sam Newmann @berndruecker
- 39. A god serviceis only created by bad API design! @berndruecker
- 40.
- 41.
- 42.
- 43. Example Order Payment If thecredit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker
- 44. Example Order Payment Client ofdumb endpoints easily become a god services. If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker
- 45. Payment failed Who is responsibleto deal with problems? Order Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received @berndruecker
- 46. Payment failed Long running services OrderPayment Credit Card Retrieve Payment Rejected Payment received Smart endpoints are potentially long-running @berndruecker
- 47. Persist thing (Entity, Actor,…) State machine or workflow engine Typical concerns DIY = effort, accidental complexity Scheduling, Versioning, operating, visibility, scalability, … Handling State @berndruecker
- 48. Workflow engines arepainful Complex, proprietary, heavyweight, central, developer adverse, … @berndruecker
- 49. Avoid the wrongtools! Death by properties panel Low-code is great! (You can get rid of your developers!) Complex, proprietary, heavyweight, central, developer adverse, … @berndruecker
- 50. Workflow engines, state machines Itis relevant in modern architectures @berndruecker
- 51. CADENCE Silicon valley has recognized Workflowengines, state machines @berndruecker
- 52.
- 53. public static voidmain(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } What do I mean by „leightweight?“ @berndruecker
- 54. public static voidmain(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Build engine in one line of code (using in- memory H2) @berndruecker
- 55. public static voidmain(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL @berndruecker
- 56. public static voidmain(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL @berndruecker
- 57. BPMN Business Process Model andNotation ISO Standard @berndruecker
- 58. public static voidmain(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } We can attach code… @berndruecker
- 59. public static voidmain(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } …that is called when workflow instances pass through @berndruecker
- 60. public static voidmain(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Start instances @berndruecker
- 61. Payment Now you havea state machine! @berndruecker
- 62. Payment Easy to handletime @berndruecker
- 63.
- 64. Example with synchronouscommunication REST Order Payment Credit Card @berndruecker
- 65. Example with synchronouscommunication REST Order Payment Credit Card @berndruecker
- 66. Example with synchronouscommunication REST Order Payment Credit Card @berndruecker
- 67. Example with synchronouscommunication REST Order Payment Credit Card Stateful Retry @berndruecker
- 68. Works also forasynchronous communication Order Payment Credit Card Monitor Timeouts @berndruecker
- 69.
- 70. It is impossibleto differentiate certain failure scenarios: Independant of communication style! Service Provider Client @berndruecker
- 71. Distributed systems introducecomplexity you have to tackle! Credit Card Payment REST @berndruecker
- 72. Distributed systems introducecomplexity you have to tackle! Credit Card Payment REST The service can be long running. You get a better API and less gods @berndruecker
- 73. Workflows live insideservice boundaries @berndruecker
- 74.
- 75. Pat Helland “ Distributed SystemsGuru Worked at Amazon, Microsoft & Salesforce @berndruecker
- 76. Pat Helland Grown-Ups Don’tUse Distributed Transactions “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
- 77. Distributed transactions usingcompensation * Compensation @berndruecker
- 78. Homework: Try to dothis purely event-driven! Send to: mail@berndruecker.io @berndruecker
- 79. Biz Dev Leverage state machine& workflow engine Living documentation Visibility in testing improve communication improve communication Ops @berndruecker
- 80. Visual HTML reportsfor test cases @berndruecker
- 81. Living documentation forlong-running behaviour @berndruecker
- 82.
- 83. Biz Dev Leverage state machine& workflow engine Living documentation Visibility in testing Operate with visibility and context Understand and discuss business processes Evaluate optimizations in-sync with implementation improve communication improve communication Ops @berndruecker
- 84. Monitoring Workflows AcrossMicroservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices @berndruecker
- 85. Tracking Checkout Inventory Payment Shipment EventBus Workflow Engine https://www.confluent.io/kafka-summit-sf18/the_big_picture @berndruecker
- 86. Journey towards moreorchestration @berndruecker
- 87. Vodafone, Liongate &WDW Presented at CamundaCOn Berlin 2018
- 88. Before mapping processes explicitlywith BPMN, the truth was buried in the code and nobody knew what was going on. Jimmy Floyd, 24 Hour Fitnesse „ @berndruecker
- 89. … It addresses oneof the core issues in a distributed microservices architecture—where is the source of truth for the coordinated interaction of the entire system? … the system we are replacing uses a complex peer- to-peer choreography that requires reasoning across multiple codebases to understand. https://medium.com/@sitapati/node-js-client-for-zeebe-microservices-orchestration-engine-72287e4c7d94 Josh Wulf Credit Sense @berndruecker
- 90. Lightweight workflow enginesare great – don‘t DIY* *e.g. enabling potentially long-running services, solving hard developer problems, can run decentralized @berndruecker
- 91. Sales-Order & Order-Fulfillment viaCamunda for every order worldwide (Q2 2017: 22,2 Mio) @berndruecker
- 92.
- 93. Event-driven example InventoryPaymentOrder ShippingCheckoutMonitor https://github.com/berndruecker/flowing-retail/ Human Tasks H2 H2 @berndruecker
- 94. # Events decreasecoupling: sometimes read-models, but no complex peer-to-peer event chains! # Orchestration needs to be avoided: sometimes no ESB, smart endpoints/dumb pipes, balance orchestration and choreography # Workflow engines are painful: some of them lightweight engines are easy to use and can run decentralized, they solve hard developer problems, don‘t DIY @berndruecker
- 95.
- 96.