Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

JAX 2019 - Workflow automation reinvented

369 views

Published on

Slides from my talk at JAX 2019

Published in: Technology
  • Be the first to comment

JAX 2019 - Workflow automation reinvented

  1. 1. Workflow Automation Reinvented @berndruecker
  2. 2. What people think, when I say „Workflow Automation“…
  3. 3. What I think, when I say „Workflow Automation“ …
  4. 4. What people think, when I say „BPM“… Death by properties panel Low-code is great! (You can get rid of your developers!) @berndruecker
  5. 5. What I think, when I say „BPM“ +
  6. 6. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } What do I mean by „lightweight?“ @berndruecker
  7. 7. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Build engine in one line of code (using in- memory H2) @berndruecker
  8. 8. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL @berndruecker
  9. 9. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL @berndruecker
  10. 10. BPMN Business Process Model and Notation ISO Standard @berndruecker
  11. 11. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } We can attach code… @berndruecker
  12. 12. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } …that is called when workflow instances pass through @berndruecker
  13. 13. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Start instances @berndruecker
  14. 14. Architecture choices Your app or service WF Engine Your app or service WF Engine WF Engine logic or service WF Engine logic or service Call, send message, connector, … (Long-)Polling Pub/Sub PushEmbedded Server Pull vs vs See also https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
  15. 15. Why workflow automation?
  16. 16. https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/
  17. 17. <= milliseconds seconds minutes, weeks, … Business IT long runningalways short running short running, but potentially long running Use cases for workflow automation
  18. 18. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation long runningalways short running short running, but potentially long running Use cases for workflow automation
  19. 19. Real-life examples
  20. 20. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation long runningalways short running short running, but potentially long running Use cases for workflow automation
  21. 21. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation long runningalways short running short running, but potentially long running Use cases for workflow automation Communication in distributed systems
  22. 22. Payment Ever called a RESTAPI? Credit Card REST
  23. 23. Distributed systems “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.” Lamport
  24. 24. Payment Stateful retry Credit Card REST
  25. 25. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  26. 26. It is impossible to differentiate certain failure scenarios. Independant of communication style! Service Provider Client
  27. 27. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  28. 28. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST Cancel charge
  29. 29. Live hacking
  30. 30. Warning: Contains Opinion
  31. 31. Berlin, Germany mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda
  32. 32. Live hacking https://github.com/berndruecker/flowing-retail/tree/master/rest/java/payment-camunda https://github.com/berndruecker/flowing-retail-payment-rest-spring-statemachine
  33. 33. Embedded, Push Your app or service WF Engine logic or service Call, send message, connector, … Push Embedded Raises questions around Threads, Transactions, …
  34. 34. Spring StateMachine docs
  35. 35. Aspects to consider Camunda Spring State Machine Workflow Definition BPMN 2.0 (graphical, XML, Model API) Java API, UML-Generator Visual? YES No Tooling Modeler, Cockpit, Optimize - Storage Runtime RDMS up to you Storage History RDMS - Scalability Stateless Engine, RDMS is limit, Sharding possible up to you Fault tolerance If RDMS is HA up to you Supported programming languages Java, REST, Language Clients (JS, C#) Java HOMEWORK
  36. 36. See more on http://berndruecker.io/
  37. 37. Long running services* provide a better API! @berndruecker * Services that have the capability to keep state and thus can be potentially long running
  38. 38. Example Order Payment Retrieve Payment @berndruecker
  39. 39. Example Order Payment Credit Card Retrieve Payment @berndruecker
  40. 40. Example Order Payment Credit Card Retrieve Payment Rejected @berndruecker
  41. 41. Example Order Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker
  42. 42. Example Order Payment Client of dumb endpoints easily become a god services. If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker
  43. 43. Payment failed Who is responsible to deal with problems? Order Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received @berndruecker
  44. 44. Payment failed Long running services Order Payment Credit Card Retrieve Payment Rejected Payment received Smart endpoints are potentially long-running @berndruecker
  45. 45. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation long runningalways short running short running, but potentially long running Use cases for workflow automation Communication in distributed systems
  46. 46. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation Orchestration
  47. 47. Microservices… Service B Some Service Some Service Some Service Service C Service A Service D Monolith Functionality A Functionality B Functionality C Functionality D
  48. 48. Simplified version… @berndruecker
  49. 49. (Micro-)services Checkout Payment Inventory Shipment @berndruecker
  50. 50. Order Placed Payment Received Goods Fetched Notification Event-driven architecture @berndruecker Checkout Payment Inventory Shipment
  51. 51. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched @berndruecker
  52. 52. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched @berndruecker
  53. 53. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  54. 54. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  55. 55. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  56. 56. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment @berndruecker
  57. 57. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Goods fetched Order placed Payment received Goods shipped @berndruecker
  58. 58. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0 @berndruecker
  59. 59. Order Extract the end-to-end responsibility Checkout Payment Inventory Shipment *Commands have an intent about what needs to happen in the future Payment received Order placed Retrieve payment @berndruecker
  60. 60. Order It still can be messaging! Checkout Payment Inventory Shipment Order placed Retrieve payment @berndruecker
  61. 61. Workflows implement stateful orchestration logic @berndruecker
  62. 62. No BPM(N) monoliths https://blog.bernd-ruecker.com/avoiding-the-bpm-monolith-when-using-bounded-contexts-d86be6308d8 @berndruecker
  63. 63. Microservice pioneers have become aware
  64. 64. https://netflix.github.io/conductor/metadata/kitchensink/
  65. 65. Live hacking https://github.com/berndruecker/flowing-retail/
  66. 66. Architecture choices WF Engine Service Server Pull Service WF Engine Service Server Queue / Topic Name Topic Name Service Pull
  67. 67. Aspects to consider Zeebe Conductor Workflow Definition BPMN 2.0 (graphical, XML, Model API) JSON Visual? YES No Tooling Modeler, Operate Simple UI Storage Runtime File System (via RocksDB) Dynomite + Elasticsearch Storage History Elasticsearch - Supported programming languages grpc, Language clients (Java, C#, JS, Python, C#) REST, Language Clients (Java, Python), grpc Licensing ASL ASL HOMEWORK
  68. 68. Camunda vs. Zeebe Camunda ZeebePersistent State Persistent change RDMS
  69. 69. Workflow Instance Id Current Activity State 2 RetrievePayment running Workflow Instance Id Current Activity State 2 ShipGoods running Workflow Instance Id Current Activity State 2 OrderDelivered ended 2. 3. UPDATE 2. UPDATE 1. INSERT 3.1. RDMS Workflow Engine
  70. 70. Append-only Log1. create workflow instance workflow instance created start event occured sequence flow taken activity activated task created lock created task locked complete task task completed activity completed sequence flow taken … … 2. 2. 3.1. Workflow Engine
  71. 71. Event Handling, Replication & Single Writer Follower Follower complete task command task completed event 1 send 2 append command Broker (Leader) Stream Processor 4 process 7 store & replicate event 6 append event 3 store & replicate command 5 respond Single Writer (single thread)
  72. 72. Horizontal scalability by partitioning Partition 1 Partition 2 Partition 3 Partition 4 Every workflow instance is exactly handled by one partition instance id: 2-42 instance id: 3-66 Stream Processor Single Writer (single thread)
  73. 73. See more on http://berndruecker.io/
  74. 74. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation Orchestration
  75. 75. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation Orchestration Consistency
  76. 76. The classical example book hotel book car book flight 1. 2. 3. Failure! book trip
  77. 77. Pat Helland “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  78. 78. Pat Helland Grown-Ups Don’t Use Distributed Transactions “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  79. 79. The classical example Saga book hotel book car book flight cancel hotel cancel car 1. 2. 3. 5.6. Failure book trip Trigger compensations
  80. 80. Live hacking
  81. 81. BPMN Saga Pattern (implemented by BPMN compensation) https://github.com/berndruecker/trip-booking-saga-java
  82. 82. Step Functions https://read.acloud.guru/how-the-saga-pattern-manages-failures-with-aws-lambda-and- step-functions-bc8f7129f900
  83. 83. Flow language is important! Think of more complicated scenarios…
  84. 84. Architecture for Serverless WF Engine (Long-)Polling Push Pull function Server glue layer logic or service Easy to hop over firewalls
  85. 85. Aspects to consider Camunda AWS Step Functions Workflow Definition BPMN 2.0 (graphical, XML, Model API) JSON Visual? YES Simple, only runtime Tooling Modeler, Cockpit, Optimize - Scalability Stateless Engine With RDMS Up to ~ 1k state transitions / sec* Deployment Embedded or Server Cloud (Server) Licensing ASL Proprietary HOMEWORK
  86. 86. See more on http://berndruecker.io/
  87. 87. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Orchestration Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  88. 88. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Decision Automation Orchestration Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  89. 89. Photo by ErikaWittlieb, available under Creative Commons CC0 1.0 license.
  90. 90. Decisions with DMN
  91. 91. Workflow Automation is important in modern architectures!
  92. 92. Thoughts on the state machine | workflow engine market
  93. 93. Thoughts on the state machine | workflow engine market OSS Workflow or Orchestration Engines Stack Vendors, Pure Play BPMS Low Code Platforms Homegrown frameworks to scratch an itch Integration Frameworks Cloud Offerings Uber, Netflix, AirBnb, ING, … AWS Step Functions, Azure Durable Functions, … Camunda, Zeebe, jBPM, Activiti, Flowable, Mistral, … PEGA, IBM, SAG, … Apache Airflow, Spring Data Flow, … Apache Camel, Balerina, … Data Pipelines
  94. 94. Does it support stateful operations? Does it support the necessary flow logic? Does it support BizDevOps? Does it scale?
  95. 95. https://zeebe.io/blog/2019/03/zeebe-comparison-evaluation-criteria/
  96. 96. My personal pro-tip for a shortlist ;-) OSS Workflow or Orchestration Engines Stack Vendors, Pure Play BPMS Low Code Platforms Homegrown frameworks to scratch an itch Integration Frameworks Cloud Offerings Data Pipelines Camunda & Zeebe
  97. 97. Thank you! @berndruecker
  98. 98. mail@berndruecker.io @berndruecker https://berndruecker.io https://medium.com/berndruecker https://github.com/berndruecker https://www.infoq.com/articles/events- workflow-automation Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/

×