Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Java User Group Erfurt 2018: Zeebe.io - Event-driven Microservice Orchestration

631 views

Published on

Slides from my talk at JUG Erfurt on 22nd or march 2018.
Sample code available: https://github.com/flowing/flowing-retail/tree/zeebe
Recording in German available: https://youtu.be/lZIe02um5eI

Published in: Technology
  • Be the first to comment

Java User Group Erfurt 2018: Zeebe.io - Event-driven Microservice Orchestration

  1. 1. Zeebe.io Event-driven Microservice Orchestration @berndruecker
  2. 2. Berlin, Germany bernd.ruecker@camunda.com @berndruecker Bernd Ruecker Co-founder and Developer Advocate of Camunda
  3. 3. Camunda Open source platform for workflow and decision automation. Developer friendly! > 75 people HQ Berlin + Offices US No VC 0 20 40 60 80 100 120 140 160 180 200 2014 2015 2016 2017 2018
  4. 4. What is orchestration?
  5. 5. Orchestration is one use case of workflow automation
  6. 6. What is workflow automation?
  7. 7. <= milliseconds seconds minutes, weeks, … Business IT long runningalways short running short running, but potentially long running Use cases for workflow automation
  8. 8. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation long runningalways short running short running, but potentially long running Use cases for workflow automation
  9. 9. The „hello world“ example Human task management Service orchestration Events , e.g. timers
  10. 10. BPMN Business Process Model and Notation ISO Standard
  11. 11. More realistic examples
  12. 12. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation long runningalways short running short running, but potentially long running Use cases for workflow automation
  13. 13. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation long runningalways short running short running, but potentially long running Use cases for workflow automation
  14. 14. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  15. 15. https://www.infoworld.com/article/3254777/application-development/ 3-common-pitfalls-of-microservices-integrationand-how-to-avoid-them.html
  16. 16. Payment Ever called a RESTAPI? Credit Card REST
  17. 17. Distributed systems
  18. 18. Payment Ever called a RESTAPI? Credit Card REST
  19. 19. Payment Stateful retry Credit Card REST
  20. 20. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  21. 21. Distributed systems
  22. 22. It is impossible to differentiate certain failure scenarios. Independant of communication style! Service Provider Client
  23. 23. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  24. 24. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
  25. 25. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  26. 26. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  27. 27. Distributed systems
  28. 28. Distributed transactions using compensation * Compensation
  29. 29. Eventual consistency Temporarily inconsistent state But only temporarily!
  30. 30. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  31. 31. Long running Potentially
  32. 32. Long running = sit and wait
  33. 33. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  34. 34. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Orchestration Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  35. 35. Simplified example: dash button Photo by 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/
  36. 36. (Micro-)services Checkout Payment Inventory Shipment
  37. 37. Three steps…
  38. 38. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched
  39. 39. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched
  40. 40. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
  41. 41. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
  42. 42. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
  43. 43. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment
  44. 44. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Goods fetched Order placed Payment received Goods shipped
  45. 45. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Customers can pay via invoice Goods fetched Order placed Payment received Goods shipped …
  46. 46. Photo by born1945, available under Creative Commons BY 2.0 license.
  47. 47. Order Extract the end-to-end responsibility Checkout Payment Inventory Shipment *Commands have an intent about what needs to happen in the future Order placed Payment received Retrieve payment
  48. 48. Orchestration logic
  49. 49. Workflows live inside service boundaries
  50. 50. <= milliseconds seconds minutes, weeks, … Business IT Business processes automation Distributed Transactions Orchestration Communication in distributed systems long runningalways short running short running, but potentially long running Use cases for workflow automation
  51. 51. Traditional workflow engines
  52. 52. Performance?
  53. 53. Status quo
  54. 54. In general, we may conclude that Camunda performed better and more stable for all metrics when compared with WfMS A and WfMS B. Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns, June 2016, University of Stuttgart, Germany and University of Lugano, Switzerland „
  55. 55. Concrete numbers please!* *using my ThinkPad T460s consumer notebook in a totally unrepeatable manner while rendereing a talk recording ;-)
  56. 56. 0 50 100 150 200 250 300 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Instancesstarted/sec Time on average 181 instances started/sec
  57. 57. Scaling
  58. 58. At Google, we have different ideas about scalability. Senior Architect at Google discussing BPMN. „
  59. 59. Photo by Mimzy, available under Creative Commons CC0 1.0 license.
  60. 60. Telematics night-2420297_1920.jpg Photo by jievoyage, available under Creative Commons CC0 1.0 license.
  61. 61. IOT Photo by Mark Bradshaw, available under Creative Commons BY 2.0 license.
  62. 62. Logistics Photo by Sam Churchill, available under Creative Commons BY 2.0 license.
  63. 63. Financial trading Photo by 3844328, available under Creative Commons CC0 1.0 license.
  64. 64. Can you handle 200.000 trades/second? „
  65. 65. Workflow automation at scale! low latency, high-throughput low frequency, latency doesn‘t matter What people think workflow automation can do What we currently teach workflow automation to be able to do What workflow automation can already do today
  66. 66. New requirements -> new solutions Remove RDMS Distributed system Design for performance Resilience Flexible persistence Load
  67. 67. Client Zeebe Zeebe Broker Your applicationModel defined here and executed here Workers subscribe (pub/sub) JVM JVM
  68. 68. public static void main(String[] args) { ZeebeClient zeebe = ZeebeClient.create(new Properties()); zeebe.workflows().deploy("default-topic") .addResourceFromClasspath("sample.bpmn") .execute(); zeebe.workflows().create("default-topic") .bpmnProcessId("sample") .payload(" {'a':'42'} ") .execute(); zeebe.tasks().newTaskSubscription("default-topic") .taskType("calc") .lockTime(Duration.ofMinutes(5)) .lockOwner("worker") .handler(new TaskHandler() { public void handle(TasksClient client, TaskEvent task) { // do calculation... client.complete(task); } }).open(); }
  69. 69. Live hacking
  70. 70. https://github.com/zeebe-io/spring-zeebe
  71. 71. Client Zeebe Zeebe Broker Your application Streaming / reactive Binary protocll (MsgPack) + Batching Scale elastically and horizontally Append only log / event sourcing Peer-to-peer cluster Single Writer Consensus via Raft Model defined here and executed here Workers subscribe
  72. 72. YAML possible (converted to BPMN internally)
  73. 73. BPMN Executable and mature Easy to understand* Widespread *(for Biz and Dev)
  74. 74. Living documentation for long-running behaviour
  75. 75. Biz Dev Leverage state machine & workflow engine Living documentation Visibility in testing Operate with visibility and context Understand and discuss business processes Evaluate optimizations in-sync with implementation improve communication improve communication Ops
  76. 76. Proper Operations Visibility + Context
  77. 77. Compare to e.g. Step Functions
  78. 78. Compare to e.g. Step Functions
  79. 79. Imagine more complex stuff
  80. 80. https://netflix.github.io/conductor/metadata/kitchensink/
  81. 81. Zeebe Uber Cadence Netflix Orcherstrator AWS Step Functions Client language or API Java, Go Go Java, REST AWS SDK, REST Persistent storage Event log on file system Cassandra Netflix Dynomite (k-v) - License and business model OSS, backed by vendor (EE) OSS project OSS project Cloud Distribution Source, Binary Source Source - Flow language BPMN (XML, YAML, Java) Proprietary (YAML, Go) Proprietary (JSON) Proprietary (JSON) Visibility BPMN None Proprietary Proprietary
  82. 82. Zeebe Uber Cadence Netflix Orcherstrator AWS Step Functions Client language or API Java, Go Go Java, REST AWS SDK, REST Persistent storage Event log on file system Cassandra Netflix Dynomite (k-v) - License and business model OSS, backed by vendor (EE) OSS project OSS project Cloud Distribution Source, Binary Source Source - Flow language BPMN (XML, YAML, Java) Proprietary (YAML, Go) Proprietary (JSON) Proprietary (JSON) Visibility BPMN None Proprietary Proprietary
  83. 83. TL;DR: Do our stuff. Not the others‘ ;-)
  84. 84. Architecture Client Zeebe Broker Your application
  85. 85. Architecture
  86. 86. Zeebe Client BPMN Workflow Definition Zeebe Broker Zeebe Client Application BPMN Workflow Definition Zeebe Broker Application Inventory Service Payment Service Shipment Service Checkout Service Shipment Service Inventory Service Payment Service Checkout Service
  87. 87. Zeebe Client BPMN Workflow Definition Zeebe Broker Zeebe Client BPMN Workflow Definition Zeebe Broker Order Service Order Service Application Application Inventory Service Payment Service Shipment Service Checkout Service Shipment Service Inventory Service Payment Service Checkout Service
  88. 88. Zeebe Client BPMN Workflow Definition Zeebe Broker Kafka Client (or AMQP, REST, …) Zeebe Client BPMN Workflow Definition Zeebe Broker Order Service Order Service Application Application Inventory Service Payment Service Shipment Service Checkout Service Shipment Service Inventory Service Payment Service Checkout Service Kafka (or RabbitMQ, …)
  89. 89. Zeebe Client BPMN Workflow Definition WorkItemHandler Retrieve payment WorkItemHandler Fetch goods WorkItemHandler Ship goods Zeebe Broker Kafka Client (or AMQP, REST, …) Zeebe Client BPMN Workflow Definition Zeebe Broker Order Service Order Service Application Kafka ClientKafka ClientKafka ClientKafka Client Inventory Service Payment Service Shipment Service Checkout Service Zeebe Client WorkItemHandler Shipment Service Zeebe Client WorkItemHandler Zeebe Client WorkItemHandler Zeebe Client WorkItemHandler Inventory Service Payment Service Checkout Service Application Kafka (or RabbitMQ, …)
  90. 90. Live hacking
  91. 91. Status and roadmap • Tech preview (0.7.0) • Working towards production-ready state • Focus on core, not yet on completeness of features • We love to get your feedback and discuss!
  92. 92. Performance?
  93. 93. on average 23.414 events/sec on average 4.682 started instance/sec
  94. 94. The box said „Requires Windows 7 or better“ so I installed Linux
  95. 95. on average 213.152 events/sec on average 35.525 started instance/sec
  96. 96. Should I throw away my traditional workflow engine? A-F G-Z
  97. 97. https://db-engines.com/en/ranking_trend
  98. 98. https://db-engines.com/en/ranking_trend
  99. 99. Use Zeebe when # No RDMS # Masive scale # GoLang Use Camunda when # otherwise
  100. 100. Thank you!
  101. 101. bernd.ruecker@camunda.com @berndruecker https://bernd-ruecker.com https://blog.bernd-ruecker.com https://github.com/flowing https://www.infoq.com/articles/ events-workflow-automation With thoughts from http://flowing.io @berndruecker | @martinschimak Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html

×