Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Workflow and state machines at scale WJAX, Munic, 08th of November 2017 mail@bernd-ruecker.com | @berndruecker
What is workflow?
The „hello world“ example Human task management Service orchestration Events , e.g. timers
More realistic examples
Microservices & distributed systems
Message & Events
Timeout
Distributed transaction and compensation
Messages and events
And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
Traditional workflow engines
Performance?
Status quo
Concrete numbers please!* *using my ThinkPad T460s consumer notebook in a totally not repeatable manner while rendereing a...
0 50 100 150 200 250 300 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Instancesstarted...
Scaling
In general, we may conclude that Camunda performed better and more stable for all metrics when compared with WfMS A and Wf...
At Google, we have different ideas about scalability. Senior Architect at Google discussing BPMN. „ „
Photo by Mimzy, available under Creative Commons CC0 1.0 license.
Telematics night-2420297_1920.jpg Photo by jievoyage, available under Creative Commons CC0 1.0 license.
IOT Photo by Mark Bradshaw, available under Creative Commons BY 2.0 license.
Logistics Photo by Sam Churchill, available under Creative Commons BY 2.0 license.
Financial trading Photo by 3844328, available under Creative Commons CC0 1.0 license.
Can you handle 200.000 trades/second? „
New requirements -> new solutions Remove RDMS Distributed system Design for performance Resilience Flexible persistence Lo...
Zeebe in a nutshell https://zeebe.io/ Zeebe Broker Your application Binary (MsgPack) Support streaming & batching Client H...
Scale elastically UP DOWN 3 Nodes 30 Nodes SCALE
Distributed system with topics & partitions P1: F P2 : F P3 : L P1: L P2 : F P3 : F P1: F P2 : L P3 : F Example: - Topic w...
Data flow (simplified) Client COMPLETE command Broker - Partition Leader Brokers - Partition Followers 1 2 append replicat...
No external services needed
Publish / Subscribe Zeebe Broker Your code Zeebe Client subscribe Execute BPMNExecute Business Logic
Publish / Subscribe Zeebe Broker Service 1 Zeebe Client subscribe to A Service 2 Zeebe Client subscribe to B
Live hacking
https://github.com/zeebe-io/spring-zeebe
Performance?
on average 23.414 events/sec on average 4.682 started instance/sec
The box said „Requires Windows 7 or better“ so I installed Linux
on average 213.152 events/sec on average 35.525 started instance/sec
Status & Roadmap • Tech preview (0.4.0) • Working towards production-ready state • Focus on core, not (yet) on completenes...
Architecture choices (examples) Zeebe Service A Zeebe Client BPMN Service B Zeebe Client *Isolation and autonomy can also ...
CADENCE
https://github.com/samarabbas/cadence-samples/blob/master/cmd/samples/ CADENCE X No image available
“[…] as the number of microservices grow and the complexity of the processes increases, getting visibility into these dist...
https://netflix.github.io/conductor/metadata/kitchensink/
http://docs.aws.amazon.com/step-functions/latest/dg/tutorial-creating-lambda-state-machine.html
Zeebe Uber Cadence Netflix Orcherstrator AWS Step Functions Client language or API Java, Go Go Java, REST AWS SDK, REST Pe...
BPMN Executable* Easy to understand** Widespread **(for Biz and Dev)* (and mature)
Should I throw away my traditional workflow engine? A-F G-Z
https://db-engines.com/en/ranking_trend
https://db-engines.com/en/ranking_trend
Thank you!
Slides: https://bernd-ruecker.com Blog: https://blog.bernd-ruecker.com Feedback: https://bernd-ruecker.com/feedback
Images licensed from iStock no attribution required Own photos All icons licensed from Noun Project no attribution required
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
Upcoming SlideShare
Loading in …5
×

WJAX 2017: Workflow and state machines at scale

1,027 views

Published on

Slides from my talk at WJAX Munic on 8th of November about scaling workflow engines to infinity and beyond.

Published in: Technology
no profile picture user

  • Be the first to comment

WJAX 2017: Workflow and state machines at scale

  1. 1. Workflow and state machines at scale WJAX, Munic, 08th of November 2017 mail@bernd-ruecker.com | @berndruecker
  2. 2. What is workflow?
  3. 3. The „hello world“ example Human task management Service orchestration Events , e.g. timers
  4. 4. More realistic examples
  5. 5. Microservices & distributed systems
  6. 6. Message & Events
  7. 7. Timeout
  8. 8. Distributed transaction and compensation
  9. 9. Messages and events
  10. 10. And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
  11. 11. And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
  12. 12. And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
  13. 13. Traditional workflow engines
  14. 14. Performance?
  15. 15. Status quo
  16. 16. Concrete numbers please!* *using my ThinkPad T460s consumer notebook in a totally not repeatable manner while rendereing a talk recording ;-)
  17. 17. 0 50 100 150 200 250 300 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Instancesstarted/sec Time on average 181 instances started/sec
  18. 18. Scaling
  19. 19. In general, we may conclude that Camunda performed better and more stable for all metrics when compared with WfMS A and WfMS B. Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns, June 2016, University of Stuttgart, Germany and University of Lugano, Switzerland „
  20. 20. At Google, we have different ideas about scalability. Senior Architect at Google discussing BPMN. „ „
  21. 21. Photo by Mimzy, available under Creative Commons CC0 1.0 license.
  22. 22. Telematics night-2420297_1920.jpg Photo by jievoyage, available under Creative Commons CC0 1.0 license.
  23. 23. IOT Photo by Mark Bradshaw, available under Creative Commons BY 2.0 license.
  24. 24. Logistics Photo by Sam Churchill, available under Creative Commons BY 2.0 license.
  25. 25. Financial trading Photo by 3844328, available under Creative Commons CC0 1.0 license.
  26. 26. Can you handle 200.000 trades/second? „
  27. 27. New requirements -> new solutions Remove RDMS Distributed system Design for performance Resilience Flexible persistence Load
  28. 28. Zeebe in a nutshell https://zeebe.io/ Zeebe Broker Your application Binary (MsgPack) Support streaming & batching Client Horiziontally scalable Append only log / event sourcing Peer-to-peer cluster
  29. 29. Scale elastically UP DOWN 3 Nodes 30 Nodes SCALE
  30. 30. Distributed system with topics & partitions P1: F P2 : F P3 : L P1: L P2 : F P3 : F P1: F P2 : L P3 : F Example: - Topic with 3 partitions and replication factor 3 - PX = Partition X - L = Leader - F = Follower Replication uses the Raft protocol
  31. 31. Data flow (simplified) Client COMPLETE command Broker - Partition Leader Brokers - Partition Followers 1 2 append replicate & commit 3 4 process Stream Processor State Partition 5 decide & applyCOMPLETED response 6
  32. 32. No external services needed
  33. 33. Publish / Subscribe Zeebe Broker Your code Zeebe Client subscribe Execute BPMNExecute Business Logic
  34. 34. Publish / Subscribe Zeebe Broker Service 1 Zeebe Client subscribe to A Service 2 Zeebe Client subscribe to B
  35. 35. Live hacking
  36. 36. https://github.com/zeebe-io/spring-zeebe
  37. 37. Performance?
  38. 38. on average 23.414 events/sec on average 4.682 started instance/sec
  39. 39. The box said „Requires Windows 7 or better“ so I installed Linux
  40. 40. on average 213.152 events/sec on average 35.525 started instance/sec
  41. 41. Status & Roadmap • Tech preview (0.4.0) • Working towards production-ready state • Focus on core, not (yet) on completeness of features • We love to get your feedback and discuss!
  42. 42. Architecture choices (examples) Zeebe Service A Zeebe Client BPMN Service B Zeebe Client *Isolation and autonomy can also be achieved in this architecture Monolith Zeebe Zeebe Client BPMN Service A Zeebe Client BPMN Zeebe Service B Kafka/AMQP/… the monolith Microservices some hybrid*
  43. 43. CADENCE
  44. 44. https://github.com/samarabbas/cadence-samples/blob/master/cmd/samples/ CADENCE X No image available
  45. 45. “[…] as the number of microservices grow and the complexity of the processes increases, getting visibility into these distributed workflows becomes difficult without a central orchestrator.”
  46. 46. https://netflix.github.io/conductor/metadata/kitchensink/
  47. 47. http://docs.aws.amazon.com/step-functions/latest/dg/tutorial-creating-lambda-state-machine.html
  48. 48. Zeebe Uber Cadence Netflix Orcherstrator AWS Step Functions Client language or API Java, Go Go Java, REST AWS SDK, REST Persistent storage Event log on file system Cassandra Netflix Dynomite (k-v) - License and business model OSS, backed by vendor (EE) OSS project OSS project Cloud Distribution Source, Binary Source Source - Flow language BPMN, YAML, Java YAML, Go JSON JSON Visibility BPMN None Proprietary Proprietary
  49. 49. BPMN Executable* Easy to understand** Widespread **(for Biz and Dev)* (and mature)
  50. 50. Should I throw away my traditional workflow engine? A-F G-Z
  51. 51. https://db-engines.com/en/ranking_trend
  52. 52. https://db-engines.com/en/ranking_trend
  53. 53. Thank you!
  54. 54. Slides: https://bernd-ruecker.com Blog: https://blog.bernd-ruecker.com Feedback: https://bernd-ruecker.com/feedback
  55. 55. Images licensed from iStock no attribution required Own photos All icons licensed from Noun Project no attribution required

×