Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Workflow and state machines
at scale
WJAX, Munic, 08th of November 2017
mail@bernd-ruecker.com | @berndruecker
What is workflow?
The „hello world“ example
Human task
management
Service
orchestration
Events , e.g.
timers
More realistic examples
Microservices &
distributed systems
Message & Events
Timeout
Distributed transaction and compensation
Messages and events
And even „smaller“ use cases
Service
(e.g. Go)
Kafka
RDMS
1. 3.
2.
And even „smaller“ use cases
Service
(e.g. Go)
Kafka
RDMS
1. 3.
2.
And even „smaller“ use cases
Service
(e.g. Go)
Kafka
RDMS
1. 3.
2.
Traditional workflow engines
Performance?
Status quo
Concrete numbers
please!*
*using my ThinkPad T460s
consumer notebook
in a totally not repeatable manner
while rendereing a...
0
50
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Instancesstarted...
Scaling
In general, we may conclude that
Camunda performed better and
more stable for all metrics when
compared with WfMS A and Wf...
At Google, we have
different ideas
about scalability.
Senior Architect at Google discussing BPMN.
„
„
Photo by Mimzy, available under Creative Commons CC0 1.0 license.
Telematics night-2420297_1920.jpg
Photo by jievoyage, available under Creative Commons CC0 1.0 license.
IOT
Photo by Mark Bradshaw, available under Creative Commons BY 2.0 license.
Logistics
Photo by Sam Churchill, available under Creative Commons BY 2.0 license.
Financial trading
Photo by 3844328, available under Creative Commons CC0 1.0 license.
Can you handle
200.000 trades/second?
„
New requirements -> new solutions
Remove
RDMS
Distributed
system
Design for
performance
Resilience
Flexible
persistence
Lo...
Zeebe in a nutshell
https://zeebe.io/
Zeebe
Broker
Your
application
Binary (MsgPack)
Support streaming & batching
Client
H...
Scale elastically
UP
DOWN
3 Nodes 30 Nodes
SCALE
Distributed system with topics & partitions
P1: F
P2 : F
P3 : L
P1: L
P2 : F
P3 : F
P1: F
P2 : L
P3 : F
Example:
- Topic w...
Data flow (simplified)
Client
COMPLETE
command
Broker - Partition Leader
Brokers -
Partition Followers
1
2 append
replicat...
No external services needed
Publish / Subscribe
Zeebe
Broker
Your code
Zeebe
Client
subscribe
Execute BPMNExecute Business Logic
Publish / Subscribe
Zeebe
Broker
Service
1
Zeebe
Client
subscribe to A
Service
2
Zeebe
Client
subscribe to B
Live hacking
https://github.com/zeebe-io/spring-zeebe
Performance?
on average
23.414
events/sec
on average
4.682
started instance/sec
The box said
„Requires
Windows 7
or better“
so I installed
Linux
on average
213.152
events/sec
on average
35.525
started instance/sec
Status & Roadmap
• Tech preview (0.4.0)
• Working towards production-ready state
• Focus on core, not (yet) on completenes...
Architecture choices (examples)
Zeebe
Service A
Zeebe
Client
BPMN
Service B
Zeebe
Client
*Isolation and autonomy can also
...
CADENCE
https://github.com/samarabbas/cadence-samples/blob/master/cmd/samples/
CADENCE
X
No image
available
“[…]
as the number of
microservices grow and
the complexity of the
processes increases,
getting visibility into
these dist...
https://netflix.github.io/conductor/metadata/kitchensink/
http://docs.aws.amazon.com/step-functions/latest/dg/tutorial-creating-lambda-state-machine.html
Zeebe Uber Cadence Netflix
Orcherstrator
AWS Step
Functions
Client language or
API
Java, Go Go Java, REST AWS SDK, REST
Pe...
BPMN
Executable*
Easy to
understand**
Widespread
**(for Biz and Dev)* (and mature)
Should I throw away my traditional workflow engine?
A-F G-Z
https://db-engines.com/en/ranking_trend
https://db-engines.com/en/ranking_trend
Thank you!
Slides:
https://bernd-ruecker.com
Blog:
https://blog.bernd-ruecker.com
Feedback:
https://bernd-ruecker.com/feedback
Images licensed from iStock
no attribution required
Own photos
All icons licensed from Noun Project
no attribution required
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
WJAX 2017: Workflow and state machines at scale
Upcoming SlideShare
Loading in …5
×

WJAX 2017: Workflow and state machines at scale

620 views

Published on

Slides from my talk at WJAX Munic on 8th of November about scaling workflow engines to infinity and beyond.

Published in: Technology
  • Be the first to comment

WJAX 2017: Workflow and state machines at scale

  1. 1. Workflow and state machines at scale WJAX, Munic, 08th of November 2017 mail@bernd-ruecker.com | @berndruecker
  2. 2. What is workflow?
  3. 3. The „hello world“ example Human task management Service orchestration Events , e.g. timers
  4. 4. More realistic examples
  5. 5. Microservices & distributed systems
  6. 6. Message & Events
  7. 7. Timeout
  8. 8. Distributed transaction and compensation
  9. 9. Messages and events
  10. 10. And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
  11. 11. And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
  12. 12. And even „smaller“ use cases Service (e.g. Go) Kafka RDMS 1. 3. 2.
  13. 13. Traditional workflow engines
  14. 14. Performance?
  15. 15. Status quo
  16. 16. Concrete numbers please!* *using my ThinkPad T460s consumer notebook in a totally not repeatable manner while rendereing a talk recording ;-)
  17. 17. 0 50 100 150 200 250 300 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Instancesstarted/sec Time on average 181 instances started/sec
  18. 18. Scaling
  19. 19. In general, we may conclude that Camunda performed better and more stable for all metrics when compared with WfMS A and WfMS B. Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns, June 2016, University of Stuttgart, Germany and University of Lugano, Switzerland „
  20. 20. At Google, we have different ideas about scalability. Senior Architect at Google discussing BPMN. „ „
  21. 21. Photo by Mimzy, available under Creative Commons CC0 1.0 license.
  22. 22. Telematics night-2420297_1920.jpg Photo by jievoyage, available under Creative Commons CC0 1.0 license.
  23. 23. IOT Photo by Mark Bradshaw, available under Creative Commons BY 2.0 license.
  24. 24. Logistics Photo by Sam Churchill, available under Creative Commons BY 2.0 license.
  25. 25. Financial trading Photo by 3844328, available under Creative Commons CC0 1.0 license.
  26. 26. Can you handle 200.000 trades/second? „
  27. 27. New requirements -> new solutions Remove RDMS Distributed system Design for performance Resilience Flexible persistence Load
  28. 28. Zeebe in a nutshell https://zeebe.io/ Zeebe Broker Your application Binary (MsgPack) Support streaming & batching Client Horiziontally scalable Append only log / event sourcing Peer-to-peer cluster
  29. 29. Scale elastically UP DOWN 3 Nodes 30 Nodes SCALE
  30. 30. Distributed system with topics & partitions P1: F P2 : F P3 : L P1: L P2 : F P3 : F P1: F P2 : L P3 : F Example: - Topic with 3 partitions and replication factor 3 - PX = Partition X - L = Leader - F = Follower Replication uses the Raft protocol
  31. 31. Data flow (simplified) Client COMPLETE command Broker - Partition Leader Brokers - Partition Followers 1 2 append replicate & commit 3 4 process Stream Processor State Partition 5 decide & applyCOMPLETED response 6
  32. 32. No external services needed
  33. 33. Publish / Subscribe Zeebe Broker Your code Zeebe Client subscribe Execute BPMNExecute Business Logic
  34. 34. Publish / Subscribe Zeebe Broker Service 1 Zeebe Client subscribe to A Service 2 Zeebe Client subscribe to B
  35. 35. Live hacking
  36. 36. https://github.com/zeebe-io/spring-zeebe
  37. 37. Performance?
  38. 38. on average 23.414 events/sec on average 4.682 started instance/sec
  39. 39. The box said „Requires Windows 7 or better“ so I installed Linux
  40. 40. on average 213.152 events/sec on average 35.525 started instance/sec
  41. 41. Status & Roadmap • Tech preview (0.4.0) • Working towards production-ready state • Focus on core, not (yet) on completeness of features • We love to get your feedback and discuss!
  42. 42. Architecture choices (examples) Zeebe Service A Zeebe Client BPMN Service B Zeebe Client *Isolation and autonomy can also be achieved in this architecture Monolith Zeebe Zeebe Client BPMN Service A Zeebe Client BPMN Zeebe Service B Kafka/AMQP/… the monolith Microservices some hybrid*
  43. 43. CADENCE
  44. 44. https://github.com/samarabbas/cadence-samples/blob/master/cmd/samples/ CADENCE X No image available
  45. 45. “[…] as the number of microservices grow and the complexity of the processes increases, getting visibility into these distributed workflows becomes difficult without a central orchestrator.”
  46. 46. https://netflix.github.io/conductor/metadata/kitchensink/
  47. 47. http://docs.aws.amazon.com/step-functions/latest/dg/tutorial-creating-lambda-state-machine.html
  48. 48. Zeebe Uber Cadence Netflix Orcherstrator AWS Step Functions Client language or API Java, Go Go Java, REST AWS SDK, REST Persistent storage Event log on file system Cassandra Netflix Dynomite (k-v) - License and business model OSS, backed by vendor (EE) OSS project OSS project Cloud Distribution Source, Binary Source Source - Flow language BPMN, YAML, Java YAML, Go JSON JSON Visibility BPMN None Proprietary Proprietary
  49. 49. BPMN Executable* Easy to understand** Widespread **(for Biz and Dev)* (and mature)
  50. 50. Should I throw away my traditional workflow engine? A-F G-Z
  51. 51. https://db-engines.com/en/ranking_trend
  52. 52. https://db-engines.com/en/ranking_trend
  53. 53. Thank you!
  54. 54. Slides: https://bernd-ruecker.com Blog: https://blog.bernd-ruecker.com Feedback: https://bernd-ruecker.com/feedback
  55. 55. Images licensed from iStock no attribution required Own photos All icons licensed from Noun Project no attribution required

×