Orchestration of Microservices in the domain of Business Processes. Presentation given by Daniel Meyer at Google dev fest 2017 Berlin

1. Orchestration of Microservices
Daniel Meyer, Camunda

2. Business
Processes
Understanding the Domain
2

3. Business Processes
Triggers Outcomes / Goals
Order
Placed
DSL Line
Connected
DSL
unavailableCustomer
Business
Process
3

4. Customer Satisfaction
Order
Placed
DSL Line
Connected
DSL
unavailable
SLAs & Reliability
Customer
Business
Process
Triggers Outcomes / Goals
4

5. Is this enough?
Request
Result
System
5

6. Is this enough?
Request
Result
System
NO 
6

7. Asynchronous & Long Running
Request
Acknowledge
Frontend
System
Defer
Backend
System
7

8. Highly Complex
Order
Placed
DSL Line
Connected
Validation Provisioning Billing
8

9. Business
Processes
Understanding the Domain
Asynchronous &
Long Running
Highly
Complex
Instrumental to
Customer Happiness
9

11. Event Storming
11

12. Flow Charts
12

13. Divide an Conquer
Request
Acknowledge
Frontend
System
Defer
Large monolithic System
Microservices
13

14. Decoupling
Service A Service B
Synchronous?
Service A Service B
Asynchronous!
14

15. Events, Commands & Pub/Sub
Event Bus
Service A Service B Service C Service D
15

16. Tracing Id
Event Bus
Service A Service B Service C Service D
Where is my order ??
16

17. State Machines
Provisioning
requested
timeout
failed
success
17

18. Sagas & Compensation
Book
Flight
Book
Hotel
Buy
Theater
Tickets
Cancel
Flight
Cancel
Hotel
18

19. Unified Model: Workflow
Workflow unifies approaches for Asynchronous and Event-Driven Processing:
- Asynchronous Communication
- Events & Pub / Sub
- Tracing Id
- State Machines
- Sagas & Compensation
19

20. Uber Cadence
20

21. Netflix Conductor
21

22. Amazon Step Functions & Simple Workflow Service
22

23. How Workflow works
23

24. How Workflow works
Assign Instance Id (Tracking Id)
24

25. How Workflow works
Step through Workflow and record State
25

26. How Workflow works
Assign data & payload with instance
26

27. How Workflow works
Step through Workflow and record State
27

28. How Workflow works
Step through Workflow and record State
28

29. How Workflow works
Step through Workflow and record State
29

30. How Workflow works
Step through Workflow and record State
30

31. How Workflow works
Step through Workflow and record State
31

32. How Workflow works
Step through Workflow and record State
32

33. How Workflow works
Step through Workflow and record State
33

34. How Workflow works
Step through Workflow and record State
34

35. Workflow and Pub / Sub
Publish Command
Subscribe to Events
35

36. Workflow and Pub / Sub
Workflow
Event Bus
Service B Service C Service D
36

38. Customer Satisfaction
Order
Placed
DSL Line
Connected
DSL
unavailable
SLAs & Reliability
Customer
Business
Process
Triggers Outcomes / Goals
38

39. Resilience
[…] the workflow that defines in which order
and how (location, input parameters,
timeouts, etc.) activities are executed is
guaranteed to continue execution under
various failure conditions.
39

40. Resilient Orchestrations
40

41. Resilient Orchestrations
41

42. Resilient Orchestrations
42

43. Resilient Orchestrations
43

44. Resilient Orchestrations
retry
44

45. Resilient Orchestrations
45

46. Visibility
“[…]
as the number of
microservices grows
and the complexity of
the processes increases,
getting visibility into
these distributed
workflows becomes
difficult without a
central orchestrator.”
46

47. Visibility
State and History of a particular Instance
47

48. Visibility
Overview over all instances
1.2K 9K 0.5K
48

49. Visibility
Statistics like average durations
6h 2h 1s
49

50. Visibility
50

51. Workflow
- A unified model for Asynchronous and Event-Driven Processing
- Provides Resilient Orchestrations
- Provides Visibility and Control
51

53. • 160 customers in >20 countries
• > 85% of customers use Camunda for core business processes
• Self funded & profitable. Revenue in growth 2016: 82%
• 70 Employees
Camunda
AT&T
53

54. 54

55. Meet Zeebe
“Zeebe” a new Microservice Orchestration & Workflow Engine.
Zeebe is Open Source and available at
• https://zeebe.io (Website)
• https://github.com/zeebe-io (Github)
55

56. Zeebe’s Mission
Zeebe is pushing the boundaries of what is possible with workflow.
• Extremely high throughput: more than 100x better than trad. engines on a single node
• Horizontally scalable
• Highly available, resilient and fault tolerant
• Completely asynchronous and reactive
56

57. Client / Server
Broker
Broker
Broker
Broker
Broker
Zeebe
Cluster
Zeebe
Client Connect
My App
Golang
57

58. Demo
58

59. Distributed System
Peer to Peer
No single point of failure
No single choking point
Broker
Broker
Broker
Broker
Broker
Zeebe
Cluster
59

60. Scale
UP
DOWN
3 Nodes 30 Nodes
SCALE
60

61. Cloud Native
docker run -p 51015:51015 camunda/zeebe:0.2.0
k8s: https://github.com/zeebe-io/zeebe-operator
61

62. Connecting Microservices
Watermark
Service
Thumbnail
Service
subscribe
62

63. Persistence
Generate Event Stream
Internally Zeebe works as an Event Sourcing System.
As Zeebe advances through a Workflow Instance, it generates a stream of events.
This event stream is both history but also “drives the execution forward”.
63

64. Replication
Data is replicated across brokers using raft protocol
Broker
2
Broker
1
Broker
3
Client
64

65. Thank you.
(Come work on Zeebe.io: If you want to help us make a difference pushing the
boundaries of what is possible with workflow, Camunda is hiring)