Presentation at Open Source Summit + Embedded Linux Conference Europe 2019

1. Cadence
Orchestration as Code Platform
Maxim Fateev, Temporal Technologies, Inc.

4. Choreography
A
B
C
D

5. Choreography
A
B
C
D

6. Choreography:
A
B
C
D

7. Choreography
A
B
C
D
Status?

8. Choreography
A
B
C
D

9. C
Orchestration
A
B
D
Orchestrator

10. C
Orchestration
A
B
D
Orchestrator

11. C
Orchestration
A
B
D
Orchestrator

12. C
Choreography
A
B
D
Orchestrator
Status?

13. Why Orchestration is not Widely Used?
● Scalability
● Single point of failure

14. Why Orchestration is not Widely Used?
● Programming model

15. Orchestration as Code

16. Service Orchestration
Account
Account
transfer
withdraw
deposit

17. Service Orchestration
Account
Account
transfer
withdraw
deposit

18. Service Orchestration
Account
Account
transfer
withdraw
deposit

19. Service Orchestration
Account
Account
transfer
withdraw
deposit

20. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

21. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

22. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

23. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

24. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

25. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

26. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

27. Ad Hoc Service Orchestration
Account
Account
transfer
withdraw
deposit

28. Ad Hoc Service Orchestration
Account
Account
withdraw
transfer
deposit

29. Orchestration as Code
Account
Account
transfer
withdraw
deposit

30. Cadence: Orchestration as Code
Account
Account
transfer
withdraw
deposit

31. Cadence: Orchestration as Code
Account
Account
transfer
withdraw
deposit

32. Cadence: Orchestration as Code
Account
Account
withdraw
deposit
transfer

33. Cadence: Orchestration as Code
Account
Account
withdraw
deposit
transfer

34. Cadence Deployment Topology
Account
start
signal
query
Workflow Worker (Decider)
Account
Activity Worker

35. Demo

36. Demo

37. Demo

38. Demo: Single Deposit for a Batch of Withdrawals

39. Demo

40. Demo Recap
● Workflow worker can be restarted any time
● All workflow state including local variables and blocking calls is recovered
● Activity can be retried indefinitely according to exponential retry
● UI and CLI give full visibility into the state of an execution

41. Workflow Recovery
● Event Sourcing
● Recovery
○ Replay from the beginning
○ Apply the same events
○ Ignore commands during replay
○ Determinism
● Only events are persisted
Events
Commands

42. Workflow
Events
Workflow Task
Commands
Task Completion
low
Cod
e
orkf
low
Cod
e
● Orchestration as code
○ Fully fault-tolerant self healing
○ Not linked to a specific process
○ Unlimited duration
● Code must be deterministic
○ Time
○ Random & UUID generation
○ Multithreading
○ Versioned updates
○ Cannot call external APIs directly
● All external APIs are called through commands
○ Activity
○ Timer
○ Signal
○ Child Workflow
Workflow Worker

43. Activity
Activity Heartbeat
Command
Activity Task
● Any application specific code
● Unlimited duration
● Heartbeat with progress
● Cancellation
● Unlimited exponential retries
● Routable to specific hosts
● Flow controlled
● Rate limited
● Not linked to a process
Event
Task Completion
Progress
Activity
CodeActivity
CodeActivity
CodeActivity
Code
Activity Worker

44. Service
● Highly Scalable
○ Tested up to 200 million open workflows
○ Tested up to 10 thousand events per second
● Highly Available
○ No downtime due to host failures
○ NoSQL or SQL database storage
○ Multi-cluster replication
● Multitenant
○ 100+ use cases at Uber on a single cluster
○ Individual domain cross cluster failover
● Pluggable
○ Database backend (Cassandra and MySQL)
○ Metrics
○ Discovery
○ Logging

45. Service
● Highly Scalable
○ Tested up to 200 million open workflows
○ Tested up to 10 thousand events per second
● Highly Available
○ No downtime due to host failures
○ NoSQL or SQL database storage
○ Multi-cluster replication
● Multitenant
○ 100+ use cases at Uber on a single cluster
○ Individual domain cross cluster failover
● Pluggable
○ Database backend (Cassandra and MySQL)
○ Metrics
○ Discovery
○ Logging
Two and half
years
in production!

46. Cadence Use Cases
● Online Store
● Tipping an Uber driver
● Deploying infrastructure on AWS
● Customer rewards program
● CI/CD Pipeline
● Provisioning a new telephone number for a customer
● ETL pipeline
● Processing payments
● Customer support
● Full scan of a large partitioned database
● ML Model training and deployment

47. Open Source Contributions
● NeonForge is building a C# client for
Cadence (link)
● Python client (link)
● S3 workflow history archival (link)
● HelmCharts by Banzai Cloud
Open Source Adoption
Close to 3000 Stars on GitHub
Snap

48. Distributed System Building Blocks
● Handlers
● Discovery & Routing
● Load Balancing
● Storage
● Sharding
● Queuing
● Streaming & Event Sourcing
● Consensus
● Durable Timers
● Rate Limiting
● Flow Control

49. Distributed System Building Blocks
● Handlers
● Discovery & Routing
● Load Balancing
● Storage
● Sharding
● Queuing
● Streaming & Event Sourcing
● Consensus
● Durable Timers
● Rate Limiting
● Flow Control

50. Cadence Recap
● Orchestration as Code Platform
● Developer friendly way to write fault-tolerant distributed applications
● Highly scalable and available
● Production hardened
● Open Sourced under MIT
● cadenceworkflow.io
● Demo code at: https://github.com/mfateev/presentation-java-samples
● Commercial support by temporal.io
● Follow me on twitter at @mfateev