This document provides an overview of Cadence, an open source workflow platform. It discusses how Cadence can be used to build scalable and reliable distributed applications by providing common building blocks like activities, workflows, queues, and storage. Several case studies are presented that demonstrate how real-world problems from domains like ridesharing and customer loyalty programs can be modeled as workflows in Cadence. Key advantages of Cadence include its ability to handle large volumes of work, ensure high availability even during failures, and unify different workflow definition languages under one platform.

1. The Only Workflow Platform
You'll Ever Need
Maxim Fateev

2. Case Study:
Tips

3. void addTip(Tip t) {
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount

4. void addTip(Tip t) {
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount

5. void addTip(Tip t) {
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount

6. void OnMessage(Tip t){
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount
Queue

7. void OnMessage(Tip t){
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount
Queue

8. void OnMessage(Tip t){
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount
Queue

9. void OnMessage(Tip t){
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount
Queue

10. void OnMessage(Tip t){
debitRider(t);
}
DebitAccount
CreditAccount
Queue
void OnMessage(Tip t){
creditDriver(t);
}
Queue

11. void OnMessage(Tip t){
debitRider(t);
}
DebitAccount
CreditAccount
Queue
void OnMessage(Tip t){
creditDriver(t);
}
Queue
Status of the
transaction?

12. void OnMessage(Tip t){
debitRider(t);
}
DebitAccount
CreditAccount
Queue
void OnMessage(Tip t){
creditDriver(t);
}
Queue
Status of the
transaction?

13. void OnMessage(Tip t){
debitRider(t);
updateDB(status);
}
DebitAccount
CreditAccount
Queue
void OnMessage(Tip t){
creditDriver(t);
updateDB(status);
}
Queue
Database

14. void OnMessage(Tip t){
debitRider(t);
updateDB(status);
}
DebitAccount
CreditAccount
Queue
void OnMessage(Tip t){
creditDriver(t);
updateDB(status);
}
Queue
Database

15. void OnMessage(Tip t){
updateDB(status);
debitRider(t);
}
DebitAccount
CreditAccount
Queue
void OnMessage(Tip t){
updateDB(status);
creditDriver(t);
}
Queue
Database

16. void OnMessage(Tip t){
updateDB(status);
debitRider(t);
}
DebitAccount
CreditAccount
Queue
void OnMessage(Tip t){
updateDB(status);
creditDriver(t);
}
Queue
Database

17. void addTip(Tip t) {
debitRider(t);
creditDriver(t);
}
DebitAccount
CreditAccount

18. void addTip(Tip t) {
debitRider(t)
creditDriver(t)
}
DebitRider
CreditDriver
Db
Queue
Queue
Queue

19. void addTip(Tip t) {
debitRider(t)
creditDriver(t)
}
DebitRider
CreditDriver
Db
Queue
Queue
Queue

20. void addTip(Tip t) {
debitRider(t)
creditDriver(t)
}
DebitRider
CreditDriver
Db
Queue
Queue
Queue

21. void addTip(Tip t) {
debitRider(t)
creditDriver(t)
}
DebitRider
CreditDriver
Db
Queue
Queue
Queue

22. void addTip(Tip t) {
debitRider(t)
creditDriver(t)
}
DebitRider
CreditDriver
Db
Queue
Queue
Queue

23. Cadence Programming Model
● Activities (aka Tasks)
● Workflows

24. Cadence Activities
● Any application specific code
● Potentially long lived (heartbeating)
● Can be implemented asynchronously
● Automatically retried according to a specified retry policy
● Routable to specific hosts or processes
● Dispatched through queues
● Per worker rate and parallelism limit
● Per queue rate limit

25. Cadence Activity

26. Cadence Workflows
● Virtual Objects in Java or Go
● Transactional
● Orchestrate Activities
● React to external events
● Stateful including local variables and stack
● Queryable
● Potentially Long Lived
● Durable Timers

27. Cadence Workflows

28. Cadence Workflows

29. Cadence Workflows

30. Cadence Workflows

31. Activity Retry

32. Compensation

33. Case Study: Driver Rewards
● Driver signs up if qualifies
● Eligibility is checked every 30 days
● Participation is lost if doesn’t meet the
rewards requirements when checked
● Service listens on trip completion events to
calculate average rating

34. void OnMessage(Trip t){
State s = loadFromDb(t.driverId);
s.addTrip(t);
saveToDb(s);
}
PartnerService
Queue
Database
void onTimer(String driverId){
State s = loadFromDb(driverId);
if (s.eligible) {
activate(driverId);
} else {
deactivate(driverId);
}
s.reset();
saveToDB(s);
scheduleTimer();
}

35. void OnMessage(Trip t){
State s = loadFromDb(t.driverId);
s.addTrip(t);
saveToDb(s);
}
PartnerService
Queue
Database
void onTimer(String driverId){
State s = loadFromDb(driverId);
if (s.eligible) {
activate(driverId);
} else {
deactivate(driverId);
}
s.reset();
saveToDB(s);
scheduleTimer();
}

36. Driver Rewards Workflow

37. Driver Rewards Workflow

38. Driver Rewards Workflow

39. Driver Rewards Workflow

40. Driver Rewards Workflow

41. Driver Rewards Workflow

42. Driver Rewards Workflow

43. Use Case: Uber Flow
● UI based workflows
● Graph execution engine.
● Each edge has conditions
attached
● Some state nodes are
associated with actions

44. Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1
Flow Workflow Definition

45. Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1 - Entity1
Flow Workflow Instance

46. Event1
Entity1
Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1 - Entity1
Flow Event Matching

47. Event1
Entity1
Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1 - Entity1
Flow Condition Evaluation

48. Event1
Entity1
Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1 - Entity1
PartnerService
PartnerService
PartnerService
PartnerService
Services
Flow Condition Evaluation

49. Event1
Entity1
Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1 - Entity1
Flow Condition Evaluation

50. Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1 - Entity1
Flow Action

51. Node1
Node3
Node5
Node2
Node4
condition1
condition2
condition4
condition3
Workflow1 - Entity1
Node Action:
Send email, give
incentive, etc.
Flow Action

52. ● Potentially billions of entities
● ~ 50 workflows per entity
● ~ 10K external events per second
○ not counting duplicates
● Each event should be checked against all workflows for an entity
○ 50 * 10K = 500K condition evaluation per second
● On average one workflow per event generate an action
○ ~ 10K actions
Flow Scalability Requirements

53. void OnMessage(Message t){
List<String> workflows = getWorkflowsFor(t);
for (String workflow: workflows) {
State s = loadFromDb(workflow, t.entityId);
if (s.currentNode.evaluateConditions(t)) {
s.currentNode.executeAction();
saveToDb(s);
}
}
}
Services
Queue
Database
Services
Services
Services
Flow Original Implementation

54. void OnMessage(Message t){
List<String> workflows = getWorkflowsFor(t);
for (String workflow: workflows) {
State s = loadFromDb(workflow, t.entityId);
if (s.currentNode.evaluateConditions(t)) {
s.currentNode.executeAction();
saveToDb(s);
}
}
}
Services
Queue
Database
Services
Services
Services
Flow Original Implementation

55. List<State> workflows;
void OnMessage(Message t){
for (State workflow: workflows) {
if (s.currentNode.evaluateConditions(t)) {
s.currentNode.executeAction();
}
}
}
Services
Queue Services
Services
Services
Flow on Cadence

56. Flow on Cadence Advantages
● Reliable retries of condition evaluations
● Reliable retries of actions
● Database load is proportional to number of events per second.
○ Not (number of events) x (number of workflows per entity) which is 500K
● Cross datacenter failover
● Unified workflow engine

57. Flow Load Test
7500 events per second ingested

58. Flow Load Test
● 218000 per second condition evaluations
● 4400 actions per second

59. Cadence as a Platform
Cadence Service
BPMN
AWS States
Language
Airflow
DAG
Uber
Flow
Custom
DSL2
Java SDK Go SDK
JavApp1
JavaApp2
BPMNApp1
BPMNApp2
SLApp1
SLApp2
SLApp3
SLApp4
GoApp1
GoApp2
App1
App1
AirflowApp1
AirflowApp2
AirflowApp3
App1App1App1
DSL1App1
DSL1App2
DSL3App1
App1
App1
FlowApp1
FlowApp2
Custom
DSL1
Custom
DSL3
App1App1App1
DSL2App1
DSL2App2

60. More Uber Cadence Use Cases
● Freight load workflow
● Driver loyalty program
● Customer support workflows
● CI/CD/Deployment infrastructure
● End of month statement generation for each u4b customer
● Recalculate every hexagon on the city map every 1 minute for every city
● Tip processing in microservices architecture
● Managing Flink and Spark Jobs in Mesos or Yarn
● Customer loyalty Program
● Marketing email campaign management
● New datacenter provisioning
● Numerous other periodic jobs

61. Distributed Application Building Blocks
● Request Handlers
○ Microservices
○ Serverless
○ Actors
● Storage
○ Databases
○ Caches
● Queues
● Job Schedulers
● Consensus
○ Leader Election
○ Sharding
○ Distributed Locks

62. Distributed Application Building Blocks
● Request Handlers
○ Microservices
○ Serverless
○ Actors
● Storage
○ Databases
○ Caches
● Queues
● Job Schedulers
● Consensus
○ Leader Election
○ Sharding
○ Distributed Locks

63. Cadence Summary
● Higher level way of building distributed applications
○ Focus on business logic not plumbing
● Large scale
○ Billions of workflow instances
○ Tens of thousands of events per second
● High availability
○ Oblivious to node failures
○ Cross Datacenter Replication
● Unify all workflow solutions
○ Can support any existing workflow definition language
○ Perfect for DSL
● Open Source
○ http://cadenceworkflow.io
○ Apache 2.0 License