Spinning your Drones with Cadence Workflows and Apache Kafka

Spinning your Drones with Cadence
Workflows and Apache Kafka®
Paul Brebner
Instaclustr—Technology Evangelist
©Instaclustr Pty Limited, 2022
APIDAYS Hong Kong - August 2022

Who Am I?
Previously
§ R&D in distributed systems and performance
engineering.
Last 5 years
§ Technology Evangelist for Instaclustr by NetApp
§ 100+ Blogs, demo applications, talks
§ Open Source technologies including
o Apache Cassandra®, Spark™, Kafka®
o OpenSearch®, Redis™
o and now Uber’s Cadence®
© Instaclustr Pty Limited, 2022

Cloud Platform for Big Data
Open Source Technologies
Latest addition is
Workflow Orchestration
with
Instaclustr Managed Platform

Workflow Orchestration – task ordering
Start

Start Task 1

Start Task 1 Task 2

Start Task 1 Task 2 End

Pedalling with a high Cadence (pedal revolutions) is called Spinning!
Mashing or grinding is slow (and bad.)
(Source: Shutterstock)
With

Cadence is Fault-Tolerant— Workflows can fail!

Cadence is Fault-Tolerant—Workflows continue from point of failure

Cadence is Horizontally Scalable—the number of workflows is unlimited

Cadence Workflows Are Code
@Override
public void startWorkflow(String a) {
String b = activities.task1(a);
String c = activities.task2(b);
}
A workflow implementation with 2 tasks executed sequentially

Cadence Architecture
Instaclustr Managed Service
Workers
Run workflow logic
Customer Managed
Clients and Workers
Workers
Workers
Clients and
Workers
Instaclustr Managed
Cadence and Database Clusters
Cadence Servers
Database
Cassandra
PostgreSQL
Kafka
OpenSearch
(optional)
Java and Go Clients
are supported
APIs

§ com.uber.cadence.activity
APIs to implement activity, accessing activity info, or send heartbeat
§ com.uber.cadence.client
APIs for external application code to interact with Cadence workflows, start
workflows, send signals or query workflows.
§ com.uber.cadence.workflow
APIs to implement workflows
§ com.uber.cadence.worker
APIs to configure and start workers
And more
Cadence Client APIs

Database
Workflow History Written to Database
Start Workflow Start Task 1 End Task 1
HERE
How Does Cadence Fault-Tolerance Work?
Event-Sourcing = History + Replaying

Workflow State Recovery by
Replaying History
Database
Failure causes complete history
replay
Replay

Workflow State Recovery by
Replaying History
Database
Replay recreates Workflow state
and restart point
Replay
Restart

Cadence Activities
Activities (including
backwards cycling)
can fail

Activity: Task 2
Activity: Task 1
§ Activities are core to Cadence
§ Remote calls can fail, so wrap them in Activities
§ They can contain any code
§ Activities are executed at most once and can be automatically retried on failure
Cadence Activities
Start End
Remote
Call

Workflow Restrictions: Activities
@Override
if (a == null) a = “”;
String b = activities.task1(a); // executed at most once
long t = Workflow.currentTimeMillis();
String c = activities.task2(t, b); // executed at most once
}
Replaying must not produce new history events!
Activities executed at most once—this ensures that once they
succeed then the result is immutable (fixed)
Start Activity: Task 1 Activity: Task 2 End
Workflow
code
Workflow
code
<= 1 <= 1

@Override
if (a == null) a = “”; // executed multiple times
String b = activities.task1(a);
long t = Workflow.currentTimeMillis(); // built-in time method not replayed
String c = activities.task2(t, b);
}
Replaying must not produce new history events!
Due to replaying, workflow code executed multiple times, so must be deterministic
Use built-in methods for time, sleeping, random, and side-effects
Start Activity: Task 1 Activity: Task 2 End
Workflow
code
Workflow
code
>= 1 >= 1
Workflow Restrictions: Workflows

§ 100s to Millions of running workflows
§ Long running processes, sleeping & scheduled tasks
§ Stateful fault-tolerant applications
§ Complex workflows
§ Integration with unreliable external systems
§ Integration with streaming and event-based systems (E.g. Kafka)
§ Short workflows (100s of steps) – long workflows (1000s of steps) may take longer to
replay history
For example? Financial, retail, delivery!
Good Use Cases?

Spinning your Drones!
A Drone Delivery Application

Drone Workflow Steps
§ Charged ready to go at base
§ Get an Order to deliver
§ Fly to Order location and collect Order
§ Fly to Delivery location and drop Order
§ Fly back to base
§ Recharge
§ Repeat

Base to Order to
Delivery to Base
Example of Drone Delivery Flight

Drone Way Point Flight Calculations
§ Drone flight path is computed in an
activity
§ Using location, distance, bearing,
speed, and charge
§ Every 10 seconds
§ On failure, the drone won’t crash
and will continue flying from the last
location
Returning to Base Example

The System: Swim-lane diagrams
1st Workflow: Drone Workflow
Recharge
Start, Ready, Wait for Order, Movement Activities, Recharge, Repeat

Orders Are Also Stateful →Workflow
2nd Workflow: Order Workflow
Start, Generate locations, ready for drone, update locations, End if delivered

Orders Are Also Stateful →Workflow
2nd Workflow: Order Workflow
So we need coordination
between the workflows:
(1) Asynchronous Signaling
between workflows
1

Next, Cadence Meets Kafka
(source: https://bahumbug.wordpress.com/2014/08/03/kafka-onna-bike/)

Kafka is a distributed pub-sub streams processing
system, it allows distributed producers to send messages
to distributed consumers via a Kafka cluster. It uses
Topics to loosely couple producers and consumers.
What is
©Instaclustr Pty Limited 2019, 2021, 2022
Kafka?

Integration with Kafka adds
A Kafka Cluster with:
3 Kafka Topics
3 Kafka Producers and
2 Consumers
1

(2) Starting a workflow from Kafka
1
2

(3) Using a Kafka microservice to
coordinate Drone and Order workflows
1
3
2

Cadence Orchestration +
Kafka Choreography = Drone Ballet?
(Source: Wikimedia)

Step 1: Customer places an order—
Order Sent to Kafka New Orders Topic
1

2
Consumer
Step 2: Start New Order Workflow—
Kafka Consumer Gets Order, Starts New Order Workflow Using Cadence Client

3
Activity
Step 3: Order Ready for Drone Pickup—
Activity: Send Order Ready Message to Orders Ready Topic

4
Activity
Consumer
A Drone+Order Matching
Microservice
Step 4: Drone Workflow—
Activity: Send Drone Ready Message to Drone Ready Topic, Kafka Consumer
Gets an Order ID and Sends Signal Back to Drone to start the order pickup

5
Step 5: Drone Workflow—Fly To Order
Activity: Fly to Order Location

5
Step 5: Drone Workflow—Fly To Order
And Pickup Order

6
Step 6: Drone Workflow—Fly To Delivery Location
Activity: Fly to Delivery Location

Step 6: Drone Workflow—Fly To Delivery Location
And Drop Order; Send Location Updates to Order Workflow Every 10s
6

7
Step 7: Drone Workflow—Fly To Base
Activity: Fly to Delivery Location; recharge when back at base and start new drone workflow

8
Step 8: Order Workflow—Update Location
Receive State and Location Signal From Drone Workflow, Update State;
If Delivered Then End Workflow

Sequence…

How Many Drones Can We
Fly?

Cluster Details (VCPUS):
Client (8), Cadence (6), Cassandra (18)
6
18
8
VCPUs Per Cluster (32 total)
Cadence Cassandra Client (EC2)

Load Test 1: “Flat-Out” à
Max 50 Drones

Load Test 2: Real-Time à
2,000 Drones & 4,000 Workflows
2000 2000
4000
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Concurrent Workflows
Drones Orders Total
4000/32 = 125 workflows/core

Further Information:
Cadence Polling Cookbook
https://cadenceworkflow.io/docs/use-
cases/polling/
“MegaBurgers” polling example
Integration of Cadence with
External 3rd party REST APIs for order
state updates, using Polling

Further Information
Cadence (and other) Blogs: https://www.instaclustr.com/paul-brebner/
Try us out!

20 Drones Flying – Q&A

www.instaclustr.com
info@instaclustr.com
@instaclustr
THANK
YOU!
https://www.instaclustr.com/company/policies/terms-conditions/
Except as permitted by the copyright law applicable to you, you may
not reproduce, distribute, publish, display, communicate or transmit
any of the content of this document, in any form, but any means,
without the prior written permission of Instaclustr Pty Limited

Spinning your Drones with Cadence Workflows and Apache Kafka

Recommended

Recommended

More Related Content

Similar to Spinning your Drones with Cadence Workflows and Apache Kafka

Similar to Spinning your Drones with Cadence Workflows and Apache Kafka (20)

More from Paul Brebner

More from Paul Brebner (20)

Recently uploaded

Recently uploaded (20)

Spinning your Drones with Cadence Workflows and Apache Kafka