Webscale Workflow Engine With Kafka With Andrey Falko | Current 2022

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Webscale Workﬂow
Engine With
Apache Kafka©
Current 2022
Andrey Falko
Software Engineering Architect
https://www.linkedin.c...

This presentation contains forward-looking statements about the Company’s ﬁnancial and operating results, which may includ...

Agenda
Initial Use Case
Solution Architecture
Components
Putting It All Together: Demo
Next Steps
Webscale Workﬂow Engine ...

Initial Use Case
● Continuous Integration Systems
○ Convergence
○ Reliability and Scale
Our starting points
Webscale Workﬂ...

Initial Use Case
● Multiple Deﬁnitions
○ CIX - CI Executor
○ Jenkinsﬁle
○ Tekton
○ and others…
● Governance and Compliance...

Initial Use Case
● Our ideal
Converge CI Systems
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Initial Use Case
● 10K developers
○ 10K CI runs per hour
○ 1 hour outage very expensive
○ Delays very expensive
Scale CI S...

Generalized Use Cases
● CI systems are workﬂow processors
● Workﬂow engines require state machines
● State machines requir...

Agenda
Initial Use Case
Solution Architecture
Components
Putting It All Together: Demo
Next Steps
Webscale Workﬂow Engine ...

Solution Architecture
Big Picture
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Big Picture
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Big Picture
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Big Picture
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Big Picture
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Big Picture
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Big Picture
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Reliability Tier 0 vs. Tier 1
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Solution Architecture
Reliability Overview
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
Objective Components ...

Solution Architecture
Reliability Contrast
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
State Storage Compone...

Solution Architecture
Scale Overview
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
State Storage Main Limits S...

Agenda
Initial Use Case
Solution Architecture
Components
Kafka State Topic
Scanner
Executor
Visualization
Putting It All T...

Components
Kafka State Topic
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Components
Kafka State Topic
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Components
Workﬂow Graph
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Components
Workﬂow Key
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Agenda
Initial Use Case
Solution Architecture
Components
Kafka State Topic
Scanner
Executor
Visualization
Putting It All T...

Components
Workﬂow Scanner
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Workﬂow Scanner
First Step: Handle Request
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
1
1

Workﬂow Scanner
Second Step: Send Run ID
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
1
2
1

Workﬂow Scanner
Third Step: Download Declaration
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
1
2
1
3

Workflow Scanner
Fourth Step: Send Workflow Graph
Webscale Workflow Engine With Apache Kafka© - Andrey Falko
1
2
1
3
4

Workﬂow Scanner
On Startup
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Workﬂow Scanner
Deployment
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
● Stateless
● Scale concerns:
○ Start...

Agenda
Initial Use Case
Solution Architecture
Components
Kafka State Topic
Scanner
Executor
Visualization
Putting It All T...

Components
Executor
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Executor
First Step: Read Workﬂow Graph
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
1

Workflow Executor
Step Two: Download Workflow Definition
Webscale Workflow Engine With Apache Kafka© - Andrey Falko
1
2

Workflow Executor
Step Three: Process Workflow
Webscale Workflow Engine With Apache Kafka© - Andrey Falko
1
2
3:
Process
Work...

Workflow Executor
Step Four: Poll Workflow Progress
Webscale Workflow Engine With Apache Kafka© - Andrey Falko
1
2
3:
Process...

Workflow Executor
Step Five: Report Workflow Result
Webscale Workflow Engine With Apache Kafka© - Andrey Falko
1
2
3:
Process...

Workflow Executor
Workflow Processing
Webscale Workflow Engine With Apache Kafka© - Andrey Falko

Workflow Executor
Workflow Processing
Webscale Workflow Engine With Apache Kafka© - Andrey Falko

Workﬂow Executor
Deployment
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
● Stateless Deploy
● Scale concerns:...

Workflow Executor
Counterintuitive Behaviors
Webscale Workflow Engine With Apache Kafka© - Andrey Falko
● Only commit offset ...

Agenda
Initial Use Case
Solution Architecture
Components
Kafka State Topic
Scanner
Executor
Visualization
Putting It All T...

Solution Architecture
Visualization and Insights
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Agenda
Initial Use Case
Solution Architecture
Components
Putting It All Together: Demo
Next Steps
Webscale Workﬂow Engine ...

Putting It All Together
Demo
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko

Summary
Modest Start
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
● CI use cases
○ Polyglot workﬂow processin...

Agenda
Initial Use Case
Solution Architecture
Components
Putting It All Together: Demo
Next Steps
Webscale Workﬂow Engine ...

Next Steps
Prototype to Production
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
● Better partition assignment...

Next Steps
How can you help
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
● Open sourced prototype:
○ https://...

Takeaways
Summary
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
● Kafka compacted topics can be used to
○ Trac...

References
Further Reading
Webscale Workﬂow Engine With Apache Kafka© - Andrey Falko
● Slides and all links available here...

Thank you to
● Contributors
○ Jason Schroeder
○ Manju Prasad
● Supportive management
○ Ravinder Ramchandani
● Lots of supp...

https://github.com/salesforce/junction-workﬂow

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 56 Ad

1 of 56 Ad