Zeebe is a new open source microservice orchestration engine that simplifies defining and executing workflows across microservices. It handles failures and coordination through a distributed architecture based on event streaming. The project is currently in a tech preview stage with plans to add more capabilities and achieve production readiness over the coming months.

1. Zeebe - Microservice Orchestration Engine

2. Jakob Freund
Co-Founder and CEO
Daniel Meyer
VP Engineering
Darya Niknamian
Content
Marketing Manager

3. • 140 customers in 18 countries
• > 85% of customers use Camunda for core business processes
• Company revenue in growth 2016: 82%
• > 70 Employees
Camunda
AT&T

4. Meet Zeebe
Today, we introduce “Zeebe” a new Microservice Orchestration Engine.
Zeebe is Open Source and available at
• https://zeebe.io (Website)
• https://github.com/zeebe-io (Github)

5. Outline
How Zeebe simplifies Microservice Orchestration
Architecture & Internals
Performance
Project Status & Roadmap
Community
Q&A

6. Order Execution Example
Webshop
New Orders
Payment,
Shipment, Billing,
Backend, …

7. Challenges
Webshop
New Orders
Payment,
Shipment, Billing,
Backend, …
Complex
Long-Running,
Asynchronous,
Mission-Critical

8. Dealing with Complexity
Large, monolithic
System
Distributed Microservice Architecture
Industry Trend towards Microservice Architecture

9. Dealing with Complexity
Large, monolithic
System
Distributed Microservice Architecture
Industry Trend towards Microservice Architecture

10. Webshop
New Order Order
Inbound
Service
Payment Service
Inventory Service
Shipment Service
Rest
Our Teams‘ task: build „Order Inbound Service“

11. Webshop
New Order Order
Inbound
Service
Payment Service
Inventory Service
Shipment Service
Rest
Our Team‘s task: build „Order Inbound Service“
processNewOrder(…)
{
executePayment(…);
fetchItems(…);
triggerParcelShipment(…);
}

12. Webshop
New Order Order
Inbound
Service
Payment Service
Inventory Service
Shipment Service
Rest
What if „fetchItems(…)“ fails?
processNewOrder(…)
{
executePayment(…);
fetchItems(…);
triggerParcelShipment(…);
}

13. Uber Cadence

14. Visibility, monitoring and operations?
Using microservices turns our applications into distributed systems and along with it becomes
more and more difficult to get visibility into the system as a whole and to answer questions like
• How many orders are currently being processed and where are they pending?
• What is the state of a particular order?
• How can I fixup the payload related to an order and restart it?

15. Netflix Conductor
“[…]
as the number of
microservices grow and
the complexity of the
processes increases,
getting visibility into
these distributed
workflows becomes
difficult without a
central orchestrator.”

16. Amazon Step Functions & Simple Workflow Service

17. Zeebe‘s Solution
1 2
Define
Orchestration
Workflow using BPMN
Implement
Service Invocations
or Business Logic
Relax
And let Zeebe take care
of the heavy lifting

18. Demo

19. Outline
How Zeebe simplifies Microservice Orchestration
Architecture & Internals
Performance
Project Status & Roadmap
Community
Q&A

20. Client / Server
Broker
Broker
Broker
Broker
Broker
Zeebe
Cluster
Zeebe
Client Connect
My App
Golang

21. Distributed System
Peer to Peer
No single point of failure
No single choking point
Broker
Broker
Broker
Broker
Broker
Zeebe
Cluster

22. Scale Elastically
UP
DOWN
3 Nodes 30 Nodes
SCALE

23. Cloud Native
$ docker service scale zeebe=10

24. Topics & Partitions
- For internal processing, Zeebe applies event sourcing
- Data is organized into topics.
- Topics are partitioned for scalability.
- Partitions are log-structured
Partition
head
tailentry
0 1 2 5
write (append-only)
position
Inspiration: Apache Kafka

25. Fault Tolerance & Data Replication
Broker
Broker
Broker
Replicate Data
Zeebe
Client
Replication uses the Raft protocol

26. Distribution
P1: F
P2 : F
P3 : L
P1: L
P2 : F
P3 : F
P1: F
P2 : L
P3 : F
Example:
- Topic with 3 partitions
and replication factor 3
- PX = Partition X
- L = Leader
- F = Follower

27. Data Flow (simplified)
Client
COMPLETE
command
Broker - Partition Leader
Brokers -
Partition Followers
1
2 append
replicate &
commit
3
4 process
Stream Processor
State
Partition
5
decide & applyCOMPLETED
response
6

28. No external services needed
Zeebe does not rely on any external services like databases, queues or caches.

29. Orchestration of Microservices
Zeebe
Broker
Microservices
Distribute Work

30. Publish / Subscribe
Zeebe
Broker
Microservice
Zeebe
Client
subscribe
Inspiration: Messaging Systems!
Execute BPMNExecute Business Logic

31. Scenarios
Broker
Service
1
Client
subscribe
Service
2
Client
subscribe
Scenario 1:
Services subscribe directly to broker
Scenario 2:
Orchestration Service subscribes to broker
and calls services via rest
Broker
Service 1
Client
subscribe
Service 2
Orchestration Service
rest
rest

32. Outline
How Zeebe simplifies Microservice Orchestration
Architecture & Internals
Performance
Project Status & Roadmap
Community
Q&A

33. Zeebe is designed for Performance
Design principles:
• Batching of I/O operations
• Linear read/write data access patterns
• Compact, cache-optimized data structures
• Lock free algorithms and actor concurrency (green threads model)
• Broker is garbage-free in the hot/data path
Results: 160K – 200K events/s on modern SSDs & Gbit Ethernet
Note: Performance regressions in the 0.1.0 release, will be improved for 0.2.0 (End of August)

34. Outline
How Zeebe simplifies Microservice Orchestration
Architecture & Internals
Performance
Project Status & Roadmap
Community
Q&A

35. Project Status
Zeebe is currently in a development state that we call "tech preview". This means that:
• Zeebe is under heavy development and refactoring
• The "initial" scope of the system has not been developed completely yet
• Zeebe is not production-ready

36. What is currently available?
- Single Topic with a single partition
- Failover & replication
- Task Subscriptions & Event Subscriptions
- BPMN Workflows:
- Sequences of tasks
- Data mappings in tasks
- Incidents

37. Roadmap
Milestone 1 (Planned for End of September 2017)
Achieve Scalability through partitioning
Define topics with multiple partitions
Deploy workflows to multiple partitions
Partition-aware subscriptions
BPMN Exclusive Gateway and Conditions
Milestone 2 (Planned for End of December 2017)
Achieve production-readiness:
Checksums in data fragments to detect data corruption
Log Compaction
Long-running operations testsuite
Failure cases testsuite

38. Outline
How Zeebe simplifies Microservice Orchestration
Architecture & Internals
Performance
Project Status & Roadmap
Community
Q&A

39. How can you participate?
Zeebe is still in its early stages but there are already different ways you can contribute:
• Try out Zeebe and provide feedback in the forums https://forum.zeebe.io
• Contribute to Zeebe
• Java Framework Integrations (Spring, Spring Boot, RxJava, …)
• Client libraries in different Languages
• Documentation
• .deb/.rpm packages for Linux

40. Outline
How Zeebe simplifies Microservice Orchestration
Architecture & Internals
Performance
Project Status & Roadmap
Community
Q&A