Slides from my talk at QCon, OReilly, ...

1. Complex event flows in
distributed systems
@berndruecker With thoughts from http://flowing.io
@berndruecker | @martinschimak

2. 3 common hypotheses I check today:
# Events decrease coupling
# Orchestration needs to be avoided
# Workflow engines are painful

3. Berlin, Germany
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Developer Advocate of
Camunda

5. Simplified example:
dash button
Photo by 0xF2, available under Creative Commons BY-ND 2.0
license. https://www.flickr.com/photos/0xf2/29873149904/

6. Three steps…

7. Who is involved? Some bounded contexts…
Checkout
Payment
Inventory
Shipment

8. (Micro-)services
Checkout
Payment
Inventory
Shipment

9. Autonomous (micro-)services
Checkout
Payment
Inventory
Shipment
Dedicated Application Processes
Dedicated infrastructure
Dedicated Development Teams

10. Events decrease coupling

11. Example
Checkout
Payment
Inventory
Shipment
The button blinks if we can
ship within 24 hours

12. Request/response: temporal coupling
Checkout
Payment
Inventory
Shipment
Request
Response
The button blinks if we can
ship within 24 hours

13. Temporal decoupling with events and read models
Checkout
Payment
Inventory
Shipment
Good
Stored
Read
Model
Good
Fetched
The button blinks if we can
ship within 24 hours
*Events are facts about what happened (in the past)

14. Events can decrease coupling*
*e.g. decentral data-management, read models,
extract cross-cutting aspects

15. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched

16. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched

17. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

18. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

19. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

20. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment

21. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment
Customers can
pay via invoice
…

22. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment

23. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Goods
fetched
Order
placed
Payment
received
Goods
shipped

24. Photo by born1945, available under Creative Commons BY 2.0 license.

25. Order
Extract the end-to-end responsibility
Checkout
Payment
Inventory
Shipment
*Commands have an intent about
what needs to happen in the future
Order
placed
Payment
received
Retrieve
payment

26. Commands help to avoid (complex)
peer-to-peer event chains

27. Orchestration needs to be avoided

28. Smart ESB-like middleware
Checkout
Payment
Inventory
Shipment
Order
Order
placed
Payment
received
Good
fetched
Good
shipped

29. Dumb pipes
Checkout
Payment
Inventory
Shipment
Order
Smart endpoints
and dumb pipes
Martin Fowler

30. Danger of god services?
Checkout
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann
Payment
Inventory
Shipment

31. Danger of god services?
Checkout
Payment
Inventory
Shipment
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann

32. A god service is only created
by bad API design!

33. Example
Order Payment
Retrieve
Payment

34. Example
Order Payment
Credit
Card
Retrieve
Payment

35. Example
Order Payment
Credit
Card
Retrieve
Payment
Rejected

36. Example
Order Payment
Client of dumb endpoints easily become a god services.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Rejected

37. Payment
failed
Who is responsible to deal with problems?
Order Payment
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received

38. Payment
failed
Long-running execution
Order Payment
Clients of smart endpoints remains lean.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received
Smart endpoints are
potentially long-running

39. Persist thing
(Entity, Document, Actor, …)
State machine or
workflow engine
Typical
concerns
DIY = effort,
accidental
complexity
Scheduling, Versioning,
operating, visibility,
scalability, …
Handling
State

40. Workflow engines are painful
Complex, proprietary, heavyweight, central, developer adverse, …

41. Avoid the wrong tools!
Death by properties panel
Low-code is great!
(You can get rid
of your developers!)
Complex, proprietary, heavyweight, central, developer adverse, …

42. Workflow engines,
state machines
It is
relevant
in modern
architectures

43. CADENCE
Silicon valley
has recognized
Workflow engines,
state machines

45. CADENCE
Workflow engines,
state machines

46. CADENCE
also at scale
Workflow engines,
state machines

47. Zeebe.io

48. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
What do I mean by
„leightweight?“

49. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Build engine
in one line of
code
(using in-
memory H2)

50. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL

51. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL

52. BPMN
Business Process
Model and Notation
ISO Standard

53. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
We can attach
code…

54. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
…that is
called when
workflow
instances pass
through

55. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Start
instances

56. Payment
Now you have a state machine!

57. Payment
Easy to handle time

58. martin.schimak@plexiti.com
@martinschimak

59. Synchronous communication
REST
Order Payment
Credit
Card

60. Synchronous communication
REST
Order Payment
Credit
Card

61. Synchronous communication
REST
Order Payment
Credit
Card

62. Synchronous communication
REST
Order Payment
Credit
Card
Stateful
Retry

63. Distributed
systems

64. It is impossible to
differentiate certain
failure scenarios.
Independant of
communication style!
Service
Provider
Client

65. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

66. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

67. Distributed
systems

68. Distributed transactions using compensation *
Compensation

69. Relaxed consistency
Temporarily
inconsistent
state
But eventually
consistent
No Isolation
(as in ACID)

70. https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/

71. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
Distributed
Transactions
Orchestration
Communication in
distributed systems
long runningalways short running short running, but
potentially long running
Use cases for workflow automation

72. Biz Dev
improve
communication
improve
communication
Ops

73. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
improve
communication
improve
communication
Ops

74. Visual HTML reports for test cases

75. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
Understand and discuss
business processes
Evaluate optimizations
in-sync with
implementation
improve
communication
improve
communication
Ops

76. Living documentation for long-running behaviour

77. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
Understand and discuss
business processes
Evaluate optimizations
in-sync with
implementation
improve
communication
improve
communication
Ops

78. Biz Dev
Leverage
state machine &
workflow engine
Living
documentation
Visibility in
testing
Operate with visibility
and context
Understand and discuss
business processes
Evaluate optimizations
in-sync with
implementation
improve
communication
improve
communication
Ops

79. Proper
Operations
Visibility + Context

80. Before mapping processes
explicitly with BPMN, the truth was
buried in the code and nobody
knew what was going on.
Jimmy Floyd, 24 Hour Fitnesse
„

81. Workflows live inside service boundaries

82. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

83. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

84. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

85. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

86. …
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

87. Lightweight workflow engines are
great* – don‘t DIY
*e.g. enabling potentially long-running services, solving hard
developer problems, can run decentralized

88. Sales-Order & Order-Fulfillment
via Camunda
for every order worldwide
(Q2 2017: 22,2 Mio)

89. Code example & live demo
InventoryPaymentOrder ShippingCheckout Monitor
https://github.com/flowing/flowing-retail/
Human
Tasks
H2 H2

90. # Events decrease coupling: sometimes
read-models, but no complex peer-to-peer event chains!
# Orchestration needs to be avoided: sometimes
no ESB, smart endpoints/dumb pipes, important capabilities need a home
# Workflow engines are painful: some of them
lightweight engines are easy to use and can run decentralized,
they solve hard developer problems, don‘t DIY

91. Thank you!

92. bernd.ruecker@camunda.com
@berndruecker
https://bernd-ruecker.com
https://blog.bernd-ruecker.com
https://github.com/flowing
https://www.infoq.com/articles/events-
workflow-automation
With thoughts from http://flowing.io
@berndruecker | @martinschimak
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/