Checkout Payment Inventory Shipment The document discusses communication between microservices and distributed systems. It notes that event-driven architectures can decrease coupling, but peer-to-peer event chains are not suitable for complex flows. Distributed systems introduce challenges around consistency that require handling state. Workflow engines can help services collaborate by sorting out consistency issues and providing distributed orchestration owned by individual services.

1. Communication between
(micro-)services
@berndruecker With thoughts from http://flowing.io
@berndruecker | @martinschimak

2. Berlin, Germany
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Developer Advocate of
Camunda

4. Simplified example:
dash button
Photo by 0xF2, available under Creative Commons BY-ND 2.0
license. https://www.flickr.com/photos/0xf2/29873149904/

5. Three steps…

6. Who is involved? Some bounded contexts…
Checkout
Payment
Inventory
Shipment

7. (Micro-)services
Checkout
Payment
Inventory
Shipment

8. Autonomous (micro-)services
Checkout
Payment
Inventory
Shipment
Dedicated Application Processes
Dedicated infrastructure
Dedicated Development Teams

9. Events decrease coupling

10. Example
Checkout
Payment
Inventory
Shipment
The button blinks if we can
ship within 24 hours

11. Request/response: temporal coupling
Checkout
Payment
Inventory
Shipment
Request
Response
The button blinks if we can
ship within 24 hours

12. Request/response: temporal coupling
Checkout
Payment
Inventory
Shipment
Request
Response
Might be implemented
using REST, Messaging, …

13. Temporal decoupling with events and read models
Checkout
Payment
Inventory
Shipment
Good
Stored
Read
Model
Good
Fetched
The button blinks if we can
ship within 24 hours
*Events are facts about what happened (in the past)

14. Events can decrease coupling*
*e.g. decentral data-management, read models,
extract cross-cutting aspects

15. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched

16. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched

17. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

18. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

19. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

20. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment

21. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Goods
fetched
Order
placed
Payment
received
Goods
shipped

22. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Customers can
pay via invoice
Goods
fetched
Order
placed
Payment
received
Goods
shipped
…

23. Photo by born1945, available under Creative Commons BY 2.0 license.

24. Order
Extract the end-to-end responsibility
Checkout
Payment
Inventory
Shipment
*Commands have an intent about
what needs to happen in the future
Order
placed
Payment
received
Retrieve
payment

25. Orchestration needs to be avoided

26. Smart ESB-like middleware
Checkout
Payment
Inventory
Shipment
Order
Order
placed
Payment
received
Good
fetched
Good
shipped

27. Dumb pipes
Checkout
Payment
Inventory
Shipment
Order
Smart endpoints
and dumb pipes
Martin Fowler

28. Danger of god services?
Checkout
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann
Payment
Inventory
Shipment

29. Danger of god services?
Checkout
Payment
Inventory
Shipment
Order
A few
smart god services
tell
anemic CRUD services
what to do
Sam Newmann

30. A god service is only created
by bad API design!

31. Example
Order Payment
Retrieve
Payment

32. Example
Order Payment
Credit
Card
Retrieve
Payment

33. Example
Order Payment
Credit
Card
Retrieve
Payment
Rejected

34. Example
Order Payment
Client of dumb endpoints easily become a god services.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Rejected

35. Payment
failed
Who is responsible to deal with problems?
Order Payment
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received

36. Payment
failed
Who is responsible to deal with problems?
Order Payment
Clients of smart endpoints remains lean.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received
Smart endpoints are
potentially long-running

37. Photo by Tookapic, available under Creative Commons CC0 1.0 license.

38. „There was an error
while sending your
boarding pass“

39. Check-in
Web-UI
Me
Current situation

40. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation

41. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation

42. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation – the bad part

43. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation – the bad part
Stateful
Retry

45. We are having some technical
difficulties and cannot present you
your boarding pass right away.
But we do actively retry ourselves, so
lean back, relax and we will send it
on time.

46. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Possible situation – much better!

47. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Possible situation – much better!
Stateful
Retry

48. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Stateful
Retry
Possible situation – much better!
The failure
never leaves
this scope!

49. Persist thing
(Entity, Actor, …)
State machine or
workflow engine
Typical
concerns
DIY = effort,
accidental
complexity
Scheduling, Versioning,
operating, visibility,
scalability, …
Handling
State

50. Workflow engines are painful
Complex, proprietary, heavyweight, central, developer adverse, …

51. Avoid the wrong tools!
Death by properties panel
Low-code is great!
(You can get rid
of your developers!)
Complex, proprietary, heavyweight, central, developer adverse, …

52. Workflow engines,
state machines
It is
relevant
in modern
architectures

53. CADENCE
Silicon valley
has recognized
Workflow engines,
state machines

54. CADENCE
Workflow engines,
state machines

55. CADENCE
also at scale
Workflow engines,
state machines

56. CADENCE
for todays demo
Workflow engines,
state machines

57. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
What do I mean by
„leightweight?“

58. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Build engine
in one line of
code
(using in-
memory H2)

59. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL

60. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL

61. BPMN
Business Process
Model and Notation
ISO Standard

62. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
We can attach
code…

63. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
…that is
called when
workflow
instances pass
through

64. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Start
instances

65. Payment
Now you have a state machine!

66. Payment
Easy to handle time

67. Stateful retries
Payment
Credit
Card
REST

68. has to implement
Retry
has to implement
Idempotency
Client Service Provider

69. Distributed
systems

70. It is impossible to
differentiate certain
failure scenarios.
Independant of
communication style!
Service
Provider
Client

71. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

72. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

73. Distributed
systems

74. Distributed transactions using compensation *
Compensation

75. begin commit
{local TX}
Customer Credit Service
begin commit
{local TX}
Customer Credit Service
begin commit
{local TX}
Credit Card Service
Saga
Coordinator
A workflow engine can serve as Saga coordinator

76. Eventual consistency
Temporarily
inconsistent state
But only
temporary
No Isolation
(as in ACID)

77. Apologize!
https://blogs.msdn.microsoft.com/pathelland/2007/05/15/memories-guesses-and-apologies/

78. has to implement
Timeout, Retry,
Compensation
has to offer
Compensation
has to implement
Idempotency
Client Service Provider

79. Workflows live inside service boundaries

80. Before mapping processes
explicitly with BPMN, the truth was
buried in the code and nobody
knew what was going on.
Jimmy Floyd, 24 Hour Fitnesse
„

81. Architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

82. Architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

83. Architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

84. Architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

85. …
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

87. Zalando
Sales-Order and
Order-Fulfillment
via Camunda
for every order worldwide
Orders Q2-2017: 22,2 Mio.
Sales 2016: 3,6 Mrd. EUR
Growth 2016: 23%

88. Code example
InventoryPaymentOrder ShippingCheckout Monitor
https://github.com/flowing/flowing-retail/
Human
Tasks
H2 H2

89. Wrap-up
Checkout
Payment
Inventory
Shipment
Event-driven is often benefitial, but don‘t do
peer-to-peer event chains for complex flows
A lot of poblems exist independant of sync/async
communication (just the stack differs)
Getting your services to collaborate and sort out
consistency is the challenge. There is a lot of need
for state handling. Workflow engines can help.
Don‘t forget about orchestration. It should be
distributed and owned by services

90. Wrap-up
Checkout
Payment
Inventory
Shipment
Event-driven is often benefitial, but don‘t do
peer-to-peer event chains for complex flows
A lot of poblems exist independant of sync/async
communication (just the stack differs)
Getting your services to collaborate and sort out
consistency is the challenge. There is a lot of need
for state handling. Workflow engines can help.
Don‘t forget about orchestration. It should be
distributed and owned by services

91. Wrap-up
Checkout
Payment
Inventory
Shipment
Event-driven is often benefitial, but don‘t do
peer-to-peer event chains for complex flows
A lot of poblems exist independant of sync/async
communication (just the stack differs)
Getting your services to collaborate and sort out
consistency is the challenge. There is a lot of need
for state handling. Workflow engines can help.
Don‘t forget about orchestration. It should be
distributed and owned by services
Order

92. Wrap-up
Checkout
Payment
Inventory
Shipment
Event-driven is often benefitial, but don‘t do
peer-to-peer event chains for complex flows
A lot of poblems exist independant of sync/async
communication (just the stack differs)
Getting your services to collaborate and sort out
consistency is the challenge. There is a lot of need
for state handling. Workflow engines can help.
Don‘t forget about orchestration. It should be
distributed and owned by services
Order

93. Wrap-up
Checkout
Payment
Inventory
Shipment
Event-driven is often benefitial, but don‘t do
peer-to-peer event chains for complex flows
A lot of poblems exist independant of sync/async
communication (just the stack differs)
Getting your services to collaborate and sort out
consistency is the challenge. There is a lot of need
for state handling. Workflow engines can help.
Don‘t forget about orchestration. It should be
distributed and owned by services
Order
Stateful
retry
Long running
orchestration
Saga
Timeouts

94. Thank you!

95. bernd.ruecker@camunda.com
@berndruecker
https://bernd-ruecker.com
https://blog.bernd-ruecker.com
https://github.com/flowing
https://www.infoq.com/articles/events-
workflow-automation
With thoughts from http://flowing.io
@berndruecker | @martinschimak
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/