Integrating microservices and distributed systems is hard. In this talk I will present three challenges we observed in real-life projects and how to avoid them. 1. Communication is complex. With everything being distributed failures are normal so you need sophisticated failure handling. 2. Asynchronicity requires you to handle timeouts. This is not only about milliseconds--systems get much more resilient when you can wait for minutes or hours. 3. Distributed transactions cannot simply be delegated to protocols like XA. I will not only use slides but demonstrate concrete source code examples.

1. 3 Common Pitfalls in
Microservice Integration and
How to Avoid Them
@berndruecker

2. REST, SOAP,
Cloud, Saas,
Microservices, SCS,
FaaS, Serverless,
…

3. Distributed systems

4. https://www.infoworld.com/article/3254777/application-development/
3-common-pitfalls-of-microservices-integrationand-how-to-avoid-them.html

5. Distributed systems
Challenges of
asynchronicity
Distributed
Transactions
Remote
Communication

6. Warning:
Contains Opinion

7. Berlin, Germany
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Developer Advocate of
Camunda

8. Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Failure will happen.
Accept it!
But keep it local!
Be resilient.

9. Let‘s start with a simple example
Credit
Card
Payment
REST

10. Live hacking
https://github.com/berndruecker/flowing-retail

11. Circuit
Breaker
Photo by CITYEDV, available under Creative Commons CC0 1.0 license.

12. Fail fast
is important

13. Fail fast
is important
but not enough!

14. Photo by Tookapic, available under Creative Commons CC0 1.0 license.

15. „There was an error
while sending your
boarding pass“

16. Check-in
Web-UI
Me
Current situation

17. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation

18. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation

19. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation
Circuit
breaker

20. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation – the bad part

21. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation – the bad part

22. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Current situation – the bad part
Stateful
Retry

24. We are having some technical
difficulties and cannot present you
your boarding pass right away.
But we do actively retry ourselves, so
lean back, relax and we will send it
on time.

25. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Possible situation – much better!

26. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Possible situation – much better!
Stateful
Retry

27. Persist thing
(Entity, Document, Actor, …)
State machine or
workflow engine
Typical
concerns
DIY = effort,
accidental
complexity
Complex, proprietary,
heavyweight, slow,
don‘t scale,
developer adverse
Scheduling, Versioning,
operating, visibility,
scalability, …
Handling
State

28. Workflow engines,
state machines
It is
relevant
in modern
architectures

29. CADENCE
Silicon valley
has recognized
Workflow engines,
state machines

30. CADENCE
Workflow engines,
state machines

31. CADENCE
also at scale
Workflow engines,
state machines

32. CADENCE
for todays demo
Workflow engines,
state machines

33. Live hacking
https://github.com/berndruecker/flowing-retail

34. Payment
Now you have a state machine!
Credit
Card
REST

35. has to implement
Retry
has to implement
Idempotency
Client Service Provider

38. Don‘t worry, it will happen safely –
even if you loose connection.
Feel free to reload this page any time!

39. Photo by pixabay, available under Creative Commons CC0 1.0 license.

40. Requirement: Idempotency of services!
Photo by pixabay, available under Creative Commons CC0 1.0 license.

41. Requirement: Idempotency of services!
Photo by Chr.Späth, available under Public Domain.

42. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

43. Distributed
systems

44. It is impossible to
differentiate certain
failure scenarios.
Independant of
communication style!
Service
Provider
Client

45. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

46. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST
Cancel
charge

47. BPMN
Business Process
Model and Notation
ISO Standard

48. Proper
Operations
Visibility + Context

49. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

50. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Service A Service B
WF-Engine WF-Engine
Spring Boot Spring Boot

51. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Infrastructure
Service A Service B
WF-Engine WF-Engine
Wildfly WildFly
WARWAR

52. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Service A
WF-Engine
C#
Client
Service B
WF-Engine
Node.js
Client

53. Synchronous communication

54. Synchronous communication
is the crystal meth of
distributed programming
Todd Montgomery and Martin Thompson
in “How did we end up here” at GOTO Chicago 2015

55. Challenges of
asynchronicity

56. Check-in
Barcode
Generator
Web-UI
Me
Output
Mgmt
Asynchronous communication
You need to
monitor
timeouts

57. Workflow…

58. Workflow…

59. Who uses a message bus?

60. Who has no problems
operating a message bus?
Dead messages | No context | Inaccesible payload | Hard to redeliver |
Home-grown message hospitals | …

61. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

62. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

63. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

64. Distributed
Transactions
Distributed
Transactions

65. ACID (Atomic Consistent Isolation Durability)
Do A
Do B
+ All or
nothing
„Lost in transaction“
https://vimeo.com/289508460
@berndruecker

66. Distributed
systems

67. Pat Helland
Grown-Ups Don’t Use
Distributed Transactions
“
Distributed Systems Guru
Worked at Amazon,
Microsoft & Salesforce

68. Distributed transactions using compensation *
Compensation

69. Live hacking
https://github.com/berndruecker/flowing-retail

70. has to implement
Timeout, Retry,
Compensation
has to offer
Compensation
has to implement
Idempotency
Client Service Provider

71. has to implement
Timeout, Retry,
Compensation
has to offer
Compensation
has to implement
Idempotency
Client Service ProviderDon‘t forget
about state

72. Event-driven example also available
InventoryPaymentOrder ShippingCheckout Monitor
https://github.com/berndruecker/flowing-retail/
Human
Tasks
H2 H2

73. Workflows live inside service boundaries

75. Sales-Order & Order-Fulfillment
via Camunda
for every order worldwide
(Q2 2017: 22,2 Mio)

76. https://blog.bernd-ruecker.com/the-microservice-workflow-automation-cheat-sheet-fc0a80dc25aa

77. # Be aware of complexity of distributed systems
# Know strategies and tools to handle it
e.g. Circuit breaker (Hystrix)
Workflow engine for stateful retry, waiting, timeout
and compensation (Camunda)

78. Thank you!

79. mail@berndruecker.io
@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker
https://www.infoq.com/articles/events-
workflow-automation
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/