QCon London: Mastering long-running processes in modern architectures
•
0 likes•196 views
Slides from my talk at QCon London on 9th of April 2024
Recommended
Recommended
More Related Content
Similar to QCon London: Mastering long-running processes in modern architectures
Similar to QCon London: Mastering long-running processes in modern architectures (20)
More from Bernd Ruecker
More from Bernd Ruecker (20)
Recently uploaded
Recently uploaded (20)
QCon London: Mastering long-running processes in modern architectures
- 1. @berndruecker Are you done yet? Mastering long-running processes in modern architectures
- 2. Let‘s talk about food
- 3. How does ordering Pizza work? Pizza Place You Phone Call Synchronous blocking communication Feedback loop (ack, confirmation or rejection) Temporal coupling (e.g. busy, not answering) Pizza Place You Email Asynchronous non-blocking communication No temporal coupling Pizza Place You A feedback loop might make sense (ack, confirmation or rejection) Email Confirmation Email @berndruecker
- 4. Feedback loop != result Pizza Place You Email Confirmation Email Pizza Delivery Feedback (ACK, confirmation, rejection) Result @berndruecker
- 5. Only the first communication step is synchronous / blocking Pizza Place You PUT/order HTTP 200 Pizza Delivery The task of Pizza making is long running @berndruecker
- 6. Synchronous blocking behavior for the result? Bad user experience Does not scale well @berndruecker
- 7. Scalable Coffee Making https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html Photo by John Ingle @berndruecker
- 8. Long running
- 9. Long running
- 10. Long running basically means waiting
- 11. When do services want to wait? Some business reasons… Human work Waiting for response Let some time pass
- 12. Why is waiting a pain? Persistent state How to solve the technical challenges without adding accidential complexity? Monitoring & Operations Visibility Versioning Scheduling & Timeouts Domain Logic Scalability & Resilience
- 14. bernd.ruecker@camunda.com @berndruecker http://berndruecker.io/ Bernd Ruecker Co-founder and Chief Technologist of Camunda
- 15. Workflow Engine aka Process Engine aka Orchestration Engine
- 18. @berndruecker
- 20. There are also technical reasons why services need to wait…
- 21. When do services need to wait? Some technical reasons… Asynchronous communication Wait for responses Wait for availability Especially failure scenarios Unavailability of peers
- 23. Photo by Tookapic, available under Creative Commons CC0 1.0 license.
- 24. „There was an error while sending your boarding pass“
- 28. Some Service Some Service Some Service Some Service Some Service Some Service Some Service Failure will happen. Accept it! But keep it local! Be resilient.
- 29. Check-in Barcode Generator Web-UI Me Output Mgmt Current situation – the bad part
- 30. Check-in Barcode Generator Web-UI Me Output Mgmt Current situation – the bad part
- 31. Check-in Barcode Generator Web-UI Me Output Mgmt Current situation – the bad part Stateful Retry
- 34. Check-in Barcode Generator Web-UI Me Output Mgmt Possible situation – much better! Stateful Retry
- 35. Check-in Barcode Generator Web-UI Me Output Mgmt Stateful Retry Possible situation – much better! The failure never leaves this scope!
- 36. „But the customer wants a synchronous response!“ Check-in Barcode Generator Web-UI Me Output Mgmt „Eh – no!“
- 37. Let‘s add some money… @berndruecker Booking Payment Credit Card
- 38. Payment Stateful retry by using an orchestration engine Credit Card REST
- 39. Distributed systems introduce more complexity Credit Card Payment REST
- 40. It is impossible to differentiate certain failure scenarios. Independant of communication style! Service Provider Client
- 41. Distributed systems introduce more complexity Credit Card Payment REST
- 42. Distributed systems introduce more complexity Credit Card Payment REST Cancel charge Using an orchestration engine allows to do long running processes reliably, and end-to-end
- 43. retrievePayment HTTP 200 OK HTTP 202 ACCEPTED Payment Happy case: Synchronous response Otherwise: asynchronous Embrace asynchronicity
- 44. This allows to extend payment Options @berndruecker
- 45. Further requirements when extending payment Options begin commit {local TX} Customer Credit Service begin commit {local TX} Credit Card Service @berndruecker
- 46. Distributed transactions using compensation * Compensation @berndruecker
- 47. If you want to dive into code… https://github.com/berndruecker/flowing-retail
- 48. Long running capabilities are essential to design good service boundaries (= a good architecture) @berndruecker
- 52. Example Booking Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker ???
- 53. Payment failed Who is responsible to deal with problems? Booking Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received @berndruecker
- 54. Payment failed (Potentially) long running services Booking Payment Credit Card Retrieve Payment Rejected Payment received @berndruecker
- 55. Long running services Booking Payment Credit Card @berndruecker
- 56. Orchestration != monolithic processes Booking Payment Credit Card @berndruecker
- 57. Long running avoids „god services“ Booking Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected @berndruecker A few smart god services tell anemic CRUD services what to do Sam Newmann
- 58. Long running capabilities are essential to design good service boundaries (= a good architecture) @berndruecker
- 59. Being able to implement long running services makes it easy to distribute responsibilities correctly @berndruecker
- 60. Being able to implement long running services makes it easy to embrace async/non-blocking @berndruecker
- 61. You need a process orchestration capability that is available as-a-service within your organization to create competitive architectures
- 62. What we see most successful: Center of Excellence https://camunda.com/process-orchestration/automation-center-of-excellence/
- 63. Focus: Enablement and platform Business Unit or Domain Business Unit or Domain Business Unit or Domain Process Automation Center of Excellence Solution Delivery Solution Delivery Solution Delivery Solution Delivery Solution Delivery enable
- 64. But isn’t a central CoE harming team autonomy? (This is a rhetorical question - the answer is of course NO)
- 65. @berndruecker
- 68. Golden Paths https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem/ “We found that rumour-driven development simply wasn’t scalable” @berndruecker
- 69. https://backstage.io/ (OSS, Made with at Spotify) @berndruecker
- 70. At Twilio, we do it by offering what we call the paved path. These are mature services that you can just pull off the shelf, adopt, and get up and running super quickly. But if a team wants to diverge from that path they can make that decision. You just still have to pass the same rigorous bar of quality, reliability, and security. So what you do is you create the incentive structure for teams to take the paved path. It’s a lot easier. But if they really have to go a different route, you make it possible for them to do that.
- 72. BPMN Business Process Model and Notation ISO Standard @berndruecker
- 73. Living documentation for long-running behaviour @berndruecker
- 74. Living documentation for long-running behaviour @berndruecker
- 76. Decisions about long running behavior need to be elevated to the business level
- 77. You need to redesign your customer journeys to leverage your hipster architecture Shutterstock
- 87. Yeah - everything asynchronously :-)
- 88. Recap • You need capabilities for long running behavior for technical and business reasons • Process orchestration platforms are a great fit • This allows you to design better service boundaries, implement quicker with less accidental complexity, embrace asynchronoucity and provide a better customer experience (plus increase operational efficiency and reduces risk) • Organize central enablement (center of excellence, platform as-a- service) for easy adoption at scale @berndruecker
- 89. Want to learn more? http://camunda.com/