Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lost in transaction - Strategies to deal with (in)consistency in distributed systems

449 views

Published on

Slides from my talk held end of 2018 e.g. at FullStackFest Barcelona or JavaZone Oslo.

Published in: Technology
  • Be the first to comment

Lost in transaction - Strategies to deal with (in)consistency in distributed systems

  1. 1. @berndruecker Lost in transaction? Strategies to deal with (in)consistency in distributed systems
  2. 2. Do A Do B All or nothing + try { tx.begin(); doA(); doB(); tx.commit(); } catch (Exception e) { tx.rollback(); } @Transactional public void createCustomer(Customer cust) { // ... } Or simply: Once upon a time:
  3. 3. A C I D Atomicity Consistency Isolation Durability
  4. 4. Distributed systems
  5. 5. Distributed systems
  6. 6. Distributed systems
  7. 7. But there is two-phase commit (XA)!! TX Coordinator Resource Managers Prepare Phase Commit Phase
  8. 8. Pat Helland “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
  9. 9. Pat Helland Grown-Ups Don’t Use Distributed Transactions “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
  10. 10. Starbucks does not use two phase commit https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html Photo by John Ingle
  11. 11. Eric Brewer Atomicity Consistency Isolation Durability http://pld.cs.luc.edu/courses/353/spr11/notes/brewer_keynote.pdf
  12. 12. That means Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Local ACID Local ACID 1 (micro-)service 1 aggregate 1 program 1 resource Violates „I“ of ACID
  13. 13. Youmightknowthisfrom: Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Photo by Gerhard51, available under Creative Commons CC0 1.0 license.
  14. 14. „Building on Quicksand“ Paper A C I D 2.0Pat Helland
  15. 15. Associative Commutative Idempotent Distributed 2.0 (a + b) + c = a + (b + c) a + b = b + a f(x) = f( f(x) ) „Building on Quicksand“ Paper Pat Helland
  16. 16. Requirement: Idempotency of services! Photo by pixabay, available under Creative Commons CC0 1.0 license.
  17. 17. Requirement: Idempotency of services! Photo by Chr.Späth, available under Public Domain.
  18. 18. Associative Commutative Idempotent Distributed 2.0 (a + b) + c = a + (b + c) a + b = b + a f(x) = f( f(x) ) „Building on Quicksand“ Paper Pat Helland
  19. 19. Distributed
  20. 20. It is impossible to differentiate certain failure scenarios: Independant of communication style! Service Provider Client
  21. 21. Network problems Credit Card Payment charge
  22. 22. Strategy: retry Credit Card Payment Charge Credit Card cardNumber amount Charge Credit Card cardNumber amount transactionId Not idempotent Idempotent has to be idempotent charge
  23. 23. Strategy: Cleanup Credit Card Payment charge Make sure it is not charged! Cancel charge cardNumber amount transactionId Raise payment failed
  24. 24. Some communication challenges require state.
  25. 25. Strategy: Stateful retry Credit Card Payment charge
  26. 26. Strategy: Stateful retry Credit Card Payment charge Make sure it is not charged!
  27. 27. Warning: Contains Opinion
  28. 28. Berlin, Germany bernd.ruecker@camunda.com @berndruecker Bernd Ruecker Co-founder and Developer Advocate of Camunda
  29. 29. Let‘s use a lightweight OSS workflow engine for this:
  30. 30. Payment Stateful retry Credit Card REST
  31. 31. Stateful retry & cleanup Credit Card Payment REST
  32. 32. Live hacking https://github.com/flowing/flowing-retail/tree/master/rest
  33. 33. Embedded Engine Example (Java) https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
  34. 34. Remote Engine Example (Polyglot) https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
  35. 35. A relatively common pattern Service (e.g. Go) Kafka / Rabbit RDMS 1. Receive 4. Send additional events 2. Business Logic 3. Send response? ACK
  36. 36. Out-of-order messages https://www.enterpriseintegrationpatterns.com/patterns/messaging/Resequencer.html
  37. 37. That means Do A Do B Temporarily inconsistent Eventually consistent In case of failure: Compensate or apologise t Consistent
  38. 38. https://blogs.msdn.microsoft.com/pathelland/2007/05/15/memories-guesses-and-apologies/
  39. 39. Compensation – the classical example Saga book hotel book car book flight cancel hotel cancel car 1. 2. 3. 5.6. In case of failure trigger compensations book trip
  40. 40. 2 alterntive approaches: choreography & orchestration
  41. 41. Event-driven choreography Hotel Flight Car Trip Trip booked Flight booked Trip requested Hotel booked Car booked Request trip
  42. 42. Event-driven choreography Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled Perform undo (cancel car booking) Perform undo (cancel hotel)
  43. 43. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
  44. 44. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
  45. 45. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
  46. 46. Classical example Saga book hotel book car book flight cancel hotel cancel car 1. 2. 3. 5.6. In case of failure trigger compensations book trip
  47. 47. If your transaction involves 2 to 4 steps, choreography might be a very good fit. However, this approach can rapidly become confusing if you keep adding extra steps in your transaction as it is difficult to track which services listen to which events. Moreover, it also might add a cyclic dependency between services as they have to subscribe to one another’s events. Denis Rosa Couchbase https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/
  48. 48. Microservice pioneers have become aware
  49. 49. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first!
  50. 50. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled You have to adjust all services and redeploy at the same time! We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first!
  51. 51. Photo by born1945, available under Creative Commons BY 2.0 license.
  52. 52. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0
  53. 53. Orchestration Hotel Flight Car Trip Trip booked Request trip Book hotel Hotel booked Car booked Flight booked Book car Book flight
  54. 54. Orchestration Hotel Flight Car Trip Trip booked Request trip Book hotel Hotel booked Car booked Flight booked Book car Book flight We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first! You have to adjust one service and redeploy only this one!
  55. 55. Describe orchestration with BPMN Trip Trip booked Request trip
  56. 56. The workflow is part of the service Trip
  57. 57. The workflow is part of the service Trip Payment
  58. 58. The workflow is part of the service Trip Payment Workflow engine itself can run decentralized or centralized
  59. 59. Caitie McCaffrey | @caitie
  60. 60. Saga Execution Coordinator (SEC) Hotel Flight Car Trip Trip booked Request trip SEC
  61. 61. Saga Execution Coordinator (SEC) Hotel Flight Car Trip SEC Requirement: Resilient and scalable Owned by executed by
  62. 62. Saga Execution Coordinator (SEC) Hotel Flight Car Trip SEC Requirement: SEC is resilient and scalable
  63. 63. „Can this really handle 15k requests per second?“
  64. 64. Graphical models?
  65. 65. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
  66. 66. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
  67. 67. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
  68. 68. BPMN Business Process Model and Notation ISO Standard
  69. 69. Living documentation for long-running behaviour
  70. 70. Visual HTML reports for test cases
  71. 71. BizDevOps
  72. 72. Saga with AWS Step Functions https://theburningmonk.com/2017/07/applying-the- saga-pattern-with-aws-lambda-and-step-functions/
  73. 73. Visibility gets easy when you know where to look
  74. 74. Thoughts on the state machine | workflow engine market
  75. 75. Thoughts on the state machine | workflow engine market OSS Workflow or Orchestration Engines Stack Vendors, Pure Play BPMS Low Code Platforms Homegrown frameworks to scratch an itch Integration Frameworks Cloud Offerings Uber, Netflix, AirBnb, ING, … AWS Step Functions, Azure Durable Functions, … Camunda, Zeebe, jBPM, Activiti, Mistral, … PEGA, IBM, SAG, … Apache Airflow, Spring Data Flow, … Apache Camel, Balerina, … Data Pipelines
  76. 76. Does it support stateful operations? Does it support the necessary flow logic? Does it support BizDevOps? Does it scale?
  77. 77. My personal pro-tip for a shortlist ;-) OSS Workflow or Orchestration Engines Stack Vendors, Pure Play BPMS Low Code Platforms Homegrown frameworks to scratch an itch Integration Frameworks Cloud Offerings Data Pipelines Camunda & Zeebe
  78. 78. Recap • Grown ups don‘t use distributed transactions but eventual consistency • Idempotency is super important in distributed systems • Some consistency challenges require state • Know some strategies • Stateful retry & cleanup • Saga / Compensation • Apologies
  79. 79. Thank you!
  80. 80. bernd.ruecker@camunda.com @berndruecker https://bernd-ruecker.com https://blog.bernd-ruecker.com https://github.com/flowing https://www.infoq.com/articles/events- workflow-automation With thoughts from http://flowing.io @berndruecker | @martinschimak Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/

×