Download as PDF, PPTX
Uploaded byBilgin Ibryam
Dual write strategies for microservices
The document discusses various dual-write strategies for microservices, focusing on challenges and solutions for updating multiple resources reliably. It outlines options such as modular monolith, two-phase commit, orchestration, choreography, and parallel pipelines, highlighting their benefits and drawbacks. Each strategy provides different approaches to managing distributed transactions and ensuring data consistency in complex applications.
More Related Content
![[KubeCon EU 2022] Running containerd and k3s on macOS](https://cdn.slidesharecdn.com/ss_thumbnails/lima-220519142933-3d747f68-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Dual write strategies for microservices
Dual write strategies for microservices
- 1. Dual Write Strategiesfor Microservices Bilgin Ibryam @bibryam Product Manager Red Hat
- 2. About me 2 ▸ ProductManager at Red Hat ▸ Former Principal Architect ▸ Committer at Apache Camel ▸ Author ･ Camel Design Patterns ･ Kubernetes Patterns ▸ @ bibryam ▸ https://www.ofbizian.com
- 3. There is acatch 3 Challenges with legacy applications ▸ Frequent deployment is difficult ▸ Obstacles to scaling development ▸ Scaling the application can be difficult Modern application benefits ▸ Greater team autonomy ▸ Reliability, scalability, and other abilities ▸ Reduce time to market
- 4. The dual-write problem 4 Dual-writechallenge behind the scene ▸ Updating more than one resources ▸ Reliable inter-service communication ▸ Coordinating long-running, business transactions ▸ Recovery from failed distributed transaction ▸ Implementing idempotent operations ▸ Scaling with increasing data volume
- 5.
- 6.
- 7. Modular monolith 7 Challenges ▸ Strongdata consistency requirements ▸ Strict performance requirements ▸ Complex business transactions Solution ▸ Single process deployment unit ▸ Independent modules with clear responsibilities and well-defined interfaces/contracts ▸ Each module access only its tables
- 8. Implementation options 8 Diagram credits@axelfontaine
- 9. Modular monolith 9 Benefits ▸ Simpletransaction semantics with local transactions ▸ ACID properties Drawbacks ▸ Shared runtime hinders independent deployment, scalability, and failure isolation ▸ Shared database can lead to coupling among modules
- 10.
- 11. Two-phase commit 11 Challenges ▸ Strongdata consistency requirements ▸ Heterogeneous data sources ▸ Integrating with a third-party or legacy system ▸ Exactly-once message processing Solution ▸ 2PC protocol and X/Open XA specification ▸ JTA and WS AtomicTransaction implementations
- 12. Implementation options 12 Distributed transactionmanagers ▸ Narayana, JOTM, BTM, Atomikos, MSDTC Datasources ▸ PostgreSQL, MySQL, Db2, Oracle, SQL Server ▸ ActiveMQ, HornetQ, MSMQ, IBM MQ, Solace ▸ Infinispan, Hazelcast Non 2PC/XA distributed transactions ▸ eBay’s GRIT protocol
- 13. Two-phase commit 13 Benefits ▸ Anout-of-the-box, standard-based solution ▸ Implemented by many traditional datasources ▸ ACID properties Drawbacks ▸ A blocking protocol with database locks and performance penalty ▸ All participating services must be available ▸ Manual recovery from controller crashes
- 14.
- 15.
- 16. Challenges ▸ Business processspanning several services ▸ Business functions lasting for hours or days ▸ A single location for management and monitoring of distributed transactions Solution ▸ Implement each business transaction that spans multiple services as a sequence of local transactions Saga 16
- 17. Saga implementation options 17 Coordinationapproaches ▸ Orchestration - a controller tells the participants what local transactions to execute ▸ Choreography - business transaction coordination logic is spread among all participants Communication mechanisms ▸ Synchronous, for example HTTP, gRPC ▸ Asynchronous, for example Apache ActiveMQ, Apache Kafka Orchestration Choreography
- 18. Orchestration based Saga 18 Benefits ▸Doesn’t require all participants to be available at the same time, nor to have knowledge about each other ▸ Single place to define and monitor the transaction flow Drawbacks ▸ Complex programing model, requires coordination, compensation logic, and idempotency implementations ▸ Lacks Isolation from AC*D properties which can cause dirty reads, lost updates, non-repeatable reads
- 19.
- 20. Choreography 20 Challenges ▸ Business processspanning several services ▸ Business functions lasting for hours or days ▸ Highly scalable and available system Solution ▸ Implement each business transaction that spans multiple services as a sequence of local transactions with distributing decision making in each service
- 21. Choreography 21 Benefits ▸ Doesn't requirea transaction coordinator service ▸ Better performance compared to orchestration approach due to smaller number of interactions Drawbacks ▸ No single place to define and monitor the business transaction flow ▸ Coupling among participant with point to point interactions ▸ Lacks Isolation from AC*D properties
- 22. Implementation options 22 Publish, thenlocal-commit Local-commit, then publish Event sourcing Outbox pattern Two-phase commit
- 23. Outbox Pattern 23 Offers anapproach for services to update their data store and notify other services in a reliable and eventually consistent manner. Benefits ▸ Addresses the dual-write problem ▸ Offers “read your own writes" semantics Drawbacks ▸ Requires specialized tools, such as Debezium
- 24.
- 25. Parallel pipelines 25 Challenges ▸ Avoiddual writes to a local database and a messaging system Solution ▸ Publish a message to other services and yourself in the same messaging system
- 26. Listen to Yourself 26 Benefits ▸Simple, scalable architecture with parallel processing capabilities Drawbacks ▸ Requires temporal dismantling, not commonly applicable ▸ Hard to reason about the global system state ▸ Lacks “read your own writes" semantics
- 27.
- 28.
- 29. Dual write strategiesfor microservices 29 Modular Monolith Two-phase Commit Orchestration Choreography Parallel Pipeline Service runtime Single process Single/Multiple Multiple Multiple Multiple Datasources Single Heterogeneous (requires XA) Heterogeneous Heterogeneous Heterogeneous Point of control Centralized Centralized Centralized Distributed Distributed Steps/Flow execution At once At once (for happy paths), highly coupled Sequential, temporal decoupling Sequential, temporal decoupling Parallel, temporal decoupling Properties ACID, blocking, synchronous ACID, blocking, synchronous ACD, non-blocking, (a)synchronous, ACD, non-blocking, asynchronous, ACD, non-blocking, asynchronous, Examples Local transactions XA, JTA, WS-AT Saga/Outbox, Debezium, Kafka Saga/Outbox, Debezium, Kafka Listen to yourself pattern
- 30. 30 Red Hat OpenShiftStreams for Apache Kafka a fully managed Apache Kafka service by Red Hat http://red.ht/TryKafka Try Apache Kafka in seconds