This document summarizes Chris Richardson's presentation on using sagas to maintain data consistency in a microservices architecture. Richardson explains that ACID transactions cannot be used across services due to limitations of distributed transactions. Instead, sagas can be used, where each service performs a local transaction and messages are passed between services. There are two main approaches to coordinating sagas: choreography-based using domain events, and orchestration-based where a central saga manages the process. Countermeasures like semantic locking are also needed to prevent data anomalies from the lack of isolation in sagas.

1. @crichardson
Managing data consistency in a
microservice architecture using
Sagas
Chris Richardson
Founder of Eventuate.io
Founder of the original CloudFoundry.com
Author of Microservices Patterns and POJOs in Action
@crichardson
chris@chrisrichardson.net
http://learn.microservices.io
Copyright © 2019. Chris Richardson Consulting, Inc. All rights reserved

2. @crichardson
Presentation goal
ACID transactions
within services
Sagas between
services

3. @crichardson
About Chris
http://learn.microservices.io

4. @crichardson
About Chris
https://microservices.io/book
40% discount with code
ctwmicrocph19

5. @crichardson
Agenda
ACID is not an option
Overview of sagas
Coordinating sagas

6. The microservice architecture
structures
an application as a
set of loosely coupled
services

7. @crichardson
Loose coupling =
encapsulated data
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit
availableCredit
Change schema without coordinating with other teams

8. @crichardson
ACID transactions within
services are fine
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit
availableCredit
ACID transaction ACID transaction

9. @crichardson
But how to maintain data
consistency across services?
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit
availableCredit
?

10. @crichardson
Enforcing the customer’s credit limit
createOrder(customerId, orderTotal)
Pre-conditions:
• customerId is valid
Post-conditions
• Order was created
• Customer.availableCredit -= orderTotal
Customer class
Invariant:
availableCredit >= 0
availableCredit <= creditLimit
Spans
services

11. @crichardson
Cannot use ACID transactions
that span services
BEGIN TRANSACTION
…
SELECT ORDER_TOTAL
FROM ORDERS WHERE CUSTOMER_ID = ?
…
SELECT CREDIT_LIMIT
FROM CUSTOMERS WHERE CUSTOMER_ID = ?
…
… CHECK FOR SUFFICIENT CREDIT….
…
INSERT INTO ORDERS …
…
COMMIT TRANSACTION
Private to the
Order Service
Private to the
Customer Service
Distributed transactions

12. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks
Not supported by many NoSQL databases (or message brokers)
CAP theorem 2PC impacts availability
….

13. @crichardson
Basically
Available
Soft state
Eventually consistent
http://queue.acm.org/detail.cfm?id=1394128
ACID

14. @crichardson
Agenda
ACID is not an option
Overview of sagas
Coordinating sagas

15. @crichardson
From a 1987 paper

16. @crichardson
Saga
Use Sagas instead of 2PC
Distributed transaction
Service A Service B
Service A
Local
transaction
Service B
Local
transaction
Service C
Local
transaction
X Service C

17. @crichardson
Order Service
Create Order Saga
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
state=APPROVED
approve
order()
createOrder() Initiates saga

18. @crichardson
Seems simple but …

19. @crichardson
Challenge: rollback
BEGIN TRANSACTION
…
UPDATE …
…
INSERT ….
…
…. BUSINESS RULE VIOLATED!!!!
…
ROLLBACK TRANSACTION
Really simple ACID
transaction!
Create Order
Reserve
Credit FAILS
Already committed.
How to undo?!
VS.

20. @crichardson
Rolling back sagas
Use compensating transactions
Developer must write application logic to “rollback” eventually
consistent transactions
Careful design required!

21. @crichardson
Saga: Every Ti has a Ci
T1 T2 …
C1 C2
Compensating transactions
T1 T2 C1
FAILS

22. @crichardson
Order Service
Create Order Saga - rollback
Local transaction
Order
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
reject
order()
createOrder()
FAIL
Insufficient credit

23. @crichardson
Writing compensating
transactions isn’t always easy
Write them so they will always succeed
If a compensating transaction fails no clear way to recover
Challenge: Undoing changes when data has already been
changed by a different transaction/saga
More on this later
Actions that can’t be undone, e.g. sending email
Reorder saga and move to end

24. Structure of Create Order
Saga
T1:
Create Order
C1:
Reject Order
T2:
Reserve Credit
T3:
Approve Order
Needs to be compensatable
because Reserve Credit can
fail
Last step that can fail: Does not
need compensation
Can’t fail

25. Saga structure: 3 phases
T1 C1
…
T2 C2
Tn+1
Tn+2
….
Compensatable transactions
Pivot transaction = GO/NO GO
Retriable transactions that can’t fail
and don’t need compensation
Great for actions that cannot be undone

26. Ordering the steps of a saga
Dependencies between
steps:
Result of one step is
used as input to next
e.g. Create Order before
Reserving Credit
But often you have a choice
Simplify by moving updates
to after pivot transaction
Update
Customer
Undo Update
Customer
…
Pivot Transaction
Update
Customer
…
Pivot Transaction
No compensation required
Reorder

27. @crichardson
Challenge: Sagas are ACD
Atomicity
Saga implementation ensures that all transactions are
executed OR all are compensated
Consistency
Referential integrity within a service handled by local databases
Referential integrity across services handled by application
Durability
Durability handled by local databases

28. @crichardson
Lack of I
Outcome of concurrent execution
!=
a sequential execution
=
Data anomalies

29. @crichardson
Anomaly: Lost update
Ti: Create
Order
Tj: Approve
Order
Ti: Cancel
Order
Saga 1
Create Order
Saga 2
Cancel order
Time
Overwrites
cancelled order
Also: A compensating transaction that simply restores
previous value can cause a lost update

30. @crichardson
Anomaly: Dirty reads
Ti: Reserve
Credit
Ci: Unreserve
credit
Ti: Reserve
Credit
Reads
uncommitted
changes
Order rejected unnecessarily OR Credit limit exceeded
Time
Saga 1
Create Order
Saga 2
Create Order
Ti: Reserve
InventoryX
Temporarily changed

31. @crichardson
Use countermeasures to
prevent anomalies
Use semantic lock to prevent lost updates
e.g. a PENDING order cannot be cancelled
Reorder saga to eliminate dirty reads caused by compensating transactions
Ci undoes changes made by Ti => possibility of dirty read
Eliminate Ci by moving update to (after) pivot transaction
Use commutative updates to eliminate lost updates caused by compensating
transactions
Customer.reserveCredit() and releaseCredit()
Account.debit() and credit()
…
https://www.slideshare.net/chris.e.richardson/saturn-2018-managing-data-consistency-in-a-microservice-architecture-using-sagas

32. @crichardson
Agenda
ACID is not an option
Overview of sagas
Coordinating sagas

33. @crichardson
How to sequence the saga
transactions?
After the completion of transaction Ti “something” must
decide what step to execute next
Success: which T(i+1) - branching
Failure: C(i - 1)

34. @crichardson
Use asynchronous, broker-
based messaging
Order Service
Customer
Service
….
Message broker
Guaranteed atleast once delivery ensures a saga
complete when its participants are temporarily unavailable

35. @crichardson
Saga step = a transaction local
to a service (a.k.a. participant)
Service
Database Message Broker
update Send message/event
How to
make atomic
without 2PC?*
*Transactional Outbox pattern or Event Sourcing

36. @crichardson
Choreography: distributed decision making
vs.
Orchestration: centralized decision making

37. @crichardson
Option #1: Choreography-based
coordination using events
Order
Service
Customer
Service
Order created
Credit Reserved
Credit Limit Exceeded
Create Order
OR
Customer
creditLimit
creditReservations
Order
state
total
create()
reserveCredit()
approve()/
reject()
Order events channel
Customer events channel

38. @crichardson
Order Service: publishing
domain events
Publish event
Create order

39. @crichardson
Customer Service: consuming
domain events…
https://github.com/eventuate-tram/eventuate-tram-examples-customers-and-orders
Subscribe to event

40. @crichardson
Customer Service: consuming
domain events
Attempt to
Reserve credit
Publish event on
success
Publish event on
failure

41. Benefits and drawbacks of
choreography
Benefits
Simple, especially when using event
sourcing
Participants are loosely coupled
Drawbacks
Decentralized implementation -
potentially difficult to understand
Cyclic dependencies - services listen
to each other’s events, e.g.
Customer Service must know about
all Order events that affect credit
Overloads domain objects, e.g.
Order and Customer know too
much
Events = indirect way to make
something happen
https://github.com/eventuate-examples/eventuate-examples-java-customers-and-orders

42. @crichardson
Order Service
Option #2: Orchestration-based saga
coordination
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
state=APPROVED
approve
order()
createOrder()
CreateOrderSaga
InvokesInvokesInvokes

43. @crichardson
A saga (orchestrator)
is a persistent object
that
implements a state machine
that
invokes the participants and
handles replies

44. Saga orchestrator behavior
On create:
Invokes a saga participant
Persists state in database
Wait for a reply
On reply:
Load state from database
Determine which saga
participant to invoke next
Invokes saga participant
Updates its state
Persists updated state
Wait for a reply
…

45. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
Order
state
total…
reserveCredit
CreateOrder
Saga
OrderService
create()
create()
approve()
Credit Reserved
Customer command channel
Saga reply channel
https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders

46. @crichardson
CreateOrderSaga definition
Sequence of
steps
step = (Ti, Ci)
Saga’s Data

47. @crichardson
Customer Service command
handler Route command
to handler
Reserve
credit
Make reply message

48. @crichardson
Eventuate Tram Sagas
Open-source Saga orchestration framework
Currently for Java
https://github.com/eventuate-tram/eventuate-tram-sagas
https://github.com/eventuate-tram/eventuate-tram-sagas-
examples-customers-and-orders

49. Benefits and drawbacks of
orchestration
Benefits
Centralized coordination
logic is easier to understand
Current state is in database
Reduced coupling, e.g.
Customer knows less
Reduces cyclic
dependencies
Drawbacks
Risk of smart sagas
directing dumb services

50. @crichardson
Summary
Microservices tackle complexity and accelerate development
Database per service is essential for loose coupling
Use ACID transactions within services
Use orchestration-based or choreography-based sagas to
maintain data consistency across services
Use countermeasures to reduce impact of anomalies caused
by lack of isolation

51. @crichardson
@crichardson chris@chrisrichardson.net
http://learn.microservices.io
Questions?
40% discount with code
ctwmicrocph19