This is a talk I gave at QCON ArchSummit in Shenzhen. The microservice architecture structures an application as a set of loosely coupled, collaborating services. Maintaining data consistency is challenging since each service has its own database to ensure loose coupling. To make matters worse, for a variety of reasons distributed transactions using JTA are not an option for modern applications. In this talk we describe an alternative transaction model known as a saga. You will learn about the benefits and drawbacks of using sagas. We describe how sagas are eventually consistent rather than ACID and what this means for developers. You will learn how to design and implement sagas in a Java application.

1. @crichardson
Using sagas to maintain data
consistency in a microservice
architecture
Chris Richardson
Founder of Eventuate.io
Founder of the original CloudFoundry.com
Author of POJOs in Action
@crichardson
chris@chrisrichardson.net
http://eventuate.io

2. @crichardson
Presentation goal
Distributed data management challenges
in a microservice architecture
Sagas as the transaction model

3. @crichardson
About Chris

4. @crichardson
About Chris

5. @crichardson
About Chris

6. @crichardson
About Chris

7. @crichardson
About Chris

8. @crichardson
About Chris
Consultant and trainer
focusing on modern
application architectures
including microservices
(http://www.chrisrichardson.net/)

9. @crichardson
About Chris
Founder of a startup that is creating
an open-source/SaaS platform
that simplifies the development of
transactional microservices
(http://eventuate.io)

10. @crichardson
For more information
http://learnmicroservices.io

11. @crichardson
Agenda
ACID is not an option
Overview of sagas
Coordinating sagas

12. The microservice architecture
structures
an application as a
set of loosely coupled
services

13. @crichardson
The microservice architecture
tackles complexity through
modularization

14. @crichardson
The microservice architecture
tackles complexity through
modularization
* might improve scalability too

15. @crichardson
Microservice architecture
Browser
Mobile
Device
Store
Front UI
API
Gateway
Customer
Service
Order
Service
…
Service
Customer
Database
Order
Database
…
Database
HTML
REST
REST

16. @crichardson
Microservice architecture
Browser
Mobile
Device
Store
Front UI
API
Gateway
Customer
Service
Order
Service
…
Service
Customer
Database
Order
Database
…
Database
HTML
REST
REST
Database per service

17. @crichardson
Loose coupling =
encapsulated data
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit

18. @crichardson
How to maintain data
consistency?!?!?
Invariant:
sum(open order.total) <= customer.creditLimit

19. @crichardson
Cannot use ACID transactions
BEGIN TRANSACTION
…
SELECT ORDER_TOTAL
FROM ORDERS WHERE CUSTOMER_ID = ?
…
SELECT CREDIT_LIMIT
FROM CUSTOMERS WHERE CUSTOMER_ID = ?
…
INSERT INTO ORDERS …
…
COMMIT TRANSACTION

20. @crichardson
Cannot use ACID transactions
BEGIN TRANSACTION
…
SELECT ORDER_TOTAL
FROM ORDERS WHERE CUSTOMER_ID = ?
…
SELECT CREDIT_LIMIT
FROM CUSTOMERS WHERE CUSTOMER_ID = ?
…
INSERT INTO ORDERS …
…
COMMIT TRANSACTION
Private to the
Order Service
Private to the
Customer Service

21. @crichardson
Cannot use ACID transactions
BEGIN TRANSACTION
…
SELECT ORDER_TOTAL
FROM ORDERS WHERE CUSTOMER_ID = ?
…
SELECT CREDIT_LIMIT
FROM CUSTOMERS WHERE CUSTOMER_ID = ?
…
INSERT INTO ORDERS …
…
COMMIT TRANSACTION
Private to the
Order Service
Private to the
Customer Service
Distributed transactions

22. @crichardson
2PC is not an option

23. @crichardson
2PC is not an option
Guarantees consistency

24. @crichardson
2PC is not an option
Guarantees consistency
BUT

25. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure

26. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)

27. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks

28. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks
Not supported by many NoSQL databases (or message brokers)

29. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks
Not supported by many NoSQL databases (or message brokers)
CAP theorem 2PC impacts availability

30. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks
Not supported by many NoSQL databases (or message brokers)
CAP theorem 2PC impacts availability
….

31. @crichardson
Agenda
ACID is not an option
Overview of sagas
Coordinating sagas

32. @crichardson
From a 1987 paper

33. @crichardson
Saga
Use Sagas instead of 2PC
Distributed transaction
Service A Service B
Service A
Local
transaction
Service B
Local
transaction
Service C
Local
transaction
X Service C

34. @crichardson
Order Service
Create Order Saga

35. @crichardson
Order Service
Create Order Saga
Local transaction
Order
state=PENDING
createOrder()
createOrder()

36. @crichardson
Order Service
Create Order Saga
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
createOrder()

37. @crichardson
Order Service
Create Order Saga
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
state=APPROVED
approve
order()
createOrder()

38. @crichardson
If only it were this easy…

39. @crichardson
Rollback using compensating
transactions
ACID transactions can simply rollback
BUT
Developer must write application logic to “rollback” eventually
consistent transactions
Careful design required!

40. @crichardson
Saga: Every Ti has a Ci
T1 T2 …

41. @crichardson
Saga: Every Ti has a Ci
T1 T2 …
C1 C2
Compensating transactions

42. @crichardson
Saga: Every Ti has a Ci
T1 T2 …
C1 C2
Compensating transactions
T1 T2 C1
FAILS

43. @crichardson
Order Service
Create Order Saga - rollback
Local transaction
Order
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
reject
order()
createOrder()
FAIL
Insufficient credit

44. @crichardson
Sagas complicate API design
Request initiates the saga. When to send back the response?

45. @crichardson
Sagas complicate API design
Request initiates the saga. When to send back the response?
Option #1: Send response when saga completes:
+ Response specifies the outcome
- Reduced availability

46. @crichardson
Sagas complicate API design
Request initiates the saga. When to send back the response?
Option #1: Send response when saga completes:
+ Response specifies the outcome
- Reduced availability
Option #2: Send response immediately after creating the saga
(recommended):
+ Improved availability
- Response does not specify the outcome. Client must poll or be
notified

47. @crichardson
Revised Create Order API
createOrder()
returns id of newly created order
NOT fully validated
getOrder(id)
Called periodically by client to get outcome of validation

48. @crichardson
Minimal impact on UI
UI hides asynchronous API from the user
Saga will usually appear instantaneous (<= 100ms)
If it takes longer UI displays “processing” popup
Server can push notification to UI

49. @crichardson
Sagas complicate the
business logic
Changes are committed by each step of the saga

50. @crichardson
Sagas complicate the
business logic
Changes are committed by each step of the saga
Other transactions see “inconsistent” data, e.g. Order.state =
PENDING more complex logic

51. @crichardson
Sagas complicate the
business logic
Changes are committed by each step of the saga
Other transactions see “inconsistent” data, e.g. Order.state =
PENDING more complex logic
Interaction between sagas and other operations
e.g. what does it mean to cancel a PENDING Order?
“Interrupt” the Create Order saga
Wait for the Create Order saga to complete?

52. @crichardson
Agenda
ACID is not an option
Overview of sagas
Coordinating sagas

53. @crichardson
How to sequence the saga
transactions?
After the completion of transaction Ti “something” must
decide what step to execute next
Success: which T(i+1) - branching
Failure: C(i - 1)

54. @crichardson
Order Service
Orchestration-based saga coordination
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
state=APPROVED
approve
order()
createOrder()
CreateOrderSaga

55. @crichardson
Complex coordination logic is
centralized 😀

56. @crichardson
Services expose APIs that are
invoked by saga 😄

57. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
OrderService

58. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
CreateOrder
Saga
OrderService
create()

59. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
Order
state
total…
CreateOrder
Saga
OrderService
create()
create()

60. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
Order
state
total…
reserveCredit()
CreateOrder
Saga
OrderService
create()
create()

61. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
Order
state
total…
reserveCredit()
CreateOrder
Saga
OrderService
create()
create() creditReserved()

62. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
Order
state
total…
reserveCredit()
CreateOrder
Saga
OrderService
create()
create()
approve()
creditReserved()

63. @crichardson
Saga orchestrators are state
machines
RESERVING
CREDIT
APPROVED
REJECTED
/ customerService.
reserveCredit()
credit reserved
/ order.approve()
credit limit exceed
/ order.reject()
event [guard] / action
Initial action

64. @crichardson
Saga orchestrators are state
machines
RESERVING
CREDIT
APPROVED
REJECTED
/ customerService.
reserveCredit()
credit reserved
/ order.approve()
credit limit exceed
/ order.reject()
event [guard] / action
Initial action
reply and action

65. @crichardson
Create Order Saga code
Eventuate Saga framework

66. @crichardson
Create Order Saga code
Stateless singleton: Behavior
Eventuate Saga framework

67. @crichardson
Create Order Saga code
Enum
Stateless singleton: Behavior
Eventuate Saga framework
State

68. @crichardson
Create Order Saga code
Enum Persistent data
Stateless singleton: Behavior
Eventuate Saga framework
State

69. @crichardson
Create Order Saga

70. @crichardson
Create Order Saga
State machine definition

71. @crichardson
Initializing the saga

72. @crichardson
Initializing the saga
Create order

73. @crichardson
Initializing the saga
Invoke saga participant
Create order

74. @crichardson
Handling a reply

75. @crichardson
Handling a reply
Update Order

76. @crichardson
Customer Service - command
handling

77. @crichardson
Customer Service - command
handling
Reserve credit

78. @crichardson
Saga
Participant
About Saga orchestrator
participant communication
Saga
Orchestrator
Saga
Participant
Saga must complete even if there are transient failures

79. @crichardson
Saga
Participant
About Saga orchestrator
participant communication
Saga
Orchestrator
Saga
Participant
command
Saga must complete even if there are transient failures

80. @crichardson
Saga
Participant
About Saga orchestrator
participant communication
Saga
Orchestrator
Saga
Participant
command
reply
Saga must complete even if there are transient failures

81. @crichardson
Use asynchronous
messaging

82. @crichardson
Create Order Saga messaging
Order Service
Message Broker
Customer Service
Begin TXN
Read data
Send reply
Commit TXN
Credit Reserved
Credit Limit Exceeded
Create Order
Saga
Orchestrator
reserve credit
Customer Service
Request Channel
Customer Service
Reply Channel
Order
create()
approve()
reject()

83. @crichardson
Create Order Saga messaging
Order Service
Message Broker
Customer Service
Begin TXN
Read data
Send reply
Commit TXN
Credit Reserved
Credit Limit Exceeded
Create Order
Saga
Orchestrator
reserve credit
Customer Service
Request Channel
Customer Service
Reply Channel
Order
create()
approve()
reject()
Commands

84. @crichardson
Create Order Saga messaging
Order Service
Message Broker
Customer Service
Begin TXN
Read data
Send reply
Commit TXN
Credit Reserved
Credit Limit Exceeded
Create Order
Saga
Orchestrator
reserve credit
Customer Service
Request Channel
Customer Service
Reply Channel
Order
create()
approve()
reject()
Commands
Replies

85. @crichardson
Messaging must be
transactional
Service
Database Message Broker
update publish
How to
make atomic
without 2PC?

86. @crichardson
Option #1: Use database
table as a message queue
See BASE: An Acid Alternative, http://bit.ly/ebaybase
Customer
Service
ORDER_ID CUSTOMER_ID TOTAL
99
CUSTOMER_CREDIT_RESERVATIONS table
101 1234
ID TYPE DATA DESTINATION
MESSAGE table
84784 OrderCreated {…} …
INSERT INSERT
Message
Publisher
QUERY
Message
Broker
Publish
Local transaction

87. @crichardson
Option #1: Use database
table as a message queue
ACID
transaction
See BASE: An Acid Alternative, http://bit.ly/ebaybase
DELETE
Customer
Service
ORDER_ID CUSTOMER_ID TOTAL
99
CUSTOMER_CREDIT_RESERVATIONS table
101 1234
ID TYPE DATA DESTINATION
MESSAGE table
84784 OrderCreated {…} …
INSERT INSERT
Message
Publisher
QUERY
Message
Broker
Publish
Local transaction

88. @crichardson
Option #1: Use database
table as a message queue
ACID
transaction
See BASE: An Acid Alternative, http://bit.ly/ebaybase
DELETE
Customer
Service
ORDER_ID CUSTOMER_ID TOTAL
99
CUSTOMER_CREDIT_RESERVATIONS table
101 1234
ID TYPE DATA DESTINATION
MESSAGE table
84784 OrderCreated {…} …
INSERT INSERT
Message
Publisher
QUERY
Message
Broker
Publish
Local transaction
?

89. @crichardson
Publishing messages
Poll the MESSAGE table
OR
Tail the database transaction log

90. @crichardson
Option #2: Event sourcing:
event-centric persistence
Service
Event Store
save events
and
publish
Event table
Entity type
Event
id
Entity
id
Event
data
Event
type
Every state change event

91. @crichardson
Option #2: Event sourcing:
event-centric persistence
Service
Event Store
save events
and
publish
Event table
Entity type
Event
id
Entity
id
Event
data
Event
type
Order 901101 …OrderCreated
Every state change event

92. @crichardson
Option #2: Event sourcing:
event-centric persistence
Service
Event Store
save events
and
publish
Event table
Entity type
Event
id
Entity
id
Event
data
Order 902101 …OrderApproved
Event
type
Order 901101 …OrderCreated
Every state change event

93. @crichardson
Option #2: Event sourcing:
event-centric persistence
Service
Event Store
save events
and
publish
Event table
Entity type
Event
id
Entity
id
Event
data
Order 902101 …OrderApproved
Order 903101 …OrderShipped
Event
type
Order 901101 …OrderCreated
Every state change event

94. @crichardson
Summary
Microservices tackle complexity and accelerate development
Database per service is essential for loose coupling
Use sagas to maintain data consistency across services
Use transactional messaging to make sagas reliable

95. @crichardson
@crichardson chris@chrisrichardson.net
http://learnmicroservices.io
Questions?