About data management perspectives in microservices practices and architecture.

1. Data Management in
Microservice - Part I
Okis Chuang
1

2. Patterns of Data Management in
Microservices
Databas
e per
Service
Event-
driven
Architectur
e
CQRS
Command
Query
Responsibili
ty
Segregation
From Chris Richardson
2
Database
as a queue
(Application
events)
Event-
sourcing
Mining
Transact
ion Log

3. Agenda
• Monolith vs. Microservices
• Distributed data management in microservice
• Database per Service pattern
• Event-driven architecture
• Patterns to achieve Atomicity
• Database as a queue
• Transaction log mining
• Event sourcing
3

4. Monolith vs. Microservices
4

5. Let’s imagine you’re building a large,
complex application, e.g. an online store
5

6. Successful software
development
Architecture
Process Organization
Agile
Continuous Delivery
…
Small
Autonomous Team
…
？
6

7. But successful application
keeps growing…
7

8. Architecture
Process Organization
Agile
Continuous Delivery
…
Small
Autonomous Team
…
Monolithic
8

10. Architecture
Process Organization
✅Agile
✅Continuous Delivery
…
✅Small
✅Autonomous Team
…
Microservice
10

11. Problem
What’s the database architecture in a microservice
application?
11
How to deal with distributed data management problems?

12. Forces
• Services must be loosely coupled so that they
can be developed, deployed, and scaled
independently.
12

13. Patterns of Data Management in
Microservices
Event-
driven
Architectur
e
CQRS
Command
Query
Responsibili
ty
Segregation
From Chris Richardson
13
Database
as a queue
(Application
events)
Event-
sourcing
Mining
Transact
ion Log
Databas
e per
Service

14. Decentralized Data Management
Pattern: Keep a service’s persistent data private
14
Atomicity
Consistency
Isolation
Durability

15. Shared Database
Database per
Service
Database Architecture
15

16. Shared Database
Order Service Customer Service …Service
Order Table
order_total
Customer Table
credit_limit
…
…
Tight Coupling😭 Simple and ACID ✌️
16

17. Database per Service
Order Service Customer Service
Order Table
order_total
Customer Table
credit_limit
Loosely coupling 🤘🤘 but more complex 😧
17

18. DB Architecture in a
microservice application
• Microservice-based applications often use mixture of
SQL/NoSQL databases, which called polyglot
persistence
• Database per Service (Keep a service’s persistent data
private)
• Private-tables-per-service
• Schema-per-service
• Database-server-per-service
19

19. How we guarantee data
consistency in microservice?
20

20. 21

21. Use 2PC algorithm?
22

22. Use 2 Phase Commit to
cope (a.k.a distributed
transaction) ?
24

23. 25
Coordinator Coordinator
Cohort Cohort

24. 2PC is not an option
• It’s a BLOCKING protocol
• According to CAP theorem, you need to choose
between availability and ACID-style consistency
• Availability is usually a better choice
• Moreover, most NoSQL databases do not support
2PC
• BUT maintaining data consistency across services and
databases is essential
26

25. Patterns of Data Management in
Microservices
Databas
e per
Service
CQRS
Command
Query
Responsibili
ty
Segregation
From Chris Richardson
27
Database
as a queue
(Application
events)
Event-
sourcing
Mining
Transact
ion Log
Event-
driven
Architectu
re

26. 28

27. 29

28. 30

29. The Architecture provides
that
• Each service atomically updates its database and
publish events
• The message broker guarantees that events are
delivered at least once then you can implement
business transactions that span multiple services
• trading some consistency for availability can lead to
dramatic improvements in scalability (1)
• Eventual consistency instead of Strong
consistency
31

30. use events to maintain materialized views
that pre-join data owned by multiple
microservices.
could be NoSQL like
document-based
MongoDB

31. We’re not done yet…
What if Order service crashes
after updating the database
but before publishing the event
We might be loosing Atomicity
(Atomically update state and publish events)

32. lution to transactions that span multiple services and provide eve
Enables an application to maintain materialized views
ogramming model is more complex than when using ACID transac
plement compensating transaction to recover application-level fail
Materialized view is not up to date
Duplicate event should be detect

33. Patterns of Data Management in
Microservices
Databas
e per
Service
Event-
driven
Architectur
e
CQRS
Command
Query
Responsibili
ty
Segregation
From Chris Richardson
35
Event-
sourcing
Mining
Transact
ion Log
Database
as a queue
(Applicatio
n events)

34. Publishing Events Using Local Transactions
1. Create EVENT table for storing state of business entities
2. Local transaction for business logic and EVENT table
3. A separate application thread or process queries the EVENT table,
publish events to Message Broker, then use local transaction to mark
the events as published

35. Patterns of Data Management in
Microservices
Databas
e per
Service
Event-
driven
Architectur
e
CQRS
Command
Query
Responsibili
ty
Segregation
From Chris Richardson
37
Database
as a queue
(Application
events)
Event-
sourcing
Mining
Transac
tion Log

36. Mining Database transaction Log
LinkedIn opensource
Stream mechanism in AWS DynamoDB

37. Patterns of Data Management in
Microservices
Databas
e per
Service
Event-
driven
Architectur
e
CQRS
Command
Query
Responsibili
ty
Segregation
From Chris Richardson
39
Database
as a queue
(Application
events)
Mining
Transact
ion Log
Event-
sourcin
g

38. • Different and unfamiliar style of programming
• The event store only directly supports the lookup of
business entities by primary key
• Should use CQRS to implement query

39. Summary
• In microservices, each service has its own private
datastore which might be different SQL/NoSQL
databases
• While the database architecture has significant benefits,
it also derive distributed data management challenge
• How to implement business transaction that maintain
consistency across multiple DBs
• If adopting Event-driven, how to atomically update
state and publish event