Data Patterns in Microservice Applications - The pros and cons of different patterns of data consistency in microservice applications.

1. Data Patterns in
Microservice Applications
Ryan Knight - CEO / CTO Grand Cloud
@knight_cloud

2. Ryan Knight
● CEO / CTO of Grand Cloud - Boutique consulting company working
at the intersection of Distributed Systems and Data Engineering
● Experience ranges across traditional software development and
architecture to sales engineering, consulting, solution architecture
and developer advocacy.
● Worked across wide range of companies from small startups such as
Lightbend and DataStax to Large Corporations such as Starbucks
and Capital One.
● Consulting Experience spans over 50 companies and 10 Countries
● Currently Consulting at Brighthouse Financial

3. Distributed System Design
Heart of distributed system design is a requirement
for a consistent, performant, and reliable way of
managing data - Jonas Bonér

4. Cloud Native -> New Requirements
Users: 1 million+ Data volume: TB–PB–EB
Locality: Global
Performance: Milliseconds–microseconds
Request rate: Millions
Access: Web, mobile, IoT, devices
Scale: Up-down, Out-in
Economics: Pay for what you use
Developer access: No assembly required

5. Challenges with Data
Consistency

6. ● RDBMS
● CAP Theorem
● Trade-off between
Consistency and Scale
● Rise of Eventual Consistency
● NoSQL Databases
EASY
COMPLEX
ACID TXN / Strong
Consistency
Eventual Consistency
(D)evolution of Consistency

7. Challenges
with Eventual
Consistency
Credit to this tweet

8. CAP Theorem

9. Challenges with Application Tier Consistency
● Consistency problems are far harder to solve in the application tier
● Increased Corner Case Bugs
○ Consistency is really hard to get right in the Application Tier!
○ Consistency is really hard to test and verify
● Increased Complexity

10. Business Impact of Consistency
● Travel Booking of Flight, Hotel, etc. - Inconsistencies could either
lead to double bookings or lost bookings.
● Rewards Program - Very difficult to prevent fraudulent redemptions.
Potential for monetary loss.
● Physical Allocation of Resources vs. Digital Realm
● Inventory / Limited Sales

11. Direct Business Value of “Strong Consistency”
● Increases accuracy of sales and reduces lost business revenue
● Cost Savings with reduced operational complexity and increased
visibility into business operations.
● Weak Consistency is a Security Concern - Possible financial loss
from inconsistent views of data.
● ACIDRain Attack - Todd Warszawski, Peter Bailis
○ 22 critical ACIDRain attacks that allow attackers to corrupt store
inventory,over-spend gift cards, and steal inventory.
○ Bankrupt popular Bitcoin exchange

12. Eventual Consistency
● Internet of Things
● Media
● Retail
● Real-time Analytics
● Time-Series
● Monitoring
● Customer 360
Strong Consistency
● Financial Transactions
● Rewards Programs
● Inventory Management
● Global Meta-Data
● Travel Reservations
● Gaming
● Billing / Payments
● Ad Tech

13. What is Data Consistency?

14. Challenges with Understanding Consistency
● Lots of Definitions of Consistency
● Consistency in ACID is about enforcing invariants
○ Data must be valid according to all defined rules
○ Not the consistency we are looking for
● "Strong consistency" - term used to differentiate full
consistency from weaker levels of consistency such as
casual or session consistency.

15. Consistency Challenges
Dirty Reads - Read Uncommitted Write
Read Skew / Non-Repeatable Reads
Read your own Writes
Lost Updates
Write Skew

16. Write Skew
Two concurrent
transactions each
determine what they are
writing based on reading
a data set which overlaps
what the other is writing
begriffs.com

17. Consistency Models
Credit to Peter Bailis and Aphyr at jepsen.io
http://www.bailis.org/blog/linearizability-versus-serializability/

18. Linearizability
● Guarantees that the order of reads and writes to a single
register or row will always appear the same on all nodes.
● Appearance that there is only one copy of the data.
● It doesn’t group operations into transactions.
● Guarantees read-your-write behavior.

19. Linearizable Consistency in CAP
● CAP Theorem is about “atomic consistency”
● Atomic consistency refers only to a property of a single
request/response operation sequence.
● Strong Consistency in CAP is Linearizability

20. Serializable Consistency
● Transaction Isolation
● Database guarantees that two transactions have the same
effect as if they where run serially.
● multi-operation, multi-object, arbitrary total order

21. Strict Serializability
● Linearizability plus Serializability provides Strict
Serializability
● Highest level of Consistency
● Guarantee ordering and transaction isolation

22. Linearizable vs. Serializable Consistency
● Serializability - multi-operation, multi-object, arbitrary total order
● Linearizability - single-operation, single-object, real-time order
● Strict Serializability - Linearizability plus Serializability provides Strict
Serializability
Peter Bailis - Linearizability versus Serializability

23. No One Solution to Consistency
● Do you want your data right or right now? - Pat Helland
● PACELC Theorem -> More than CAP
○ In the absence of network partitions the trade-off is
between latency and consistency - Daniel Abadi
● Evaluate trade-offs in the differing approaches

24. Data Consistency in
Microservices and
Serverless

25. From Monolith to Microservices to Serverless
● Data Consistency was easy in a monolith application -
single source of truth w/ ACID transactions
● Move to microservices each service became a bounded
context that owns and manages its data.
● Data Consistency became very difficult w/ microservices
● Serverless increases the complexity even more

26. Consistency Challenges with Data in Microservices
● Traditional ACID transactions did not scale
● Data orchestration between multiple services - Number of
Microservices Increases Number of Interactions
● Stateful or Stateless
● Data rehydration for things like service failures and rolling
updates.

27. Popularity of Eventual Consistency
CAP Theorem
• Force choice between Global Scale or Strong Consistency
Eventual Consistency
• Sacrificed consistency for availability and partition tolerance.
• Really a Necessary Evil
• Write now and figure it out later
Pushed complexity of managing consistency to application tier

28. Rise of Managing
Consistency
in the Database

29. Value of Consistency in the Database
● Decrease Application Tier Complexity
● Reduce Cognitive Overhead
● Increased Developer Productivity
● Increased Focus on Business Value
● Most implementations also provide strong atomicity and isolation
● Push complexity of consistency back to the database
● Not a panacea for all data consistency challenges

30. Case Study - AdStage
● Recently migrated from Cassandra to Postgres
● Leverage Postgres DB Transactions
● Found Postgres to be extremely capable with advance
data model and query capabilities
● Significant decrease in application and operational
complexity
● Significantly reduced operational costs

31. Leveraging DB Consistency
● Ledger Pattern with Compare and Swap Like Operation
● Application reads latest ledger id from DB
● Application makes an update with what it thinks is the latest
ledger id plus one
● DB transaction / stored procedure to read the last ledger id and
make the update if the ledger id is greater than the last entry
● If update fails DB returns correct Ledger ID

32. Traditional / Hybrid NoSQL DB’s
● Cloud Operated Relation DB’s are a re-emerging trend.
● Cloud SQL w/ Postgres or MySQL
● AWS Aurora - Amazon re-designed MySQL as a
cloud-native relational database
● AWS Dynamo w/ Transactions - Multiple Object with limits
to single region

33. Next Generation Databases
● Google Spanner - Horizontally scalable, globally consistent, relational database
service. Relies on on Proprietary Atomic Clocks and Low Latency Network.
● Coackroach & YugaByte - Open Source version of Spanner with 2 Phase
Commits and Hybrid-Logical Clocks
● Fauna - Single Phase Commit with no hard dependency on clocks
● FoundationDB - Serializable Optimistic MVCC concurrency. Loosely based on
Google Percolator
● TiDB - Hybrid Transactional and Analytical Processing (HTAP) workloads.
Features “horizontal scalability, strong consistency, and high availability.”
● Microsoft Azure Cosmos DB - Configurable consistency guarantees

34. Transactions are hard. Distributed transactions are
harder. Distributed transactions over the WAN are
final boss hardness. I'm all for new DBMSs but
people should tread carefully. - Andy Pavlo
New Generation / Global Transactional Databases

35. Not All Global Databases are the Same
● Differences in Transaction Protocol
● Global Ordering Done in a Single Phase vs. Multi-Phase
● Pre or Post Commit Transaction Resolution
● Different levels of consistency
● Maximum scope of a transaction - Single Record vs. Multiple
Records
● Geographic limits of transactions - Single Region vs. Global?
● Storage Layer is an entirely other discussion beyond the
transaction protocol. Large impact on performance and stability!

36. Week Isolation Level
Scope of Transaction -
Single Row
Eventually Consistent
Strongest Isolation Level
Scope of Transaction -
Distributed Across
Partitions
Serializable Consistency
Consistency and the ACID Spectrum

37. Consistency Levels in Next Gen Databases - 1/2
● Google Spanner - External strong consistency across rows, regions, and
continents.
● Yugabyte - snapshot isolation, not serializability yet, writes must go to
partition leaders. Reliance on hybrid clocks makes it difficult to run in
virtualized environments.
● Cockroach - serializability but not strict serializability, reads and writes must
go to partition leaders, no replica reads allowed

38. Consistency Levels in Next Gen Databases - 2/2
● TiDB - read-committed within a datacenter, no serializability, timestamp oracle
must issue leases for all write transactions, replica reads unclear
● FoundationDb: Serializable Snapshot Isolation and strictly serializable within a
datacenter, timestamp oracle must issue leases for all serializable reads and
all writes, snapshot reads possible
● FaunaDB - Global pre-ordering of transactions provides strict serializable consistency
● Azure Cosmos DB - Five consistency models allow developer to choose between
latency and consistency. Highest Level of consistency is strong consistency with
linearizability guarantees. Doesn’t seem to be strict serializable?

39. Adventures in Application
Tier Consistency

40. Application Tier Consistency
Write now and figure it out later

41. Advantages of Application Tier Consistency
● Low Read / Write Latency
● High-Throughput
● Read your Writes - Same session only
● Requires application to enforce session stickiness

42. Disadvantages of Application Tier Consistency
● Consistency problems are far harder to solve in the
application tier
● Increased Complexity
● No Isolation and limited atomicity
● Corner Case Bugs - Consistency is really hard to test and
verify
● No magic pattern or technology that you can sprinkle on
data to make it consistent.

43. Options for Application Tier Consistency
● Serialization Points - i.e. Kafka Consumers pinned to session id’s.
● Akka Clustering - Stateful Services pinned to a client id.
● CRDT - Conflict Free Replicated Data Types, i.e. Associative
Counters. Data must be of a certain shape to work.
● Event Sourcing / Append Only Logging with Aggregates for running
totals. Hard to provide consistency guarantees across aggregates.
● Saga Pattern - Builds on Event Sourcing and uses a Central
Coordinator to manages complex transaction logic. Relies heavily on
idempotent services that can roll back transactions in the face of
failures.

44. Patterns for Application Tier Consistency
● Kafka Consumer Serialization Points
● Akka Clustering w/ Cluster Singletons
● CRDT - Conflict Free Replicated Data Types
● Event Sourcing / Append Only Logging
with Aggregates
● CQRS
● Saga Pattern
● Custom Distributed Transactions
WIRED
TIRED

45. CRDT’s
● CRDT - Conflict Free Replicated Data Types
● Data types that guarantee convergence to the same value without any
synchronization mechanism
● Consistency without Consensus
● Avoid distributed locks, two-phase commit, etc. Data Structure that
tells how to build the value
● Sacrifice linearizability (guaranteed ordering ) while remaining correct

46. Overview of Saga Pattern
● Central Coordinator
● Manages Complex Transaction Logic
● State managed in an distributed log
● Split work into idempotent executors / requests
● Requires compensating transactions for dealing with failures /
aborting transaction
● Effectively Once instead of Exactly Once

47. The Challenges with the Saga Pattern
● Consistency is reliant on the consistency of the distributed log
● Limited Consistency
● Weak Isolation
● No Guaranteed Atomicity - Unsafe partially committed states
● Complexity with versioning of Saga Logic
● Increased application complexity
● Rollback and recovery logic required in application tier
● Idempotency impossible for some services
● Effectively Once instead of Exactly Once

48. Data Patterns in
Microservice Applications
Ryan Knight - CEO / CTO Grand Cloud
@knight_cloud

49. Addendum

50. Global Scale Next Gen
Databases

51. Spanner
● External consistency, an isolation level even stricter than strict serializability
● Relation Integrity Constraints
● 99.999% availability SLA
● Uses a global commit timestamps to guarantee ordering of transactions via the
TrueTime API.
● Multiple Shards with 2PC
● Single Shard Avoids 2PC for Writes / Read-only Transactions also avoid 2 PC
● No Downtime upgrades - Maintenance done by moving data between nodes
● Downside is cost and some limitations to the SQL model and schema design

52. CoackroachDB
● Open source Database Inspired by Spanner
● Hybrid Logical Clock similar to a vector clock for ordering of transactions
● Challenges with clock skew - waits up to 250 MS on reads
● Provides linearizability on single key and overlapping keys
● Transactions that span disjoint set of key it only provides serializability and not
linearizability
● Some edge cases cause anomalies called “casual reverse” - Jepsen
● “Enterprise-only” features like row-level replication zones
● Supports migrating by supporting PostgreSQL syntax and drivers, however it does
not offer exact compatibility.

53. YugaByte
● Another Database Inspired by Spanner that relies on Hybrid Logical Clocks
● Currently only supports snapshot isolation
● Serializable isolation level work in progress
● Distributed Transactions to multiple partitions require a provisional record or
temporary table

54. FaunaDB - Consistency without Clocks
● Transaction resolution based on the Calvin protocol - pre-ordering of transactions
before commit
● Global transaction ordering provides serializable consistency
● Transactions can include multiple rows - not restricted to data in a single row or
shard
● Distributed log based algorithm scales throughput with cluster size by partitioning
the log
● Low Latency Snapshot Reads
● Proprietary Query Language with a high learning curve
● Optimistic concurrency model can causes high number of failures with highly
contentious workloads

55. References
● Bla-bla-microservices-bla-bla http://jonasboner.com/bla-bla-microservices-bla-bla/
● Aphyr Strong consistency models -
https://aphyr.com/posts/313-strong-consistency-models
● Achieving ACID Transactions in a Globally Distributed Database from FaunaDB
● Peter Bailis - Linearizability versus Serializability
● Calvin: fast distributed transactions for partitioned database systems