w-jax 2022: Keeping CALM – Konsistenz in verteilten Systemen leichtgemacht

© Susanne Braun
KEEPING CALM
Konsistenz in verteilten Systemen leichtgemacht
Susanne Braun
08.11.2022
w-jax

© Susanne Braun
2
@susannebraun
James Hamilton
Infrastructure Efficiency, Reliability and
Scaling Guru
Vice President & Distinguished Engineer at AWS
“The first principle of successful scalability
is to
batter the consistency mechanisms down
to a minimum,
move them off the critical path,
hide them in a rarely visited corner of the
system,
and then make it as hard as possible for
application developers to get permission to
use them.”

© Susanne Braun
3
@susannebraun
Distributed Transactions
System 4
System 1
Local ACID
Transaction
System 2
Local ACID
Transaction
System 3
Local ACID
Transaction
Local ACID
Transaction
Global XA Transaction
X/Open DTP standard
uses 2-Phase-Commit to
ensure global atomicity
It’s implemented by
most relational DBs &
some message brokers
JTA API for Jakarta
EE

© Susanne Braun
4
@susannebraun
Disadvantages of Distributed Transactions
n 2PC itself performs rather poor because of the communication overhead
n If one system fails, the other systems will be blocked and cannot release locks held in the DB, thereby blocking
other transactions
n Blocking nature of 2-Phase-Commit (2PC) negatively impacts availability:
n 3 systems with an availability of 99.5% each, yield overall availability of
n 99.5% x 99.5% x 99.5 % = 98.5%
n NoSQL DBs & Modern message brokers such as RabbitMQ and Kafka do not support XA transactions
Suited for Modern Software Architecture ?

© Susanne Braun
5
@susannebraun
Quality Attributes impacted by Global Coordination
Fault
Tolerance
Resilience
Loose Coupling
Availability
Scalability
Network Partition
Tolerance
Low Latency
Responsiveness
Autonomy
Global Coordination

© Susanne Braun
7
@susannebraun
Eric Brewer
Distributed Systems Researcher
Coined the CAP theorem, Contributed to
Spanner
Prof. emeritus University of California, Berkeley,
works now for Google
“But we forfeit C and I of ACID for
availability, graceful degradation and
performance.”
ACM Symposium on Principles of Distributed Computing, 2000

© Susanne Braun
8
@susannebraun
ACID vs. BASE
ACID BASE
ACID Consistency
(in the sense of one-copy-consistency)
Isolation
(in the sense of one-copy-serializability)
Pessimistic Synchronization
(global locks, synchronous update propagation)
Global Commits
(2PC, majority consensus, …)
Atomicity
Consistency
Isolation
Durability
Eventual Consistency
(stale data & approximate answers)
Availability
(top priority)
Optimistic Synchronization
(no locks, asynchronous update propagation)
Independent Local Commits
(conflict resolution, reconciliation, …)
Atomicity
Consistency
Isolation
Durability
Database is in a consistent state &
all invariants are being met!
This is about Concurrency Control!
This is about Convergence!

© Susanne Braun
9
@susannebraun
Douglas Terry
Coined the term Eventual Consistency in the
90ties
Former Prof. University of California, Berkeley,
worked for Microsoft, Samsung, AWS
“A system providing eventual consistency
guarantees that replicas would eventually
converge to a mutually consistent state,
i.e., to identical contents, if update activity
ceased.”
Int. Conference on Parallel and Distributed Information Systems, 1994

© Susanne Braun
10
@susannebraun
Douglas Terry
Coined the term Eventual Consistency in the
90ties
Former Prof. University of California, Berkeley,
worked for Microsoft, Samsung, AWS
Pragmatic Definition
A system provides eventual consistency if:
(1)each update operation is eventually received by each
replica
(2)non-commutative update operations are performed
in the same order at each replica
(3)the outcome of a sequence of update operations is
the same at each replica (determinism)
Replicated Data Management for Mobile Computing, 2008

© Susanne Braun
11
@susannebraun
Remember:
You do not get any isolation guarantees like ‘Repeatable Read’
Hard to test
Issues emerge randomly in production
… are hard to reproduce
… are hard to debug
Application needs to handle concurrency control:
Huge source of human error!

© Susanne Braun
13
@susannebraun
Remember:
The only guarantee you get:
convergence to identical state
Events / Operations coming out of order
Outdated Data
Conflicts
Potential Concurrency Anomalies
Application needs to handle:
Huge source of human error!

© Susanne Braun
17
@susannebraun
Conflict Handling Strategies
Avoidance
Primary
Copy
Ignoring
Last
writer
wins
Reducing the Chance
Reduce
granularity
of
conflicting
items,
inconsistency
windows,
etc.
Syntactic
Timestamps,
logical
clocks Semantic
Domain-specific
rules
Interesting stuff!

© Susanne Braun
18
@susannebraun
Avoidance
Primary
Copy
Ignoring
Last
writer
wins
Reducing the Chance
Reduce
granularity
of
conflicting
items,
inconsistency
windows,
etc.
Syntactic Semantic
Domain-specific
rules
Interesting stuff!
Timestamps,
logical
clocks

© Susanne Braun
19
@susannebraun
Pat Helland
Database & Distributed Systems Guru
Architect of multiple transaction &
database systems (e.g. DynamoDB)
Worked at Microsoft, Amazon, SalesForce, …
Conference on Innovative Data Systems Research, 2009
A
C
I
D
2.0
Associative
Commutative
Idempotent
Distributed
(ab)c = a(bc)
ab = ba
aa = a
Distributed
Operations

© Susanne Braun
25
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
ACID 2.0 By Example – Shopping Cart with State-based Conflict Resolution
Shopping Cart
－ Soap
－ Lotion
Shopping Cart
－ Lotion
－ Brush
Join
Shopping Cart
－ Soap
－ Lotion
－ Brush
associativity: 𝑠 ∨ 𝑡 ∨ 𝑢 = 𝑠 ∨ 𝑡 ∨ 𝑢
commutaXvity: 𝑠 ∨ 𝑡 = 𝑡 ∨ 𝑠
idempotence: 𝑠 ∨ 𝑠 = 𝑠
Deleted items
might reappear
* Werner Vogels, Communications of the ACM, Volume 52, Issue 1, 2009

© Susanne Braun
27
@susannebraun
Can we do
better?

© Susanne Braun
28
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
Clever Programmer’s Shopping Cart
Shopping Cart Bounded Context
ShoppingCart
id: Identifier
lineItems: Set<LineItem>
handleAddItemAction(AddItemAction a) Product
AddItemAction
id: Identifier
Customer
id: Identifier
1
1
1
*
1 *
Bold Font: Aggregate Root
Italic Font: Value Object
: Cross-Aggregate Reference
1
1

© Susanne Braun
29
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
Clever Programmer’s Shopping Cart
Commutative

© Susanne Braun
30
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
What happens if we allow deletion of items?
ShoppingCart
id: Identifier
lineItems: Set<LineItem>
handleShoppingCartAction(ShoppingCartAction a) Product
ShoppingCartAction
id: Identifier
Customer
id: Identifier
1
1
1
*
1 *
1
1
AddItemAction
id: Identifier
DeleteItemAction
id: Identifier

© Susanne Braun
31
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
What happens if we allow deletion of items?
Non-Commutative

© Susanne Braun
32
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
Actions might be processed in different orders at different Replicas…
Replica 1
Replica 2
add(Rose)
add(Lupine) delete(Rose)
add(Lupine) delete(Rose)
1 x Rose Leonardo da Vinci 1 x Rose Leonardo da Vinci
1 x Lupine – Russel’s Hybrids
1 x Lupine – Russel’s Hybrids 1 x Lupine – Russel’s Hybrids
1x Foxgloves Mix
add(Foxgloves)
add(Foxgloves) add(Rose)
1 x Lupine – Russel’s Hybrids 1 x Lupine – Russel’s Hybrids 1 x Lupine – Russel’s Hybrids
1x Foxgloves Mix
1 x Lupine – Russel’s Hybrids
1x Foxgloves Mix
1 x Rose Leonardo da Vinci

© Susanne Braun
33
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
Item Deletion introduces Oder Dependence
n Adding items to a set is a monotonic function
n processing add-item-actions always increases the result set
n Allowing deletion of items destroys this property
n State toggles back and forth: item in the set, item not in the set, item in the set, …
0
1
2
3
4
5
6
1 2 3 4 5 6 7 8 9 10
F(x)
F(x)
Processing shopping cart actions becomes non-commutative!

© Susanne Braun
34
@susannebraun
Can we do
better?

© Susanne Braun
35
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
The cleverest programmers do the following…
ShoppingCart
id: Identifier
addedItems: Set
deletedItems: Set
handleShoppingCartAction(ShoppingCartAction a)
getLineItems(): Set<LineItem>
Product
ShoppingCartAction
id: Identifier
Customer
id: Identifier
1
1
1
*
1 *
1
1
AddItemAction
id: Identifier
DeleteItemAction
id: Identifier
Monotonic

© Susanne Braun
36
@susannebraun
S
h
o
p
p
i
n
g
C
a
r
d
The cleverest programmers do the following…
Monotonic
Commutative
Derives
shopping cart
state

© Susanne Braun
37
@susannebraun
I
n
s
i
g
h
t
Monotonic Problems
n By modelling state differently:
n two monotonic insert-only sets
n Inserts into these sets commute
n Enabled us to process requests (ShoppingCartActions) independent of order
* Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed consistency is easy. Commun. ACM 63, 9 (2020), 72–
81. https://dl.acm.org/doi/10.1145/3369736
Monotonic Problems output depends only on the the content of their input, not in
the order in which it arrives!*

© Susanne Braun
38
@susannebraun
F
o
u
n
d
a
t
i
o
n
s
The CALM Theorem
Consistency As Logical Monotonicity ( C A L M )
A problem has a consistent, coordination-free distributed implementation
if and only if
it is monotonic.
Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed consistency is easy. Commun. ACM 63, 9 (2020), 72–

© Susanne Braun
39
@susannebraun
Joseph Hellerstein
Databases & Data-Centric Computing
Researcher
Coined the CALM theorem, Founder of TriFacta
Jim Gray Professor of the University of
California, Berkeley
“Intuitively, monotonic problems are
”safe” in the face of missing
information and can proceed without
coordination.”
Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed
consistency is easy. Commun. ACM 63, 9 (2020), 72–81.

© Susanne Braun
42
@susannebraun
I
n
s
i
g
h
t
From CAP to CALM
CAP is a negative result: it captures
properties that cannot be achieved in general.
CAP only holds if we assume the
system in question is required to
execute arbitrary programs.
CALM is a positive result: it circumscribes
the class of problems which can remain
available under partition.
When the partition heals,
monotonic problems converge to
identical state without the
need for further coordination or
conflict resolution.
* Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed consistency is easy. Commun. ACM 63, 9 (2020), 72–

© Susanne Braun
43
@susannebraun
Proofs of the CALM Theorem

© Susanne Braun
45
@susannebraun
Avoidance
Primary
Copy
Ignoring
Last
writer
wins
Reducing the Chance
Reduce
granularity
of
conflicting
items,
inconsistency
windows,
etc.
Syntactic
Timestamps,
logical
clocks,
identifiers,
..
Semantic
Domain-specific
rules
Interesting stuff!

© Susanne Braun
47
@susannebraun
Pat Helland
Database & Distributed Systems Guru
Architect of multiple transaction &
database systems (e.g. DynamoDB)
Worked at Microsoft, Amazon, SalesForce, …
“Immutability Changes Everything”
ACM Queue, Volume 13, Issue 9, 2016

© Susanne Braun
48
@susannebraun
T
a
x
o
n
o
m
y
1st Level Classification of Replicated Aggregates
Mutable ?
Immutable
Aggregates*
No
Concurrent In-Place Updates ?
Yes
No
Derived
Aggregates*
Multiple Updaters ?
Yes
Dedicated
Aggregates
Nontrivial
Aggregates
Yes
No
* “Append-Only Computing” – Helland 2015
Immutable Aggregates:
• Time series data (machine sensor
data, market data, …)
• Domain events
Derived Aggregates:
• Machine generated data
(recommendations, …)
• Timeline or newsfeed data
Dedicated Aggregates:
• User generated data (reviews,
social media posts, ...)
• Dedicated master data (user
profiles, account settings)
E x a m p l e s

© Susanne Braun
49
@susannebraun
T
a
x
o
n
o
m
y
2nd Level Classification of Nontrivial Aggregates
Activity Aggregates
Collaboration Result
Aggregates
Reference Aggregates
Dedicated Aggregates
Derived Aggregates
Observed Aggregates
Update Frequency in Peak
Times
Update Simultaneity in Peak
Times
Concurrency Anomaly Probability
low improbable low
-
Nontrivial
Aggregates
Reference Aggregates Examples:
• Master data (CRM data, resources, products, …)
• Values (Valid currencies, product types, gender, …)
• Meta data (Tags, descrtiptive data of raw data, ..)

© Susanne Braun
50
@susannebraun
T
a
x
o
n
o
m
y
Activity Aggregates
Aggregates
Derived Aggregates
Observed Aggregates
Times
Times
low
high
improbable
probable
low
high
-
Nontrivial
Aggregates
Activity Aggregates Examples:
• State data of workflows, business processes, …
• Coordination data of joint activities (agricultural field
operation, meeting, …)
• Task management data, Kanban board data, …

© Susanne Braun
51
@susannebraun
T
a
x
o
n
o
m
y
Activity Aggregates
Aggregates
Derived Aggregates
Observed Aggregates
Times
Times
low
high
very high
improbable
probable
highly probable
low
high
very high
-
Nontrivial
Aggregates
Activity Aggregates Examples:
• State data of workflows, business processes, …
• Coordination data of joint activities (agricultural field
operation, meeting, …)
• Task management data, Kanban board data, …
Collaboration Result Aggregates Examples:
• Result data of collaborative knowledge work (CAD model,
crop rotation plan, whiteboard diagram, …)
• Text data as result of collaborative authorship (manuals,
scientific papers, meeting protocols, …)

© Susanne Braun
52
@susannebraun
T
a
x
o
n
o
m
y
Activity Aggregates
Aggregates
Derived Aggregates
Immutable Aggregates
Times
Times
“Technical Immutability Border”
low
high
very high
improbable
probable
highly probable
low
high
very high
- low
improbable
Nontrivial
Aggregates

© Susanne Braun
55
@susannebraun
B
e
s
t
P
r
a
c
t
i
c
e
Make Well-Informed Trade-Off Decisions
Concurrency Anomaly
Probability
Fixing Costs of Data
Corruption
low
high
very high
very high
high
very high
moderate
Activity Aggregates
Aggregates
Derived Aggregates
low
Consequences of Data
Corruption
critical
major
critical
minor
Nontrivial
Aggregates
“Technical Immutability Border”
Trivial
Aggregates
A Classification of Replicated Data for the Design of Eventually Consistent Domain Models, S. Braun, S. Dessloch,
ICSA 2020

© Susanne Braun
56
@susannebraun
is standard
Estimation - Frequency of Distribution
Trad. Enterprise IS
(ERP, CRM, Workflow Management)
Social Media Apps
(Facebook, Twitter)
Next Data-Intensive Systems
(Smart Farming, Industrie 4.0)
30%
30%
9 % 45%
4%
1%
20%
20%
20%
20%
“Technical
Immutability
Border”
Activity Aggregates
Aggregates
Derived Aggregates
1 % 50% 20%
30 %

© Susanne Braun
57
@susannebraun
B
e
s
t
P
r
a
c
t
i
c
e
Trivial Aggregates First
n Whenever feasible, model aggregates as trivial aggregates
Examples of Immutable Aggregates
• Domain Events !!
• A confirmed order
• A released shift plan
• A submitted questionnaire
• A submitted offer
• A placed bid
• A survey gone live
• ….
Examples of Derived Aggregates
Derived Monotonic State (CALM Theorem!)
• The highest bid (max) of an auction
• The state of progressive workflows (state changes
increase monotonically)
• The number of in-stock items (goods receipts and
order confirmations are monotonic sets)
Derived Views
• Mobile-optimized representations
• …
Low hanging fruit that is often overlooked!

© Susanne Braun
59
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
C
o
n
t
e
x
t
A n t i - P a t t e r n
ShoppingCart
id: Identifier
LineItem
productRef: Identifier
number: int
0..*
1
Concurrent In-place Updates
Nontrivial Activity Aggregate
Product
productId: Identifier
Reference Aggregate
DeleteItemAction
AddItemAction
0..*
1
0..*
1

© Susanne Braun
60
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
F
o
r
c
e
s
n Performance and availability of functionality implemented with or based on the aggregate is a crucial
business factor.
n A conflict resolution scheme that is “good enough” cannot be implemented with reasonable effort.

© Susanne Braun
61
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
S
o
l
u
t
i
o
n
Activity
fun deriveCurrentSate(…)
Entity
Value Object
Activity State
Entity
Value Object
Entity
Value Object
Entity
Value Object
Domain Event
Entity
Value Object
Less complex Activity
Aggregate
Derived Aggregate Immutable Aggregates
Cross-Aggregate References
Factor Out
Activity
Entity
Value Object
Entity
Event

© Susanne Braun
62
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
C
o
n
t
e
x
t
ShoppingCart
id: Identifier
LineItem
number: int
0..*
1
Concurrent In-place Updates
Product
productId: Identifier
Reference Aggregate
DeleteItemAction
AddItemAction
0..*
1
0..*
1

© Susanne Braun
63
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
S
o
l
u
t
i
o
n
E
x
a
m
p
l
e
ShoppingCart
id: Identifier
Derived Aggregate
LineItem
number: int
1
1..*
LineItems
cartRef: Identifier
Product
id: Identifier
Calculated
periodically or on
demand
AddItemAction
id: Identifier
cartRef: Identifier
DeleteItemAction
id: Identifier
cartRef: Identifier
getLineItems(): LineItem

© Susanne Braun
65
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
B
e
n
e
f
i
t
s
n Increased share of trivial aggregates
n Size and chance for conflicts of remaining nontrivial aggregates is considerably lower
n Complexity of the resulting model is reduced
n Aggregates are consisting of fewer domain objects with fewer object associations

© Susanne Braun
67
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
C
o
n
t
e
x
t
Field Assignment DocRecord
0..*
0..*
1
1
Job
id: Identifier
status: JobStatusEnum
OperationType
id: Identifier
name: String
StaffAssignment
MachineAssignment
Machine StaffMember
0..1
0..*
0..*
0..*
0..*
1
0..*
1
1
0..*
id: Identifier id: Identifier id: Identifier
id: Identifier id: Identifier
Dedicated to single
user
Reference data that
is rarely updated

© Susanne Braun
68
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
F
o
r
c
e
s
n Performance and availability of functionality implemented with or based on the aggregate is a crucial
business factor.
n A conflict resolution scheme that is “good enough” cannot be implemented with reasonable effort.

© Susanne Braun
69
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
S
o
l
u
t
i
o
n
Complex non-trivial
Aggregate
Dedicated
Aggregates
Cross-Aggregate References
Less complex non-
trivial Aggregate
Aggregate
Entity
Value Object
Reference
Aggregates
Entity
Value Object
Entity
Value Object
Aggregate
Entity
Value Object
Entity
Value Object
Entity
Value Object
Aggregate
Entity
Value Object
Derived
Aggregates
Entity
Value Object
Entity
Value Object
Aggregate
Entity
Value Object
Immutable
Aggregates
Entity
Value Object
Entity
Value Object
Aggregate
Entity
Value Object
Refactor Into
Aggregate
Entity
Value Object
Entity
Event Entity
Value Object
Event

© Susanne Braun
70
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
C
o
n
t
e
x
t
Field Assignment DocRecord
0..*
0..*
1
1
Job
id: Identifier
OperationType
id: Identifier
name: String
StaffAssignment
MachineAssignment
Machine StaffMember
0..1
0..*
0..*
0..*
0..*
1
0..*
1
1
0..*
id: Identifier id: Identifier id: Identifier
id: Identifier id: Identifier
Dedicated to single
user
Reference data that
is rarely updated

© Susanne Braun
71
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
S
o
l
u
t
i
o
n
E
x
a
m
p
l
e
Less complex Activity Aggregate
Assignment
0..*
1
Job
id: Identifier
operationType: OperationTypeEnum
fields: Set<Identifier>
docRecords: Set<Identifier>
MachineAssignment
id: Identifier
machineRef: Identifier
StaffAssignment
id: Identifier
staffMemberRef: Identifier
Field
id: Identifier
Machine
id: Identifier
StaffMember
id: Identifier
DocRecord
id: Identifier
jobRef: Identifier
staffMemberRef: Identifier
Dedicated Aggregate

© Susanne Braun
73
@susannebraun
D
e
s
i
g
n
P
a
t
t
e
r
n
B
e
n
e
f
i
t
s
n By segregation of domain objects belonging to different classes large aggregates can be significantly
reduced in terms of size, complexity and ramification of object associations
n As aggregates become smaller the chances for conflicting updates are reduced
n By factoring out immutable, derived and dedicated data into standalone aggregates the advantages of
trivial aggregates can be exploited as for these aggregates no conflict resolution schemes are required.
n By factoring out reference aggregates low update frequency of reference data can be exploited
n As chances for conflicting updates are low simple automatic reconciliation schemes or even manual
reconciliation schemes are usually sufficient.

© Susanne Braun
77
@susannebraun
Open Access Design Guidelines
n Code Examples
n https://github.com/brausu/workshop-eventual-consistency
n Follow me on Twitter and GitHub
@susannebraun
EventuallyConsistentDDD/design-guidelines
We’ re on
Github!
Thanks!

© Susanne Braun
79
@susannebraun
Checkout our other publications
https://doi.org/10.1109/ICSA-C50368.2020.00014 https://doi.org/10.1145/3447865.3457969

© Susanne Braun
80
@susannebraun
#Thanx
SAP Signavio, Berlin
Susanne Braun
Principal Tech Lead
susanne.braun01@sap.com
@susannebraun
EventuallyConsistentDDD/design-guidelines

w-jax 2022: Keeping CALM – Konsistenz in verteilten Systemen leichtgemacht

Recommended

Recommended

More Related Content

What's hot

What's hot (16)

Similar to w-jax 2022: Keeping CALM – Konsistenz in verteilten Systemen leichtgemacht

Similar to w-jax 2022: Keeping CALM – Konsistenz in verteilten Systemen leichtgemacht (20)

Recently uploaded

Recently uploaded (20)

w-jax 2022: Keeping CALM – Konsistenz in verteilten Systemen leichtgemacht