Software development has advanced immensely throughout the years, whether it's technologies, tools or methodologies. Be that as it may, it is quite surprising to see how some concepts remain extremely difficult still. One of the big challenges in developing any system is dealing with consistency, and going towards more extreme cases of distributed systems makes it even a more serious conundrum. What can we you do about it then? How do you address it? What approaches and tools should you consider? How can you design (distributed) systems and achieve high consistency guarantees? Join me in this talk as we tackle such questions head-on and leave better equipped to build distributed systems that maintain application state and workflows with consistency in mind.

1. Dealing with
Consistency in
Distributed
Systems
Amir Zuker
2 8 . 0 6 . 2 3

3. Fancy Books
(decades ago)

4. Decades Ago
RDBMS
Service
Service
Service
Use-case: Submit Order

5. Decades Ago
RDBMS
Service
Service
Service
Use-case: Submit Order
function submitOrder(request) {
try {
const tx = createDbTransaction();
const order = await createOrder(tx, request);
const invoice = await performPayment(tx, order);
await initiateShipment(tx, order);
await updateInventory(tx, order);
await tx.commit();
return formatResponse(request, invoice);
} catch (error) {
tx.rollback();
throw error;
}
}

6. Fancy Books
(today)

7. Today
Db
Payment
Service
Order
Service
Shipping
Service
Db
Db
Inventory
Service
Db
Use-case: Submit Order

8. About Me
10
Amir Zuker
 Senior software architect, developer
 Over 25 years of experience in the field
 Leads teams and R&D divisions
 Several publications and contributions
 CTO at CodeValue
@amirzuker
amirz@codevalue.net

9. Agenda
 Consistency in Distributed Systems
 Immediate vs. Eventual Consistency
 Distributed Sagas
 Consistency as a Feature
11

10. Consistency in
Distributed Systems
12

11. Consistency – A Broad Term
 Database systems
 Data is “correct”
 Data consistency
 Sync across replicas
 Software engineering
 Business transactions
 Overall state is sound (integrity included)
13
RDBMS
Replica Replica Replica
Service
Service
Service

12. Consistency + Reliability
Error
Handling
Fault
Tolerance
Durability
Testing
Race
Conditions
Business
Transactions
Redundancy
Data
Integrity
Data
Consistency
Concurrency

13. Distributed Business Flows
 A business transaction can span multiple services
 Distributed systems:
“components are located on different networked computers”
15
Db
Service
Service
Service
Db
Db

14. 16

15. To distribute
or not?
17

16. Why Distribute?
 Nature of the application
 Organization
 Structure and communication flow
 Politics
 Technical needs
 Scale, Redundancy, Availability, Polyglot, etc.
 Masochism

17. Distribute!
19

18. ACID
 ACID
 Atomic, Consistent, Isolated, Durable
 Strong and immediate consistency
 All or nothing
 Correct
 No side effects
 Persisted
Db
Service

19. Can we do ACID in distributed flows?

20. ACID-like Distributed Business Transaction
22
Db
Db
Db
Service
Service
Service
Service

21. ACID-like Distributed Business Transaction
23
Db
Db
Db
Service
Service
Service
Service

22. ACID-like Distributed Business Transaction
24
Db
Db
Db
Service
Service
Service
Service
A  B
A  B
A  B
Service
A

23. ACID-like Distributed Business Transaction
25
Db
Db
Db
Service
Service
Service
Service
A  B
A  B
Service
A

24. ACID-like Distributed Business Transaction
26
Db
Db
Db
Service
Service
Service
Service
A  B
A  B
Service
A

25. ACID in Distributed Systems
 Difficult and complex
 No silver bullet
 Today’s expectations
 Need reliable distributed flows?
 Probably look elsewhere
27

26. Eventual Consistency
 Opposite of immediate consistency
 If new updates aren’t made to a data item:
Eventually, all reads will return the last updated value
 Weak consistency guarantees
 Only liveness
28

27. Life
is
Eventually
Consistent

28. How can we achieve reliable distributed flows
with Eventual Consistency?

29. Distributed Sagas
31

30. Distributed Sagas
 Aggregates smaller (atomic) operations
 Uses compensation to handle failures
 Coordinated via orchestration or choreography
 Many implementations exist in the wild Db Db
Db
Service
Service Service

31. 1. Orchestration, Commands, Sequential
33
Order
Orchestrator
Payment
Inventory
Shipment
Message Broker
Reply Queue

32. 1. Orchestration, Commands, Sequential
34
Order
Orchestrator
Payment
Inventory
Shipment
Message Broker
Reply Queue

33. 2. Choreography, Events, Sequential
35
Order
Payment
Inventory
Shipment
Message Broker
Order Created
Paid
Stock Updated
Shipped

34. 2. Choreography, Events, Sequential
36
Order
Payment
Inventory
Shipment
Message Broker
Order Created
Paid
Out Of Stock

35. Distributed Sagas
 Guarantees either:
 All requests were completed successfully
OR..
 A subset of requests and their compensating requests were executed
 The system is consistent, eventually
 No atomicity
 No isolation

36.  Asynchronous messaging
 Persisted workflows
 Scheduler, agent, supervisor
39

37. Asynchronous Messaging
 Persistence
 No message loss
 Retries
 …requires Idempotence!
 Concurrency / Ordered
 Relevance
 Time-to-live (TTL)
 Dead-lettering
 Need to decide what to do with it afterwards though
 Loose coupling
 In terms of connectivity especially
40

38. Persisted Workflows
42
Source: https://github.com/Azure-Samples/saga-orchestration-serverless

39. Scheduler, Agent, Supervisor
43
Source: https://learn.microsoft.com/en-us/azure/architecture/patterns/scheduler-agent-supervisor

40. Oh, and one more small thing..
44
Test,
test,
and test..

41. Consistency in not only
about engineering 45

42. Consistency as a Feature
46

43. Product and UX/UI

44. Consistency as a Feature
 Unmet expectations  Bugs
 Affects
 Quality
 Customer trust and satisfaction
 Involve the product
 Negotiate, both ways

45. 50
Eventual Consistency is
NOT
the new
Inconsistent

46. Product and UX/UI Mitigations

47. Design Considerations
 What must happen “immediately”
 What can take place later (eventually)
 What can be happy path only
 What can be made in parallel
 What level of control and accuracy is needed

48. Design Considerations
 What must happen “immediately”
 What can take place later (eventually)
 What can be best effort or happy path only
 What can be made in parallel
 What level of accuracy is needed

49. Key Takeaways
54

50. Key Takeaways
 Design
 Define business transaction boundaries
 Identify required consistency guarantees
 Not as an afterthought
 Involve product
 Realize and negotiate requirements
 Distributed or not, a choice
 Service granularity
 Consistency, an integrator force
 Test
 Address technical aspects
 Patterns
 Sagas, scheduler-agent-supervisor, etc.
 Coordination
 Orchestration or choreography
 Async messaging
 Persistence, retries, idempotence, TTL, DLQ
 Concurrency, connectivity, etc.
 Persisted workflows
 Be consistent
 ..as needed!
55

52. 57
Slides:
https://www.slideshare.net/amirzuker/
business-transactions-in-distributed-systems
Dealing with Consistency
in Distributed Systems
Senior Software Architect
CTO at CodeValue
amirz@codevalue.net
Amir Zuker
@amirzuker