The document tells the story of Adi and his journey working with event sourcing. It begins by introducing Adi and his dream. It then describes some of the challenges Adi faces with event streams exhibiting out of order, conflicting, and impossible state events. The document discusses how time works in distributed systems and challenges with clock drift. It proposes using causality and Lamport timestamps to provide a partial ordering of events while avoiding reliance on real time. Finally, it provides suggestions for Adi's data model and write process to fully order events and detect conflicts while respecting causality.

1. THE STORY OF ADI

2. Meet
Adi Awanta.

3. He has a dream.

4. Adi is fashionable
• Event sourcing is
the future, they say
• Event sourcing is
the shit, he agrees

5. Adi is fashionable
• Event sourcing is
the future, they say
• Event sourcing is
the shit, he agrees
• … and designs an
event model
Created
Modified
Published
Archived
Restored

6. Column Type Key Null
site_id binary(16)
event_time timestamp
event_type enum(…)
payload mediumblob
Adi is fashionable

7. HEY, KIDS
Can you guess what happens next?

8. Adi is confused
• Bad shit happens
• Event streams exhibit:
– Out of order events
– Conflicting events
– Impossible states

9. How shit works:
Time
Tomer Gabel
Wix Engineering Conference
May 2017
Image: Vera Kratochvil (public domain)

10. Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

11. Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

12. Modeling time
• Encoding
– Resolution
– Epoch
“Real” time
Epoch
Resolution
T1 T2 T3 … Tn

13. Modeling time
• Encoding
– Resolution
– Epoch
T3
Instant
(or “event”)

14. Modeling time
• Encoding
– Resolution
– Epoch
T3
A
B
Conflicting
events

15. Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
Epoch
???

16. Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
• Updating
T1 T2

17. Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
• Updating T = T+1
T1 T2
W
hen?

18. System Clock (RTC)
Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
• Updating
Image: Jeremy Saglimbeni on Vimeo (CC BY-SA 3.0)

19. RTC isn’t perfect
• Alas, clocks drift
• Subtle causes
– Temperature
– Power supply
– General relativity
– Cosmic radiation
– Alien gamma rays
Image showing ~220µs clock drift by Luke Bigum

20. Distributed time
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event Stream Event
Store
RTC
event_time = ?

21. Time source: Application
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event Stream Event
Store
RTC
1
2
3
4

22. Time source: Application
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event Stream Event
Store
RTC
1
2
3
4
Clocks must be
synchronized!

23. What about NTP?
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event
Store
RTC
NTP

24. What about NTP?
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event
Store
RTC
NTP
Accurate within
~10ms
Fig. 1, “Characterizing Quality of Time and Topology in a Time Synchronization Network”, Murta et al

25. DEALING WITH TIME IS HARD
TL;DR

26. Reframing the problem
• When reading events
– Do we need wall time?
– We don’t care about it
– It’s just metadata
• We only care about
ordering events
Created
(2017-05-03 10:15)
Updated
(2017-05-03 10:27)
Archived
(2017-05-03 10:55)

27. Reprise: Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

28. Reprise: Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

29. Causality
• Take any two events
• What relationship can
they have?
Created
Updated Updated
Archived

30. Causality
• Take any two events
• What relationship can
they have?
– Happens before: A→B
A. Created
B. Updated Updated
Archived

31. Causality
• Take any two events
• What relationship can
they have?
– Happens before: A→B
– Concurrent: A↛B and
B↛A
Created
A. Updated B. Updated
Archived

32. Lamport timestamps
• Provides partial
ordering of events
• Advantages:
– Respects causality
– Low overhead
– Simple to implement
Event
Store
Host
A
Host
C
Host
B
“Time, Clocks, and the Ordering of Events in a Distributed System”, Leslie Lamport, 1978

33. Event
Store
Host
A
Host
C
Host
B
Lamport timestamps
• Each host maintains
local logical clock
• Start with T=0
T=0
T=0T=0

34. Event
Store
Host
A
Host
C
Host
B
Lamport timestamps
• Each host maintains
local logical clock
• Start with T=0
• On send (i.e. write):
– Increment local clock
– Attach to message
T=0
T=0T=0
T=1

35. Lamport timestamps
• On receive (i.e. read):
– Process event if T < Tin
Event
Store
Host
A
Host
C
Host
B
T=0
Tin=1

36. Lamport timestamps
• On receive (i.e. read):
– Process event if T < Tin
– Update clock past the
latest event:
T = max(T, Tin) + 1
• All done!
Event
Store
Host
A
Host
C
Host
B
T=0
Tin=1
T=2

37. … well, almost
• This is a partial order
• Causality is dealt with
– A→B ⇒ T(A) < T(B)
• What about concurrency?
– A↛B and B↛A ⇒ ?

38. Final touches
• We want total ordering
– Must be stable
– Need not correspond to
real time
Created
Updated Updated
Archived
T=1
T=5
T=2T=2

39. Final touches
• We want total ordering
– Must be stable
– Need not correspond to
real time
• Breaking the tie:
– Use any consistent key
– Host name, MAC, IP…
Created
Updated Updated
Archived
T=1
T=5
T=2T=2

40. What did we gain?
• We can assign versions that…
– Respect causality
– Are totally ordered
– Don’t rely on the RTC
• We’ve enabled serializability!

41. MAKE ADI
GREAT AGAIN!
Enough theory. Let’s

42. Versioning
• Our versioning
mechanism must:
– Respect causality
– Detect conflicting
writes
– Not rely on wall time
• We’re almost there!

43. Relax
• It’s fairly simple
in practice
• First, the schema
Column Type Key Null
site_id binary(16)
event_time timestamp
event_type enum(…)
payload mediumblob

44. Relax
• It’s fairly simple
in practice
• First, the schema
– Don’t key on
timestamp
Column Type Key Null
site_id binary(16)
event_time timestamp
event_type enum(…)
payload mediumblob

45. Relax
• It’s fairly simple
in practice
• First, the schema
– Don’t key on
timestamp
– Add explicit
version
Column Type Key Null
site_id binary(16)
version mediumint
event_time timestamp
event_type enum(…)
payload mediumblob

46. Finishing touches
• On writes:
MySQL
Server
Client
C1-Cn

47. Finishing touches
• On writes:
– Read latest version V0
MySQL
Server
Client
C1-Cn
V0

48. Finishing touches
• On writes:
– Read latest version V0
– Generate events V1...Vn
– Write atomically
MySQL
Server
Client
C1-Cn
V1-Vn

49. Finishing touches
• On writes:
– Read latest version V0
– Generate events V1...Vn
– Write atomically
• Success!
– Return new version
MySQL
Server
Client
C1-Cn
V1-Vn
Vn

50. Finishing touches
• On writes:
– Read latest version V0
– Generate events V1...Vn
– Write atomically
• Duplicate primary key?
– Optimistic lock failure!
– Retry, propagate, resolve
MySQL
Server
Client
C1-Cn
V1-Vn
Error
PK violation

51. GREAT
SUCCESS!

52. KFIR ‘ADI’ BLOCH
A round of applause
for our special guest:
... and for the terrific
camera work I butchered:
VAIDAS PILKAUSKAS

53. QUESTIONS?
Thank you for listening
tomer@tomergabel.com
@tomerg
http://engineering.wix.com
On GitHub:
https://github.com/holograph
This work is licensed under a Creative
Commons Attribution-ShareAlike 4.0
International License.