1. THE STORY OF ADI

2. Meet
Adi Awanta.

3. He has a dream.

4. Adi is fashionable
• Event sourcing is
the future, they say
• Event sourcing is
the shit, he agrees

5. Adi is fashionable
• Event sourcing is
the future, they say
• Event sourcing is
the shit, he agrees
• … and designs an
event model
Created
Modified
Published
Archived
Restored

6. Column Type Key Null
site_id binary(16)
event_time timestamp
event_type enum(…)
payload mediumblob
Adi is fashionable

7. HEY, KIDS
Can you guess what happens next?

8. Adi is confused
• Bad shit happens
• Event streams exhibit:
– Out of order events
– Conflicting events
– Impossible states

9. How shit works:
Time
Tomer Gabel
Wix Engineering Conference
May 2017
Image: Vera Kratochvil (public domain)

10. Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

11. Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

12. Modeling time
• Encoding
– Resolution
– Epoch
“Real” time
Epoch
Resolution
T1 T2 T3 … Tn

13. Modeling time
• Encoding
– Resolution
– Epoch
T3
Instant
(or “event”)

14. Modeling time
• Encoding
– Resolution
– Epoch
T3
A
B
Conflicting
events

15. Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
Epoch
???

16. Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
• Updating
T1 T2

17. Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
• Updating T = T+1
T1 T2
W
hen?

18. System Clock (RTC)
Modeling time
• Encoding
– Resolution
– Epoch
• Bootstrapping
– Manual
– Battery-backed
– NTP
• Updating
Image: Jeremy Saglimbeni on Vimeo (CC BY-SA 3.0)

19. RTC isn’t perfect
• Alas, clocks drift
• Subtle causes
– Temperature
– Power supply
– General relativity
– Cosmic radiation
– Alien gamma rays
Image showing ~220µs clock drift by Luke Bigum

20. Distributed time
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event Stream Event
Store
RTC
event_time = ?

21. Time source: Application
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event Stream Event
Store
RTC
1
2
3
4

22. Time source: Application
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event Stream Event
Store
RTC
1
2
3
4
Clocks must be
synchronized!

23. What about NTP?
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event
Store
RTC
NTP

24. What about NTP?
Host A
Host C
Host B
Host D
RTC RTC
RTC RTC
Event
Store
RTC
NTP
Accurate within
~10ms
Fig. 1, “Characterizing Quality of Time and Topology in a Time Synchronization Network”, Murta et al

25. DEALING WITH TIME IS HARD
TL;DR

26. Reframing the problem
• When reading events
– Do we need wall time?
– We don’t care about it
– It’s just metadata
• We only care about
ordering events
Created
(2017-05-03 10:15)
Updated
(2017-05-03 10:27)
Archived
(2017-05-03 10:55)

27. Reprise: Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

28. Reprise: Time (noun)
“… the system of those sequential
relations that any event has to any other,
as past, present, or future; indefinite and
continuous duration regarded as that in
which events succeed one another.”
-- dictionary.com

29. Causality
• Take any two events
• What relationship can
they have?
Created
Updated Updated
Archived

30. Causality
• Take any two events
• What relationship can
they have?
– Happens before: A→B
A. Created
B. Updated Updated
Archived

31. Causality
• Take any two events
• What relationship can
they have?
– Happens before: A→B
– Concurrent: A↛B and
B↛A
Created
A. Updated B. Updated
Archived

32. Lamport timestamps
• Provides partial
ordering of events
• Advantages:
– Respects causality
– Low overhead
– Simple to implement
Event
Store
Host
A
Host
C
Host
B
“Time, Clocks, and the Ordering of Events in a Distributed System”, Leslie Lamport, 1978

33. Event
Store
Host
A
Host
C
Host
B
Lamport timestamps
• Each host maintains
local logical clock
• Start with T=0
T=0
T=0T=0

34. Event
Store
Host
A
Host
C
Host
B
Lamport timestamps
• Each host maintains
local logical clock
• Start with T=0
• On send (i.e. write):
– Increment local clock
– Attach to message
T=0
T=0T=0
T=1

35. Lamport timestamps
• On receive (i.e. read):
– Process event if T < Tin
Event
Store
Host
A
Host
C
Host
B
T=0
Tin=1

36. Lamport timestamps
• On receive (i.e. read):
– Process event if T < Tin
– Update clock past the
latest event:
T = max(T, Tin) + 1
• All done!
Event
Store
Host
A
Host
C
Host
B
T=0
Tin=1
T=2

37. … well, almost
• This is a partial order
• Causality is dealt with
– A→B ⇒ T(A) < T(B)
• What about concurrency?
– A↛B and B↛A ⇒ ?

38. Final touches
• We want total ordering
– Must be stable
– Need not correspond to
real time
Created
Updated Updated
Archived
T=1
T=5
T=2T=2

39. Final touches
• We want total ordering
– Must be stable
– Need not correspond to
real time
• Breaking the tie:
– Use any consistent key
– Host name, MAC, IP…
Created
Updated Updated
Archived
T=1
T=5
T=2T=2

40. What did we gain?
• We can assign versions that…
– Respect causality
– Are totally ordered
– Don’t rely on the RTC
• We’ve enabled serializability!

41. MAKE ADI
GREAT AGAIN!
Enough theory. Let’s

42. Versioning
• Our versioning
mechanism must:
– Respect causality
– Detect conflicting
writes
– Not rely on wall time
• We’re almost there!

43. Relax
• It’s fairly simple
in practice
• First, the schema
Column Type Key Null
site_id binary(16)
event_time timestamp
event_type enum(…)
payload mediumblob

44. Relax
• It’s fairly simple
in practice
• First, the schema
– Don’t key on
timestamp
Column Type Key Null
site_id binary(16)
event_time timestamp
event_type enum(…)
payload mediumblob

45. Relax
• It’s fairly simple
in practice
• First, the schema
– Don’t key on
timestamp
– Add explicit
version
Column Type Key Null
site_id binary(16)
version mediumint
event_time timestamp
event_type enum(…)
payload mediumblob

46. Finishing touches
• On writes:
MySQL
Server
Client
C1-Cn

47. Finishing touches
• On writes:
– Read latest version V0
MySQL
Server
Client
C1-Cn
V0

48. Finishing touches
• On writes:
– Read latest version V0
– Generate events V1...Vn
– Write atomically
MySQL
Server
Client
C1-Cn
V1-Vn

49. Finishing touches
• On writes:
– Read latest version V0
– Generate events V1...Vn
– Write atomically
• Success!
– Return new version
MySQL
Server
Client
C1-Cn
V1-Vn
Vn

50. Finishing touches
• On writes:
– Read latest version V0
– Generate events V1...Vn
– Write atomically
• Duplicate primary key?
– Optimistic lock failure!
– Retry, propagate, resolve
MySQL
Server
Client
C1-Cn
V1-Vn
Error
PK violation

51. GREAT
SUCCESS!

52. KFIR ‘ADI’ BLOCH
A round of applause
for our special guest:
... and for the terrific
camera work I butchered:
VAIDAS PILKAUSKAS

53. QUESTIONS?
Thank you for listening
tomer@tomergabel.com
@tomerg
http://engineering.wix.com
On GitHub:
https://github.com/holograph
This work is licensed under a Creative
Commons Attribution-ShareAlike 4.0
International License.