This document discusses timekeeping in distributed systems. It begins by explaining how different types of clocks work, from pendulum clocks to atomic clocks. It then discusses key concepts like UTC, leap seconds, and how time is represented in Unix. The document also covers challenges of keeping time across distributed systems and algorithms like NTP, vector clocks, and logical clocks that help order events in a distributed system.

1. A SHORT STORY ABOUT TIME

2. INTRODUCTION
Bartosz Sypytkowski
▪ @Horusiath
▪ b.sypytkowski@gmail.com
▪ bartoszsypytkowski.com

3.  Timekeeping in distributed systems
 UTC
 How clocks work
 System time
 NTP
 Logical time
AGENDA

4. Timekeeping
in
distributed
systems
Node 1
Bob Joe
SET X = 2SET X = 1

5. Timekeeping
in
distributed
systems
Node 1
Node 2
Bob Joe
SET X = 2SET X = 1

6. “Time is not a thing, it’s a subjective
experience.”

7. CLOCKS

8. PENDULUM
CLOCK
• Frequency: < 1Hz
• Precision: ~1 sec.

9. DIGITAL CLOCK • Frequency: 50Hz – 60Hz
• Precision: 16-20ms

10. QUARTZ
CLOCK
• Frequency: 32kHz – 2MHz
• Precision: 0.5µs - 31µs

11. ATOMIC CLOCK • Frequency (Cessium-133):
9,192,631,770Hz
• Precision: ~0.1ns

12. GPS CLOCK
SYNC

13. UNIX TIME
 Number of seconds since epoch (1970/01/01)
 It has exactly 86 400s per day…

14. • Day does not take 24h…
• … it’s not even a constant.
A TRUTH ABOUT DAY

15. UTC
ATOMIC TIME
BASED ON ATOMIC CLOCK
SOLAR/ASTRONOMICAL TIME
BASED ON EARTH ROTATION
UTC
A COMPROMISE BASED ON ATOMIC
TIME IRREGULARLY ADJUSTED TO
SOLAR TIME

16. LEAP
SECONDS
UTC
23:59:58
23:59:59
23:59:60
00:00:00
00:00:01
UNIX
23:59:58
23:59:59
23:59:59
00:00:00
00:00:01

17. CONSEQUENCES OF
NON-MONOTONIC
TIME
Node 1
Node 2
Bob Joe
ts = 60
SET X = 2SET X = 1
ts = 20

18. LEAP SECONDS IN PRACTICE

19. NETWORK
TIME
PROTOCOL

20. NETWORK
TIME
PROTOCOL
1. Skew – speedup/slowdown clock tick
rate
2. Step – set a new value when difference
is too big (128ms)

21. NTP PROBLEMS
▪ Stepping interferes with happen-before
relationships.
▪ It’s slow.
▪ It assumes, that network is symmetric.
PRECISION TIME
PROTOCOL
▪ Available in Windows Server 2019
(opt-in)
▪ Takes into account a latency per
each network device

22. OS TIME
API
WINDOWS
wall-clock time
LINUX
wall-clock time
WINDOWS
monotonic time
LINUX
monotonic time
clock_gettime(CLOCK_REALTIME)
clock_gettime(CLOCK_MONOTONIC)QueryPerformanceCounter(&time)
GetSystemTimePreciseAsFileTime(&date)

23. “In transactions, timekeeping is more
about order of operations than actual
time.”

24. SEQUENCERS
Set X = 1 ts = 1
Set Y = 2 ts = 2
Set X = 3 ts = 3
Delete Y ts = 4
Set Y = 1 ts = 5
Delete X ts = 6
Monotonically incrementing counter
timeline
Node A

25. DOTS
Set X = 1 A:1
Set X = 3 A:2
Set Y = 1 A:3
Delete X A:4
timeline
Node A
Set Y = 2 B:1
Delete Y B:2
timeline
Node B

26. VECTOR
CLOCKS
Set X = 1
Set X = 3
Set Y = 1
Delete X
Node A
Set Y = 2
Delete Y
Node B
A 1
A 1
B 1
A 2
B 1 A 2
B 2
A 3
B 2
A 4
B 2

27. COMPAR
ING
VECTOR
CLOCKS
PARTIAL
ORDERING
Equals Greater than
Less than Concurrent
A 2
B 3
C 1
A 2
B 3
C 1
A 2
B 3
C 2
A 2
B 3
C 1
A 2
B 2
C 1
A 2
B 3
C 1
A 2
B 3
C 2
A 2
B 2
C 1

28. COMPAR
ING
VECTOR
CLOCKS
RESOLVING
CONCURRENT
UPDATES
Node 1
Node 2
Bob Joe
t1
SET X = 2SET X = 1
t2

29. COMPAR
ING
VECTOR
CLOCKS
RESOLVING
CONCURRENT
UPDATES
Node 1
Node 2
Bob Joe
t1
SET X = 2SET X = 1
t2
max (t1, t2)
max (t1, t2)

30. LAST
WRITE
WINS
MAX (2018-09-15T15:53:00, 2019-01-12T17:47:13)
=
2019-01-12T17:47:13

31. VECTOR
CLOCK
MERGE
MAX ( , )
=
A 1
B 3
A 2
B 1
C 1
A 2
B 3
C 1

32. Overhead:
Nr. of nodes * (Key size + seq. nr.)
Examples:
3 × (4 + 8) = 36 bytes
10 × (16 + 8) = 240 bytes
DATETIME / SEQ. NR.
Overhead:
8 bytes
VECTOR CLOCKS

33. DOTTED
VERSION
VECTORS

34. DOTTED
VERSION
VECTORS
REPRESENTATION
D 4
C 5
B 7
A 5
D 8
C 10
C 11
A 9
version vector dot cloud
+

35. HOW TO DETERMINE “SEEN BY ALL”?
DOT VECTOR
CLOCK
???

36. MATRIX
CLOCKS
2 2 1
2 3 1
2 2 2
A
B
C
A B C
Observers
Observed

37.  Keeping time in real systems: https://www.youtube.com/watch?v=BRvj8PykSc4
 Windows plans for leap seconds: https://blogs.technet.microsoft.com/networking/2018/07/18/top10-ws2019-hatime/
 Dotted version vectors: http://basho.com/posts/technical/vector-clocks-revisited-part-2-dotted-version-vectors/
REFERENCES

38. THANK YOU