Existing clock synchronization algorithms assume a bounded clock reading error. This, in turn, results in an inflexible design that typically requires node crashes whenever the given bound might be violated. We propose a novel, adaptive internal clock synchronization algorithm which allows to compute the deviation between the clocks during runtime. The computed deviation can be propagated to the application layer to allow it to adapt its behavior according to the current clock deviation. The contributions of this paper are: (1) a new specification of a relaxed clock synchronization problem, and (2) a new clock synchronization algorithm with a novel approach to dealing with crash failures.

1. Motivation Background Adaptive Clock Sync. Evaluation Summary 1 of 25 slides
Adaptive Internal Clock Synchronization
SRDS 2008
Zbigniew Jerzak, Robert Fach, and Christof Fetzer
October 6, 2008
{Zbigniew.Jerzak, Robert.Fach, Christof.Fetzer}@inf.tu-dresden.de
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

2. Motivation Background Adaptive Clock Sync. Evaluation Summary 2 of 25 slides
Introduction
Build safer distributed systems using weaker assumptions [FC03]
In this paper/presentation:
Relaxed specification for internal clock synchronization
New clock synchronization algorithm
New way to deal with crash failures
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

3. Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides
Current Approach
1. Assume a bounded deviation ∆max
2. However, in timed asynchronous applications:
messages get dropped or are arbitrarily delayed
different nominal and observed oscillator frequencies
clocks’ speeds fluctuate with time
processes crash
3. ...thus 1 cannot be guaranteed
Result:
An inherently inflexible and a failure-prone system
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

4. Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides
Current Approach
1. Assume a bounded deviation ∆max
2. However, in timed asynchronous applications:
messages get dropped or are arbitrarily delayed
different nominal and observed oscillator frequencies
clocks’ speeds fluctuate with time
processes crash
3. ...thus 1 cannot be guaranteed
Result:
An inherently inflexible and a failure-prone system
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

5. Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides
Current Approach
1. Assume a bounded deviation ∆max
2. However, in timed asynchronous applications:
messages get dropped or are arbitrarily delayed
different nominal and observed oscillator frequencies
clocks’ speeds fluctuate with time
processes crash
3. ...thus 1 cannot be guaranteed
Result:
An inherently inflexible and a failure-prone system
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

6. Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides
Current Approach
1. Assume a bounded deviation ∆max
2. However, in timed asynchronous applications:
messages get dropped or are arbitrarily delayed
different nominal and observed oscillator frequencies
clocks’ speeds fluctuate with time
processes crash
3. ...thus 1 cannot be guaranteed
Result:
An inherently inflexible and a failure-prone system
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

7. Motivation Background Adaptive Clock Sync. Evaluation Summary 4 of 25 slides
Our Approach
No upper bound for the deviation (∆max )
compute deviation between processes in real-time
use computed deviation to adapt system behavior
No bound on the maximum number of crashed processes
mask crashes using clock readings extrapolation
Result:
An adaptive, safer system
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

8. Motivation Background Adaptive Clock Sync. Evaluation Summary 4 of 25 slides
Our Approach
No upper bound for the deviation (∆max )
compute deviation between processes in real-time
use computed deviation to adapt system behavior
No bound on the maximum number of crashed processes
mask crashes using clock readings extrapolation
Result:
An adaptive, safer system
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

9. Motivation Background Adaptive Clock Sync. Evaluation Summary 4 of 25 slides
Our Approach
No upper bound for the deviation (∆max )
compute deviation between processes in real-time
use computed deviation to adapt system behavior
No bound on the maximum number of crashed processes
mask crashes using clock readings extrapolation
Result:
An adaptive, safer system
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

10. Motivation Background Adaptive Clock Sync. Evaluation Summary 5 of 25 slides
Use Case
p is assigned a time slot [S, T ] for access to R
[S, T ] defined w.r.t. some abstract clock
p is not synchronized with a priori known ∆max , instead...
Use Ep – maximum local clock synchronization error
an upper bound on the deviation from some abstract clock
calculated by every process
propagated to application layer for error handling
p uses R only within [S + Ep , T − Ep ]
p knows how well it is synchronized
p knows when it is allowed to access R
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

11. Motivation Background Adaptive Clock Sync. Evaluation Summary 5 of 25 slides
Use Case
p is assigned a time slot [S, T ] for access to R
[S, T ] defined w.r.t. some abstract clock
p is not synchronized with a priori known ∆max , instead...
Use Ep – maximum local clock synchronization error
an upper bound on the deviation from some abstract clock
calculated by every process
propagated to application layer for error handling
p uses R only within [S + Ep , T − Ep ]
p knows how well it is synchronized
p knows when it is allowed to access R
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

12. Motivation Background Adaptive Clock Sync. Evaluation Summary 5 of 25 slides
Use Case
p is assigned a time slot [S, T ] for access to R
[S, T ] defined w.r.t. some abstract clock
p is not synchronized with a priori known ∆max , instead...
Use Ep – maximum local clock synchronization error
an upper bound on the deviation from some abstract clock
calculated by every process
propagated to application layer for error handling
p uses R only within [S + Ep , T − Ep ]
p knows how well it is synchronized
p knows when it is allowed to access R
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

13. Motivation Background Adaptive Clock Sync. Evaluation Summary 6 of 25 slides
Processes and Hardware Clocks
based on TADSM [CF99]
Set of N = {p0 , p1 , . . . , pn } processes
Connected via communication bus
Hardware clocks – Hp (t):
bounded drift:
∀p∈N ∀t : |ρp (t)| ≤ ρmax
within linear envelope of real time:
(t − s)(1 − ρmax ) ≤ Hp (t) − Hp (s) ≤ (t − s)(1 + ρmax )
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

14. Motivation Background Adaptive Clock Sync. Evaluation Summary 8 of 25 slides
Generic Internal Clock Synchronization Algorithm
1 C l o c k V a l Ap ; // c u r r e n t a d j u s t m e n t
2 C l o c k V a l T ; // end o f c u r r e n t r o u n d
4 void i n i t () {
5 ( Ap ,T) = i n i t i a l A d j u s t m e n t ( ) ;
6 // e v e r y P s t a r t i n g a t T
7 schedule ( synchronizer , P, T) ;
8 }
10 void synchronizer () {
11 // N − number o f p r o c e s s e s
12 ClockVal c l k [N] ,
13 ClockVal e r r [N ] ;
14 // r e m o t e c l o c k r e a d i n g
15 readClocks ( clk , e r r ) ;
16 // a d j u s t m e n t f o r t h e n e x t r o u n d
17 Ap = a d j u s t ( Ap , T , c l k , e r r ) ;
18 // s e t T t o n e x t r o u n d
19 T = T + P;
20 }
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

15. Motivation Background Adaptive Clock Sync. Evaluation Summary 9 of 25 slides
Software Clocks and Remote Clock Reading
We do not apply Ap directly to the Hp (t)
Instead we use software clocks:
Sp (t) = Hp (t) + ap (t)
Ap is calculated based on the remote clock readings
probabilistic remote clock reading [FC99b]
provides values of the remote clocks clk[]
and accompanying remote clock reading errors err[]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

16. Motivation Background Adaptive Clock Sync. Evaluation Summary 10 of 25 slides
Failure Model
Messages are supposed to be delivered within δ
might be delayed
delivered out of order
get dropped
Arbitrary hardware clock failures: |ρp (t)| > ρmax
converted to stop failures [FC99a]
Processes’ value (byzantine) failures
handled by Software Encoded Processing [WF07]
Specifically:
No upper bound
on the frequency of communication and process failures
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

17. Motivation Background Adaptive Clock Sync. Evaluation Summary 10 of 25 slides
Failure Model
Messages are supposed to be delivered within δ
might be delayed
delivered out of order
get dropped
Arbitrary hardware clock failures: |ρp (t)| > ρmax
converted to stop failures [FC99a]
Processes’ value (byzantine) failures
handled by Software Encoded Processing [WF07]
Specifically:
No upper bound
on the frequency of communication and process failures
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

18. Motivation Background Adaptive Clock Sync. Evaluation Summary 11 of 25 slides
Problem Statement
∀p,q∈N : Sp (t) = Sq (t) (1)
However, achieving 1 is not possible
Minimize & bound difference between software clocks:
∀p,q∈N:p,q ¬crashed(t) ∀t : |Sp (t) − Sq (t)| ≤ Ep (t) + Eq (t)
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

19. Motivation Background Adaptive Clock Sync. Evaluation Summary 11 of 25 slides
Problem Statement
∀p,q∈N : Sp (t) = Sq (t) (1)
However, achieving 1 is not possible
Minimize & bound difference between software clocks:
∀p,q∈N:p,q ¬crashed(t) ∀t : |Sp (t) − Sq (t)| ≤ Ep (t) + Eq (t)
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

20. Motivation Background Adaptive Clock Sync. Evaluation Summary 11 of 25 slides
Problem Statement
∀p,q∈N : Sp (t) = Sq (t) (1)
However, achieving 1 is not possible
Minimize & bound difference between software clocks:
∀p,q∈N:p,q ¬crashed(t) ∀t : |Sp (t) − Sq (t)| ≤ Ep (t) + Eq (t)
q Abstract Clock p
Eq Ep
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

21. Motivation Background Adaptive Clock Sync. Evaluation Summary 12 of 25 slides
Ep – Maximum Clock Synchronization Error
Ep (T ) = |Cp (T , p) − cfn()| + Ep (T )
cfn() = mid Lip , Up
i
determines the Ap
←
− →
−
Ep (T ) = max cfn() − M p (T ), M p (T ) − cfn()
determines the midpoint estimation error
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

22. Motivation Background Adaptive Clock Sync. Evaluation Summary 12 of 25 slides
Ep – Maximum Clock Synchronization Error
Ep (T ) = |Cp (T , p) − cfn()| + Ep (T )
cfn() = mid Lip , Up
i
determines the Ap
←
− →
−
Ep (T ) = max cfn() − M p (T ), M p (T ) − cfn()
determines the midpoint estimation error
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

23. Motivation Background Adaptive Clock Sync. Evaluation Summary 13 of 25 slides
Convergence Functions: cfn() = mid Lip , Up
i
Ep (T ) = |Cp (T , p) − cfn()| + Ep (T )
A convergence function:
determines an abstract clock value which...
...should be reached at the end of the next round
Specifically:
cfnV 0 () — based on [WL88]
cfnV 1 () — based on [CF94]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

24. Motivation Background Adaptive Clock Sync. Evaluation Summary 13 of 25 slides
Convergence Functions: cfn() = mid Lip , Up
i
Ep (T ) = |Cp (T , p) − cfn()| + Ep (T )
A convergence function:
determines an abstract clock value which...
...should be reached at the end of the next round
Specifically:
cfnV 0 () — based on [WL88]
cfnV 1 () — based on [CF94]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

25. Motivation Background Adaptive Clock Sync. Evaluation Summary 14 of 25 slides
Convergence Functions: cfn() = mid Lip , Up
i
Ep (T ) = |Cp (T , p) − cfn()| + Ep (T )
Perfect World (no transmission delays, no drift):
p Cq(T,p) Cr(T,p)
cfn()
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

26. Motivation Background Adaptive Clock Sync. Evaluation Summary 14 of 25 slides
Convergence Functions: cfn() = mid Lip , Up
i
Ep (T ) = |Cp (T , p) − cfn()| + Ep (T )
Real Life (transmission delays, drift):
p Cq(T,p) Cr(T,p)
εq(T,p) εr(T,p)
cfn()
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

27. Motivation Background Adaptive Clock Sync. Evaluation Summary 15 of 25 slides
→
− ←
−
Midpoint Estimation Error: M p (T ), M p (T )
Ep (T ) = |Cp (T , p) − cfn()| + Ep (T )
Possible values a midpoint can take:
←
− ←
− ←
−
M p (T ) = mid L p (T ), U p (T )
→
− −
→ −
→
M p (T ) = mid L p (T ), U p (T )
cfn()
← →
Mp() Mp()
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

28. Motivation Background Adaptive Clock Sync. Evaluation Summary 16 of 25 slides
Extrapolation & Intersection
Re-use clock readings from previous round
Mask transient clock reading failures (extrapolation)
Improve readings with large error (intersection)
Extrapolation:
in round i use the reading from round i − 1
shift clock value by P, extend error by (k + 2)Pρmax
Intersection:
intersect readings from round i with extrapolation from i − 1
both correct → result also correct
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

29. Motivation Background Adaptive Clock Sync. Evaluation Summary 16 of 25 slides
Extrapolation & Intersection
Re-use clock readings from previous round
Mask transient clock reading failures (extrapolation)
Improve readings with large error (intersection)
Extrapolation:
in round i use the reading from round i − 1
shift clock value by P, extend error by (k + 2)Pρmax
Intersection:
intersect readings from round i with extrapolation from i − 1
both correct → result also correct
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

30. Motivation Background Adaptive Clock Sync. Evaluation Summary 16 of 25 slides
Extrapolation & Intersection
Re-use clock readings from previous round
Mask transient clock reading failures (extrapolation)
Improve readings with large error (intersection)
Extrapolation:
in round i use the reading from round i − 1
shift clock value by P, extend error by (k + 2)Pρmax
Intersection:
intersect readings from round i with extrapolation from i − 1
both correct → result also correct
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

31. Motivation Background Adaptive Clock Sync. Evaluation Summary 17 of 25 slides
Evaluation Environment
OMNeT++ discrete event based simulator
global observer view only possible in simulation
TDMA
10 Mbit, half-duplex, shared Ethernet
2 · 105 km/s signal propagation speed
4 hosts:
A 10 [m] B 10 [m] C 10 [m] D
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

32. Motivation Background Adaptive Clock Sync. Evaluation Summary 18 of 25 slides
Imprecision – cfnV 0 () & cfnV 1 ()
60
V0 - 0% V0 - 25% V1 - 1%
V0 - 1% V1 - 0% V1 - 25%
50
40
imprecision [µs]
30
20
10
0
25 30 35 40 45 50 55 60
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

33. Motivation Background Adaptive Clock Sync. Evaluation Summary 19 of 25 slides
Intersection Gain
0% -host A 1% - host D
0.6 0% - host D 25% - host A
1% - host A 25% - host D
0.5
mean improvement [%]
0.4
0.3
0.2
0.1
0
5 10 15 20 25 30
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

34. Motivation Background Adaptive Clock Sync. Evaluation Summary 19 of 25 slides
Intersection Gain
7
0% -host A 1% - host D
0% - host D 25% - host A
6 1% - host A 25% - host D
5
improvement [%]
4
3
2
1
0
8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 9
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

35. Motivation Background Adaptive Clock Sync. Evaluation Summary 20 of 25 slides
Extrapolation and Omissions
10000
max. sync. error
real sync. error
1000
infinity
100
error [µs]
10
1
0.1
0.01
0.001
21.6 21.8 22 22.2 22.4
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

36. Motivation Background Adaptive Clock Sync. Evaluation Summary 20 of 25 slides
Extrapolation and Omissions
10000
max. sync. error
real sync. error
1000
no infinity
100 (extrapolation)
error [µs]
10
1
0.1
0.01
0.001
21.6 21.8 22 22.2 22.4
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

37. Motivation Background Adaptive Clock Sync. Evaluation Summary 21 of 25 slides
Asymmetric Permanent Omissions
120
host A
host B
100 host C
host D
max. sync. error [µs]
80
60
40
20
0
4 5 6 7 8 9 10
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

38. Motivation Background Adaptive Clock Sync. Evaluation Summary 21 of 25 slides
Asymmetric Permanent Omissions
120
host A - max sync err
host B - max sync err
100 host C - max sync err
host D - max sync err
host A - real sync err
real/max sync err [µs]
80 host B - real sync err
host C - real sync err
host D - real sync err
60
40
20
0
4 5 6 7 8 9 10
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

39. Motivation Background Adaptive Clock Sync. Evaluation Summary 21 of 25 slides
Asymmetric Permanent Omissions
120
host C - max sync err
host D - max sync err
100 host C - real sync err
host D - real sync err
real/max sync err [µs]
80
60
40
20
0
4 5 6 7 8 9 10
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

40. Motivation Background Adaptive Clock Sync. Evaluation Summary 22 of 25 slides
Crash Failures
120
host A - max. sync. error
host A - real sync. error
100 host B - max. sync. error
host B - real sync. error
host D - max. sync. error
80 host D - real sync. error
error [µs]
60
40
20
0
34 36 38 40 42 44 46 48 50
real-time [s]
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

41. Motivation Background Adaptive Clock Sync. Evaluation Summary 23 of 25 slides
Summary
Need to build critical systems, however:
underlying components are not synchronous
need to use COTS to cut cost
need to use wireless to cut weight
We propose Adaptive Internal Clock Synchronization
copes with message delays and process failures
bounds deviation Ep (T ) from correct clocks in the system
propagates the deviation Ep (T ) to upper level applications
Applications use the Ep (T ) to adapt their behavior
no more synchronization layer crashes
the system is safer
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

42. Motivation Background Adaptive Clock Sync. Evaluation Summary 23 of 25 slides
Summary
Need to build critical systems, however:
underlying components are not synchronous
need to use COTS to cut cost
need to use wireless to cut weight
We propose Adaptive Internal Clock Synchronization
copes with message delays and process failures
bounds deviation Ep (T ) from correct clocks in the system
propagates the deviation Ep (T ) to upper level applications
Applications use the Ep (T ) to adapt their behavior
no more synchronization layer crashes
the system is safer
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

43. Motivation Background Adaptive Clock Sync. Evaluation Summary 23 of 25 slides
Summary
Need to build critical systems, however:
underlying components are not synchronous
need to use COTS to cut cost
need to use wireless to cut weight
We propose Adaptive Internal Clock Synchronization
copes with message delays and process failures
bounds deviation Ep (T ) from correct clocks in the system
propagates the deviation Ep (T ) to upper level applications
Applications use the Ep (T ) to adapt their behavior
no more synchronization layer crashes
the system is safer
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

44. Motivation Background Adaptive Clock Sync. Evaluation Summary 24 of 25 slides
Thank You!
http://wwwse.inf.tu-dresden.de/
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

45. Motivation Background Adaptive Clock Sync. Evaluation Summary 25 of 25 slides
References
Flaviu Cristian and Christof Fetzer.
Probabilistic internal clock synchronization.
In Proceedings of the Thirteenth Symposium on Reliable Distributed Systems (SRDS1994), pages 22–31,
October 1994.
Flaviu Cristian and Christof Fetzer.
The timed asynchronous distributed system model.
IEEE Trans. on Parallel and Distr. Systems, 10(6):642–657, June 1999.
Christof Fetzer and Flaviu Cristian.
Building fault-tolerant hardware clocks.
In Proceedings of the Seventh IFIP Int. Working Conf. on Dependable Computing for Critical Applications,
pages 59–78, San Jose, USA, Jan 1999.
Christof Fetzer and Flaviu Cristian.
A fail-aware datagram service.
In Iain Bate and Alan Burns, editors, IEE Proceedings - Softw. Eng., volume 146, pages 58–74. IEE, April
1999.
Christof Fetzer and Flaviu Cristian.
Fail-awareness: An approach to construct fail-safe systems.
Journal of Real-Time Systems, 24(2):203–238, March 2003.
Ute Wappler and Christof Fetzer.
Hardware failure virtualization via software encoded processing.
In 5th IEEE Int. Conf. on Industrial Inf. (INDIN 2007), volume 2, pages 977–982, June 2007.
Jennifer Lundelius Welch and Nancy Lynch.
A new fault-tolerant algorithm for clock synchronization.
Information and Computing, 77(1):1–36, 1988.
Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer