This document presents a talk on building soft state in publish/subscribe systems. It discusses challenges in distributed systems like unpredictable network latencies and failures. It proposes using an estimated link uncertainty to extend advertisement and subscription validity periods in order to cope with failures and latency variations. This allows the system state to be lost temporarily without disruption of system functionality, making the system survivable. The approach is evaluated on PlanetLab using different publish/subscribe implementations.

1. Overview System Model Soft State Evaluation Summary 1 of 23 slides
Soft State in Publish/Subscribe
DEBS 2009
Zbigniew Jerzak and Christof Fetzer
Systems Engineering Group, Dresden University of Technology
Thursday 9th July, 2009
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

2. Overview System Model Soft State Evaluation Summary 2 of 23 slides
What is Soft State?
System state which can be lost without permanent disruption of
system features [Cla88]
Soft state system =⇒ a survivable system:
can cope with transient failures
state is held at the edge of the system
node crash =⇒ loss of state owned by it
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

3. Overview System Model Soft State Evaluation Summary 2 of 23 slides
What is Soft State?
System state which can be lost without permanent disruption of
system features [Cla88]
Soft state system =⇒ a survivable system:
can cope with transient failures
state is held at the edge of the system
node crash =⇒ loss of state owned by it
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

4. Overview System Model Soft State Evaluation Summary 3 of 23 slides
In This Talk
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

5. Overview System Model Soft State Evaluation Summary 3 of 23 slides
In This Talk
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

6. Overview System Model Soft State Evaluation Summary 3 of 23 slides
In This Talk
How to build a soft pub/sub system using a practical system
model, i.e., one which:
makes realistic assumptions about the type and frequency of
failures
can be implemented in a real, distributed environment, i.e.,
PlanetLab
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

7. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

8. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

9. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

10. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

11. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

12. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

13. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

14. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

15. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

16. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

17. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

18. Overview System Model Soft State Evaluation Summary 4 of 23 slides
Approach Overview
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

19. Overview System Model Soft State Evaluation Summary 5 of 23 slides
Challenges
LAN: inf.tu-dresden.de
MAN: tu-dresden.de
WAN: tu-dresden->tu-berlin
1
0.75
CDF
0.5
0.25
10-1 100 101 102 103 104
rtt [ms]
Figure 1: Network latencies cannot be predicted, nor bounded!
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

20. Overview System Model Soft State Evaluation Summary 5 of 23 slides
Challenges
Lab PC
PlanetLab node
1
0.75
CDF
0.5
0.25
100 101 102 103 104 105
request processing time [µs]
Figure 1: Request processing time cannot be predicted, nor bounded!
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

21. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

22. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

23. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

24. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

25. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

26. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

27. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

28. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

29. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

30. Overview System Model Soft State Evaluation Summary 6 of 23 slides
Challenges – contd.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

31. Overview System Model Soft State Evaluation Summary 7 of 23 slides
Timed Asynchronous Distributed System Model [CF99]
All processes are timed (σmax )
. . . however, no guarantee that a request is indeed answered
within σmax
Processes expose crash-stop failure semantics
. . . and we know [WF07] how to enforce it
Messages might be arbitrarily delayed or dropped
. . . specifically, no upper bound on message transmission delay
Only local hardware clocks, with bounded drift rate (ρmax )
. . . and crash-stop failure semantics
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

32. Overview System Model Soft State Evaluation Summary 7 of 23 slides
Timed Asynchronous Distributed System Model [CF99]
All processes are timed (σmax )
. . . however, no guarantee that a request is indeed answered
within σmax
Processes expose crash-stop failure semantics
. . . and we know [WF07] how to enforce it
Messages might be arbitrarily delayed or dropped
. . . specifically, no upper bound on message transmission delay
Only local hardware clocks, with bounded drift rate (ρmax )
. . . and crash-stop failure semantics
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

33. Overview System Model Soft State Evaluation Summary 7 of 23 slides
Timed Asynchronous Distributed System Model [CF99]
All processes are timed (σmax )
. . . however, no guarantee that a request is indeed answered
within σmax
Processes expose crash-stop failure semantics
. . . and we know [WF07] how to enforce it
Messages might be arbitrarily delayed or dropped
. . . specifically, no upper bound on message transmission delay
Only local hardware clocks, with bounded drift rate (ρmax )
. . . and crash-stop failure semantics
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

34. Overview System Model Soft State Evaluation Summary 7 of 23 slides
Timed Asynchronous Distributed System Model [CF99]
All processes are timed (σmax )
. . . however, no guarantee that a request is indeed answered
within σmax
Processes expose crash-stop failure semantics
. . . and we know [WF07] how to enforce it
Messages might be arbitrarily delayed or dropped
. . . specifically, no upper bound on message transmission delay
Only local hardware clocks, with bounded drift rate (ρmax )
. . . and crash-stop failure semantics
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

35. Overview System Model Soft State Evaluation Summary 8 of 23 slides
Publish/Subscribe Model
Our approach works with any pub/sub model
topic-based
type-based
content-based
rule-based
It does not require bounds on message propagation delays
It does not require clock synchronization
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

36. Overview System Model Soft State Evaluation Summary 8 of 23 slides
Publish/Subscribe Model
Our approach works with any pub/sub model
It does not require bounds on message propagation delays
propagation delay = transmission delay + processing delay
It does not require clock synchronization
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

37. Overview System Model Soft State Evaluation Summary 8 of 23 slides
Publish/Subscribe Model
Our approach works with any pub/sub model
It does not require bounds on message propagation delays
It does not require clock synchronization
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

38. Overview System Model Soft State Evaluation Summary 9 of 23 slides
Approach
Use upper bound on message propagation delay
to estimate the link uncertainty
Link uncertainty:
a measure of the link latency variation
calculated using upper bound on propagation delay [JFF07]
Extend the advertisement/subscription validity
using calculated link uncertainty
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

39. Overview System Model Soft State Evaluation Summary 9 of 23 slides
Approach
Use upper bound on message propagation delay
to estimate the link uncertainty
Link uncertainty:
a measure of the link latency variation
calculated using upper bound on propagation delay [JFF07]
Extend the advertisement/subscription validity
using calculated link uncertainty
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

40. Overview System Model Soft State Evaluation Summary 9 of 23 slides
Approach
Use upper bound on message propagation delay
to estimate the link uncertainty
Link uncertainty:
a measure of the link latency variation
calculated using upper bound on propagation delay [JFF07]
Extend the advertisement/subscription validity
using calculated link uncertainty
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

41. Overview System Model Soft State Evaluation Summary 10 of 23 slides
Extending the Lease Time
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

42. Overview System Model Soft State Evaluation Summary 10 of 23 slides
Extending the Lease Time
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

43. Overview System Model Soft State Evaluation Summary 10 of 23 slides
Extending the Lease Time
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

44. Overview System Model Soft State Evaluation Summary 10 of 23 slides
Extending the Lease Time
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

45. Overview System Model Soft State Evaluation Summary 10 of 23 slides
Extending the Lease Time
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

46. Overview System Model Soft State Evaluation Summary 11 of 23 slides
Estimating (One-Way) Link Latencies
latency(sub) = (D − C )
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

47. Overview System Model Soft State Evaluation Summary 11 of 23 slides
Estimating (One-Way) Link Latencies
latency(sub) = (D − C )
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

48. Overview System Model Soft State Evaluation Summary 11 of 23 slides
Estimating (One-Way) Link Latencies
latency(sub) ≥ (D − A)
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

49. Overview System Model Soft State Evaluation Summary 11 of 23 slides
Estimating (One-Way) Link Latencies
latency(sub) ≥ (D − A)
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

50. Overview System Model Soft State Evaluation Summary 11 of 23 slides
Estimating (One-Way) Link Latencies
latency(sub) ≥ (D − A) − (C − B)
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

51. Overview System Model Soft State Evaluation Summary 11 of 23 slides
Estimating (One-Way) Link Latencies
latency(sub) ≥ (D − A) − (C − B) − δmin
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

52. Overview System Model Soft State Evaluation Summary 11 of 23 slides
Estimating (One-Way) Link Latencies
latency(sub) ≥ (D − A)(1 + ρmax ) − (C − B)(1 − ρmax ) − δmin
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

53. Overview System Model Soft State Evaluation Summary 12 of 23 slides
Evaluation Environment
F-A P/S
Zbigniew Jerzak, Robert Fach, and Christof Fetzer. Fail-aware
publish/subscribe. In NCA 07: Sixth IEEE International
Symposium on Network Computing and Applications, pages
113125, Cambridge, MA, USA, July 2007.
XSiena
Apache Mina
PlanetLab
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

54. Overview System Model Soft State Evaluation Summary 12 of 23 slides
Evaluation Environment
F-A P/S
XSiena
http://wwwse.inf.tu-dresden.de/xsiena/
Apache Mina
PlanetLab
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

55. Overview System Model Soft State Evaluation Summary 12 of 23 slides
Evaluation Environment
F-A P/S
XSiena
Apache Mina
http://mina.apache.org/
PlanetLab
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

56. Overview System Model Soft State Evaluation Summary 12 of 23 slides
Evaluation Environment
F-A P/S
XSiena
Apache Mina
PlanetLab
http://www.planet-lab.eu/
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

57. Overview System Model Soft State Evaluation Summary 13 of 23 slides
Local Setup
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

58. Overview System Model Soft State Evaluation Summary 14 of 23 slides
Local Setup – Upper Bound on Processing Delay
18
u-b pd (p1->s3)
16 r-t pd (p1->s3)
u-b pd (p1->b2)
14 r-t pd (p1->b2)
12
time [ms]
10
8
6
4
2
0
0 10 20 30 40 50
event number
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

59. Overview System Model Soft State Evaluation Summary 15 of 23 slides
European PlanetLab Setup
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

60. Overview System Model Soft State Evaluation Summary 16 of 23 slides
PlanetLab Europe – First Broker
1000 last 200 msgs (di.unito.it)
last 20s (di.unito.it)
u-b pd (di.unito.it)
time/uncertainty [ms]
100
10
300 320 340 360 380 400
refresh message number
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

61. Overview System Model Soft State Evaluation Summary 17 of 23 slides
PlanetLab Europe – Last Broker
last 200 msgs (cs.uit.no)
last 20s (cs.uit.no)
1000 u-b pd (cs.uit.no)
time/uncertainty [ms]
100
10
300 320 340 360 380 400
refresh message number
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

62. Overview System Model Soft State Evaluation Summary 18 of 23 slides
Intercontinental PlanetLab Setup
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

63. Overview System Model Soft State Evaluation Summary 19 of 23 slides
PlanetLab World – Advertisement Presence
uoit.ca - ext hiit.fi - ext
uoit.ca - noext hiit.fi - noext
yes
advertisement present
no
yes
no
yes
no
yes
no
26 28 30 32 34
time [s]
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

64. Overview System Model Soft State Evaluation Summary 20 of 23 slides
PlanetLab World – Total Time without Advertisement
uoit.ca - ext hiit.fi - ext
uoit.ca - noext hiit.fi - noext
total time without advertisement [%]
100
80
60
40
20
0
1 10 100
time [s]
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

65. Overview System Model Soft State Evaluation Summary 21 of 23 slides
Summary
Keep the state with the state owner
Limit the impact of failures
Use upper bound on propagation delay to ensure eventuall
consistency
For more information:
Jerzak, Z.; Fach, R. & Fetzer, C. Fail-Aware Publish/Subscribe
NCA ’07: Proceedings of the Sixth IEEE International
Symposium on Network Computing and Applications, IEEE
Computer Society, 2007, 113-125
Jerzak, Z.; Fach, R. & Fetzer, C. Adaptive Internal Clock
Synchronization SRDS ’08: 27th International Symposium on
Reliable Distributed Systems, IEEE Computer Society, 2008,
217-226
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

66. Overview System Model Soft State Evaluation Summary 22 of 23 slides
Thank You!
http://wwwse.inf.tu-dresden.de/xsiena/softstate
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

67. Overview System Model Soft State Evaluation Summary 23 of 23 slides
References
Flaviu Cristian and Christof Fetzer.
The timed asynchronous distributed system model.
IEEE Transactions on Parallel and Distributed Systems, 10(6):642–657, June
1999.
D. Clark.
The design philosophy of the DARPA internet protocols.
In SIGCOMM ’88: Symposium proceedings on Communications architectures
and protocols, pages 106–114, New York, NY, USA, 1988. ACM.
Zbigniew Jerzak, Robert Fach, and Christof Fetzer.
Fail-aware publish/subscribe.
In NCA ’07: Proceedings of the Sixth IEEE International Symposium on Network
Computing and Applications, pages 113–125, Cambridge, MA, USA, July 2007.
IEEE Computer Society.
Ute Wappler and Christof Fetzer.
Software encoded processing: Building dependable systems with commodity
hardware.
In Francesca Saglietti and Norbert Oster, editors, SAFECOMP ’07: 26th
International Conference on Computer Safety, Reliability, and Security, volume
4680 of Lecture Notes in Computer Science, pages 356–369, Nuremberg,
Germany, September 2007. Springer.
Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer