This document discusses distributed systems concepts in both theory and practice. It covers operating systems research into concurrency and synchronization primitives. It then discusses distributed consensus algorithms from the 1970s and 1980s and the use of leases for distributed locking and leader election. It also discusses how staged event-driven architectures, leases, and allowing some inaccuracy can help build robust, scalable systems pipelines in industry. The document advocates using computer science research to inform design decisions and mitigate technical risks.

1. Aysylu Greenberg
March 25, 2017
Distributed Systems in Practice,
in Theory

2. * 2
Aysylu Greenberg

3. * 3
Aysylu Greenberg
paperswelove.org

4. * 4
Aysylu Greenberg
@aysylu22
paperswelove.org

5. How I got into reading
papers as a
practitioner in industry

6. Computer Science
Research
In
Distributed Systems
Industry

7. Operating systems research

8. Operating systems research

9. Operating systems research
Concurrency

10. Operating systems research
Concurrency
Concurrency primitives:
mutex & semaphore

11. Operating systems research
Concurrency
Concurrency primitives:
mutex & semaphore
Processes execute at
different speeds

12. Time in distributed systems
https://www.flickr.com/photos/national_archives_of_norway/6263353228
1970

13. Time in distributed systems
1970
https://www.flickr.com/photos/national_archives_of_norway/6263353228

14. https://www.flickr.com/photos/national_archives_of_norway/6263353228
Time in distributed systems
Pipelining
1970

15. 1980
https://www.flickr.com/photos/findyoursearch/6848657291

16. Distributed consensus
1980
https://www.flickr.com/photos/findyoursearch/6848657291

17. Distributed consensus
1980
https://www.flickr.com/photos/findyoursearch/6848657291

18. Distributed consensus
1980
https://www.flickr.com/photos/findyoursearch/6848657291

19. Distributed consensus
1980 Paxos
https://www.flickr.com/photos/findyoursearch/6848657291

20. Reconsider large systems

21. Reconsider large systems
Shared infrastructure
...

22. CS Research is Timeless
Inform decisions
Mitigate technical risk

23. Today
● Staged Event-Driven Architecture

24. Today
● Staged Event-Driven Architecture
● Leases

25. Today
● Staged Event-Driven Architecture
● Leases
● Inaccurate Computations

26. Today
● Staged Event-Driven Architecture
● Leases
● Inaccurate Computations

27. Staged Event
Driven
Architecture
&
Deep
Pipelines
2001

28. Hardware to Data Pipelines

29. Hardware to Data Pipelines
https://en.wikipedia.org/wiki/Graphics_pipeline

31. Staged Event Driven Architecture

32. Staged Event Driven Architecture
+ -

33. Single-machine pipeline
generalizes to distributed pipelines
Staged Event Driven Architecture

34. Search Indexing Pipelines

35. Search Indexing Pipelines

36. Search Indexing Pipelines

37. Search Indexing Pipelines

38. Search Indexing Pipelines

39. Search Indexing Pipelines

40. Search Indexing Pipelines

41. Search Indexing Pipelines
+ -

42. Today
● Staged Event-Driven Architecture
● Leases
● Inaccurate Computations

43. Leases
as Heart Beat in
Distributed
Systems
1989

45. Leases
● Distributed locking

46. Leases
● Distributed locking
● Lease term tradeoffs
○ short

47. Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long

48. Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long
● Use of leases in modern applications
○ Leader election

49. Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long
● Use of leases in modern applications
○ Leader election
○ Liveness detection

50. Build my sculpture
Botero
Build
System

51. Build my sculpture
Botero
Build
System
OK

52. Build my sculpture
Botero
Build
System
OK
Waiting for the results

53. Build my sculpture
Botero
Build
System
OK
Waiting for the results
Build is in progress

54. Build my sculpture
Botero
Build
System
OK
Waiting for the results
Build is in progress
Waiting for the results

55. Build my sculpture
Botero
Build
System
OK
Waiting for the results
Build is in progress
Waiting for the results
Build is finished

57. Leases in Build System

58. Leases in Build System

59. Leases in Build System

60. Leases in Build System

61. Leases in Build System

62. Leases in Build System

63. Leases for heartbeat:
How long should the lease term be?

64. Today
● Staged Event-Driven Architecture
● Leases
● Inaccurate Computations

65. Inaccurate Computations
&
Serving Search Results

66. From Accurate to "Good Enough"

67. [Trade off] Inaccuracy for Performance

71. [In production]
Inaccuracy for Performance & Resilience

72. Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

75. [Designing with]
Inaccuracy for Performance & Resilience

76. [Designing with]
Inaccuracy for Performance & Resilience
simplified implementation
focus on observability
applicable to
some problem
domains

77. [Designing with]
Inaccuracy for Performance & Resilience
fuzz testing
generative testing
simplified implementation
fault injection testing
focus on observability
applicable to
some problem
domains

78. References
● F. Corbato, M. Daggett, R. Daley "An Experimental
Time-Sharing System"
● E. Dijkstra "Cooperating Sequential Processes"
● L. Lamport "Time, Clocks, and the Ordering of Events in a
Distributed System"
● B. Oki, B. Liskov "Viewstamped Replication: A New Primary
Copy Method to Support Highly-Available Distributed Systems"
● L. Lamport "The Part-Time Parliament"

79. References
● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for
Well-Conditioned, Scalable Internet Services"
● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant
Mechanism for Distributed File Cache Consistency"
● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded
Errors and Bounded Response Times on Very Large Data"

80. Robust & scalable pipelines

81. Robust & scalable pipelines
Leases for sharing &
heartbeat

82. Robust & scalable pipelines
Leases for sharing &
heartbeat
Inaccuracy for resilience &
performance

83. Robust & scalable pipelines
Leases for sharing &
heartbeat
Inaccuracy for resilience &
performance
CS research is timeless:
use it to mitigate risk

84. Aysylu Greenberg
March 25, 2017
@aysylu22
Distributed Systems in Practice,
in Theory