Distributed systems in practice, in theory (ScaleConf Colombia)

1. Aysylu Greenberg March 25, 2017 Distributed Systems in Practice, in Theory

2. * 2 Aysylu Greenberg

3. * 3 Aysylu Greenberg paperswelove.org

4. * 4 Aysylu Greenberg @aysylu22 paperswelove.org

5. How I got into reading papers as a practitioner in industry

6. Computer Science Research In Distributed Systems Industry

7. Operating systems research

8. Operating systems research

9. Operating systems research Concurrency

10. Operating systems research Concurrency Concurrency primitives: mutex & semaphore

11. Operating systems research Concurrency Concurrency primitives: mutex & semaphore Processes execute at different speeds

12. Time in distributed systems https://www.flickr.com/photos/national_archives_of_norway/6263353228 1970

13. Time in distributed systems 1970 https://www.flickr.com/photos/national_archives_of_norway/6263353228

14. https://www.flickr.com/photos/national_archives_of_norway/6263353228 Time in distributed systems Pipelining 1970

15. 1980 https://www.flickr.com/photos/findyoursearch/6848657291

16. Distributed consensus 1980 https://www.flickr.com/photos/findyoursearch/6848657291

19. Distributed consensus 1980 Paxos https://www.flickr.com/photos/findyoursearch/6848657291

20. Reconsider large systems

21. Reconsider large systems Shared infrastructure ...

22. CS Research is Timeless Inform decisions Mitigate technical risk

23. Today ● Staged Event-Driven Architecture

24. Today ● Staged Event-Driven Architecture ● Leases

25. Today ● Staged Event-Driven Architecture ● Leases ● Inaccurate Computations

27. Staged Event Driven Architecture & Deep Pipelines 2001

28. Hardware to Data Pipelines

29. Hardware to Data Pipelines https://en.wikipedia.org/wiki/Graphics_pipeline

30. Staged Event Driven Architecture

31. Staged Event Driven Architecture + -

32. Single-machine pipeline generalizes to distributed pipelines Staged Event Driven Architecture

33. Search Indexing Pipelines

40. Search Indexing Pipelines + -

42. Leases as Heart Beat in Distributed Systems 1989

43. Leases ● Distributed locking

44. Leases ● Distributed locking ● Lease term tradeoffs ○ short

45. Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long

46. Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election

47. Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election ○ Liveness detection

48. Build my sculpture Botero Build System

49. Build my sculpture Botero Build System OK

50. Build my sculpture Botero Build System OK Waiting for the results

51. Build my sculpture Botero Build System OK Waiting for the results Build is in progress

52. Build my sculpture Botero Build System OK Waiting for the results Build is in progress Waiting for the results

53. Build my sculpture Botero Build System OK Waiting for the results Build is in progress Waiting for the results Build is finished

54. Leases in Build System

60. Leases for heartbeat: How long should the lease term be?

62. Inaccurate Computations & Serving Search Results

63. From Accurate to "Good Enough"

64. [Trade off] Inaccuracy for Performance

65. [In production] Inaccuracy for Performance & Resilience

66. Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

67. [Designing with] Inaccuracy for Performance & Resilience

68. [Designing with] Inaccuracy for Performance & Resilience simplified implementation focus on observability applicable to some problem domains

69. [Designing with] Inaccuracy for Performance & Resilience fuzz testing generative testing simplified implementation fault injection testing focus on observability applicable to some problem domains

70. References ● F. Corbato, M. Daggett, R. Daley "An Experimental Time-Sharing System" ● E. Dijkstra "Cooperating Sequential Processes" ● L. Lamport "Time, Clocks, and the Ordering of Events in a Distributed System" ● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems" ● L. Lamport "The Part-Time Parliament"

71. References ● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well-Conditioned, Scalable Internet Services" ● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency" ● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data"

72. Robust & scalable pipelines

73. Robust & scalable pipelines Leases for sharing & heartbeat

74. Robust & scalable pipelines Leases for sharing & heartbeat Inaccuracy for resilience & performance

75. Robust & scalable pipelines Leases for sharing & heartbeat Inaccuracy for resilience & performance CS research is timeless: use it to mitigate risk

76. Aysylu Greenberg March 25, 2017 @aysylu22 Distributed Systems in Practice, in Theory