Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Distributed systems in practice, in theory

1,854 views

Published on

Modern systems in production rely on decades of computer science research. Over time, new architectural patterns emerge that enable more resilient and robust systems. In this talk, we'll discuss some of these patterns from systems I've worked on at Google and the related work that provide insights into the motivations behind them.

Published in: Software
  • I’ve personally never heard of companies who can produce a paper for you until word got around among my college groupmates. My professor asked me to write a research paper based on a field I have no idea about. My research skills are also very poor. So, I thought I’d give it a try. I chose a writer who matched my writing style and fulfilled every requirement I proposed. I turned my paper in and I actually got a good grade. I highly recommend ⇒ www.WritePaper.info ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Distributed systems in practice, in theory

  1. 1. Confidential + ProprietaryConfidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March 8th, 2016
  2. 2. Confidential + Proprietary
  3. 3. Confidential + Proprietary
  4. 4. Confidential + Proprietary Aysylu Greenberg @aysylu22
  5. 5. Confidential + Proprietary Towards Distributed Build System
  6. 6. Confidential + Proprietary Towards Distributed Build System
  7. 7. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  8. 8. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  9. 9. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  10. 10. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  11. 11. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  12. 12. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  13. 13. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  14. 14. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  15. 15. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  16. 16. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  17. 17. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  18. 18. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  19. 19. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  20. 20. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  21. 21. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System
  22. 22. Confidential + Proprietary
  23. 23. Confidential + Proprietary Papers We Love SF
  24. 24. Confidential + Proprietary Aysylu Greenberg @aysylu22
  25. 25. Confidential + Proprietary meetup.com/Papers-We-Love-London
  26. 26. Confidential + Proprietary Today ● Staged Event-Driven Architecture
  27. 27. Confidential + Proprietary Today ● Staged Event-Driven Architecture ● Leases
  28. 28. Confidential + Proprietary Today ● Staged Event-Driven Architecture ● Leases ● Inaccurate Computations
  29. 29. Confidential + ProprietaryConfidential + Proprietary Computer Science Research In Distributed Systems Industry
  30. 30. Confidential + Proprietary Operating systems research
  31. 31. Confidential + Proprietary Operating systems research
  32. 32. Confidential + Proprietary Operating systems research Concurrency
  33. 33. Confidential + Proprietary Operating systems research Concurrency Concurrency primitives: mutex & semaphore
  34. 34. Confidential + Proprietary Operating systems research Concurrency Concurrency primitives: mutex & semaphore Processes execute at different speeds
  35. 35. Confidential + Proprietary Time in distributed systems
  36. 36. Confidential + Proprietary Time in distributed systems
  37. 37. Confidential + Proprietary Time in distributed systems Pipelining
  38. 38. Confidential + Proprietary Time in distributed systems Pipelining
  39. 39. Confidential + Proprietary Internet
  40. 40. Confidential + Proprietary Internet Distributed consensus
  41. 41. Confidential + Proprietary Internet Distributed consensus
  42. 42. Confidential + Proprietary Internet Distributed consensus
  43. 43. Confidential + Proprietary Internet Distributed consensus Paxos
  44. 44. Confidential + Proprietary Reconsider large systems
  45. 45. Confidential + Proprietary Reconsider large systems Platform as a service
  46. 46. Confidential + Proprietary CS Research is Timeless Inform decisions Mitigate technical risk
  47. 47. Confidential + ProprietaryConfidential + Proprietary Staged Event Driven Architecture & Deep Pipelines 2001
  48. 48. Confidential + Proprietary Hardware Pipelines
  49. 49. Confidential + Proprietary Data Pipelines https://en.wikipedia.org/wiki/Graphics_pipeline
  50. 50. Confidential + Proprietary Search Indexing Pipelines
  51. 51. Confidential + Proprietary Search Indexing Pipelines
  52. 52. Confidential + Proprietary Search Indexing Pipelines
  53. 53. Confidential + Proprietary Search Indexing Pipelines
  54. 54. Confidential + Proprietary Search Indexing Pipelines
  55. 55. Confidential + Proprietary Search Indexing Pipelines
  56. 56. Confidential + Proprietary Search Indexing Pipelines
  57. 57. Confidential + Proprietary Search Indexing Pipelines
  58. 58. Confidential + Proprietary Search Indexing Pipelines
  59. 59. Confidential + Proprietary Search Indexing Pipelines
  60. 60. Confidential + Proprietary Search Indexing Pipelines
  61. 61. Confidential + Proprietary
  62. 62. Confidential + Proprietary Staged Event Driven Architecture
  63. 63. Confidential + Proprietary Single-machine pipeline generalizes to distributed pipelines Staged Event Driven Architecture
  64. 64. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers
  65. 65. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers + -
  66. 66. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning + -
  67. 67. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing + -
  68. 68. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching + -
  69. 69. Confidential + Proprietary + - Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching ○ load shedding via queue
  70. 70. Confidential + Proprietary + - Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching ○ load shedding via queue ■ backpressure
  71. 71. Confidential + Proprietary + - Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching ○ load shedding via queue ■ backpressure ■ threshold, filter, reorder, aggregate
  72. 72. Confidential + ProprietaryConfidential + Proprietary Leases as Heart Beat in Distributed Systems 1989
  73. 73. Confidential + Proprietary
  74. 74. Confidential + Proprietary Leases ● Distributed locking
  75. 75. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short
  76. 76. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short ■ delay from client and server failures minimized ■ reduced false write-sharing
  77. 77. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long
  78. 78. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ■ more efficient for frequently accessed data ■ minimized lease extension overhead on server and client
  79. 79. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election TTL (in etcd)
  80. 80. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election TTL (in etcd) ○ Liveness detection
  81. 81. Confidential + Proprietary
  82. 82. Confidential + Proprietary Do the build, s'il vous plaît! Build System Dita Roque-Gourary
  83. 83. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Dita Roque-Gourary
  84. 84. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Dita Roque-Gourary
  85. 85. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Build is in progress Dita Roque-Gourary
  86. 86. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Build is in progress Dita Roque-Gourary Waiting for the results, merci.
  87. 87. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Build is in progress Build is finished/fini. Dita Roque-Gourary Waiting for the results, merci.
  88. 88. Confidential + Proprietary Leases in Build System
  89. 89. Confidential + ProprietaryConfidential + Proprietary Inaccurate Computations & Serving Search Results
  90. 90. Confidential + Proprietary From Accurate to "Good Enough"
  91. 91. Confidential + Proprietary
  92. 92. Confidential + Proprietary
  93. 93. Confidential + Proprietary
  94. 94. Confidential + Proprietary
  95. 95. Confidential + Proprietary
  96. 96. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition
  97. 97. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness
  98. 98. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness 3. Criticality Testing
  99. 99. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness 3. Criticality Testing 4. Distortion and timing models
  100. 100. Confidential + Proprietary Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010
  101. 101. Confidential + Proprietary Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010
  102. 102. Confidential + Proprietary
  103. 103. Confidential + Proprietary
  104. 104. Confidential + Proprietary Robust & scaleable pipelines
  105. 105. Confidential + Proprietary Robust & scaleable pipelines Leases for sharing & heartbeat
  106. 106. Confidential + Proprietary Robust & scaleable pipelines Leases for sharing & heartbeat Trade off inaccuracy for resilience & performance
  107. 107. Confidential + Proprietary Robust, scaleable pipelines Leases for sharing & heartbeat Trade off inaccurate for resilient CS research is timeless: use it to mitigate risk
  108. 108. Confidential + Proprietary Thanks Ines Sombra David Greenberg Alex Hutcheson Scott Zawalski Karan Parikh
  109. 109. Confidential + Proprietary References ● T. Wurthinger, C. Wimmer et al. "One VM to Rule Them All" ● M. Rinard "Probabilistic Accuracy Bounds for Fault-Tolerant Computations that Discard Tasks" ● F. Corbato, M. Daggett, R. Daley "An Experimental Time- Sharing System" ● E. Dijkstra "Cooperating Sequential Processes" ● L. Lamport "Time, Clocks, and the Ordering of Events in a Distributed System"
  110. 110. Confidential + Proprietary References ● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems" ● L. Lamport "The Part-Time Parliament" ● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well- Conditioned, Scalable Internet Services" ● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency" ● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data"
  111. 111. Confidential + Proprietary Should I read papers? YES

×