Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Commandments of scale

125 views

Published on

Going through the ten commandments of scale.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Commandments of scale

  1. 1. Baltimore,MD
  2. 2. Thank you! Thank you! Organizers: Jason Dixon Katherine Jeschke Brand/Design & A/V Miguel Montanez Jess Brown Photography Clinton Wolfe Ciprian Tutu Staff DeAnna Gibson Pat Norton Sherry Schlossnagle
  3. 3. A Brief History of Computing 1877 switchboard... lessons? boys are childish jerks, hire women.
  4. 4. A Brief History of Computing 1877 switchboard... lessons? boys are childish jerks, hire women.
  5. 5. sundial likely of ancient greek origin digital clock invented in 1972 by the watchmaking company called Hamilton MD5 hashing by Ronald Rivest Amazon dynamo (consistent hashing of data across distributed nodes) amongst others: Werner Vogels
  6. 6. sundial likely of ancient greek origin digital clock invented in 1972 by the watchmaking company called Hamilton MD5 hashing by Ronald Rivest Amazon dynamo (consistent hashing of data across distributed nodes) amongst others: Werner Vogels
  7. 7. sundial likely of ancient greek origin digital clock invented in 1972 by the watchmaking company called Hamilton MD5 hashing by Ronald Rivest Amazon dynamo (consistent hashing of data across distributed nodes) amongst others: Werner Vogels
  8. 8. /* Process each 16-word block. */ For i = 0 to N/16-1 do /* Copy block i into X. */ For j = 0 to 15 do Set X[j] to M[i*16+j]. end /* of loop on j */ /* Save A as AA, B as BB, C as CC, and D as DD. */ AA = A BB = B CC = C DD = D /* Round 1. */ /* Let [abcd k s i] denote the operation a = b + ((a + F(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 0 7 1] [DABC 1 12 2] [CDAB 2 17 3] [BCDA 3 22 4] [ABCD 4 7 5] [DABC 5 12 6] [CDAB 6 17 7] [BCDA 7 22 8] [ABCD 8 7 9] [DABC 9 12 10] [CDAB 10 17 11] [BCDA 11 22 12] [ABCD 12 7 13] [DABC 13 12 14] [CDAB 14 17 15] [BCDA 15 22 16] /* Round 2. */ /* Let [abcd k s i] denote the operation a = b + ((a + G(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 1 5 17] [DABC 6 9 18] [CDAB 11 14 19] [BCDA 0 20 20] [ABCD 5 5 21] [DABC 10 9 22] [CDAB 15 14 23] [BCDA 4 20 24] [ABCD 9 5 25] [DABC 14 9 26] [CDAB 3 14 27] [BCDA 8 20 28] [ABCD 13 5 29] [DABC 2 9 30] [CDAB 7 14 31] [BCDA 12 20 32] /* Round 3. */ /* Let [abcd k s t] denote the operation a = b + ((a + H(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 5 4 33] [DABC 8 11 34] [CDAB 11 16 35] [BCDA 14 23 36] [ABCD 1 4 37] [DABC 4 11 38] [CDAB 7 16 39] [BCDA 10 23 40] [ABCD 13 4 41] [DABC 0 11 42] [CDAB 3 16 43] [BCDA 6 23 44] [ABCD 9 4 45] [DABC 12 11 46] [CDAB 15 16 47] [BCDA 2 23 48] /* Round 4. */ /* Let [abcd k s t] denote the operation a = b + ((a + I(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 0 6 49] [DABC 7 10 50] [CDAB 14 15 51] [BCDA 5 21 52] [ABCD 12 6 53] [DABC 3 10 54] [CDAB 10 15 55] [BCDA 1 21 56] [ABCD 8 6 57] [DABC 15 10 58] [CDAB 6 15 59] [BCDA 13 21 60] [ABCD 4 6 61] [DABC 11 10 62] [CDAB 2 15 63] [BCDA 9 21 64] /* Then perform the following additions. (That is increment each of the four registers by the value it had before this block was started.) */ A = A + AA B = B + BB C = C + CC D = D + DD end /* of loop on i */ sundial likely of ancient greek origin digital clock invented in 1972 by the watchmaking company called Hamilton MD5 hashing by Ronald Rivest Amazon dynamo (consistent hashing of data across distributed nodes) amongst others: Werner Vogels
  9. 9. /* Process each 16-word block. */ For i = 0 to N/16-1 do /* Copy block i into X. */ For j = 0 to 15 do Set X[j] to M[i*16+j]. end /* of loop on j */ /* Save A as AA, B as BB, C as CC, and D as DD. */ AA = A BB = B CC = C DD = D /* Round 1. */ /* Let [abcd k s i] denote the operation a = b + ((a + F(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 0 7 1] [DABC 1 12 2] [CDAB 2 17 3] [BCDA 3 22 4] [ABCD 4 7 5] [DABC 5 12 6] [CDAB 6 17 7] [BCDA 7 22 8] [ABCD 8 7 9] [DABC 9 12 10] [CDAB 10 17 11] [BCDA 11 22 12] [ABCD 12 7 13] [DABC 13 12 14] [CDAB 14 17 15] [BCDA 15 22 16] /* Round 2. */ /* Let [abcd k s i] denote the operation a = b + ((a + G(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 1 5 17] [DABC 6 9 18] [CDAB 11 14 19] [BCDA 0 20 20] [ABCD 5 5 21] [DABC 10 9 22] [CDAB 15 14 23] [BCDA 4 20 24] [ABCD 9 5 25] [DABC 14 9 26] [CDAB 3 14 27] [BCDA 8 20 28] [ABCD 13 5 29] [DABC 2 9 30] [CDAB 7 14 31] [BCDA 12 20 32] /* Round 3. */ /* Let [abcd k s t] denote the operation a = b + ((a + H(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 5 4 33] [DABC 8 11 34] [CDAB 11 16 35] [BCDA 14 23 36] [ABCD 1 4 37] [DABC 4 11 38] [CDAB 7 16 39] [BCDA 10 23 40] [ABCD 13 4 41] [DABC 0 11 42] [CDAB 3 16 43] [BCDA 6 23 44] [ABCD 9 4 45] [DABC 12 11 46] [CDAB 15 16 47] [BCDA 2 23 48] /* Round 4. */ /* Let [abcd k s t] denote the operation a = b + ((a + I(b,c,d) + X[k] + T[i]) <<< s). */ /* Do the following 16 operations. */ [ABCD 0 6 49] [DABC 7 10 50] [CDAB 14 15 51] [BCDA 5 21 52] [ABCD 12 6 53] [DABC 3 10 54] [CDAB 10 15 55] [BCDA 1 21 56] [ABCD 8 6 57] [DABC 15 10 58] [CDAB 6 15 59] [BCDA 13 21 60] [ABCD 4 6 61] [DABC 11 10 62] [CDAB 2 15 63] [BCDA 9 21 64] /* Then perform the following additions. (That is increment each of the four registers by the value it had before this block was started.) */ A = A + AA B = B + BB C = C + CC D = D + DD end /* of loop on i */ sundial likely of ancient greek origin digital clock invented in 1972 by the watchmaking company called Hamilton MD5 hashing by Ronald Rivest Amazon dynamo (consistent hashing of data across distributed nodes) amongst others: Werner Vogels
  10. 10. 1664 Robert Hooke: sound transmission through a distended wire 1969 first known FIFO impl. Peter Alfke at Fairchild Semiconductor Eavesdropping was developed by “bad children” 1987 “Exploiting virtual synchrony in distributed systems” by Birman and Joseph
  11. 11. 1664 Robert Hooke: sound transmission through a distended wire 1969 first known FIFO impl. Peter Alfke at Fairchild Semiconductor Eavesdropping was developed by “bad children” 1987 “Exploiting virtual synchrony in distributed systems” by Birman and Joseph
  12. 12. 1664 Robert Hooke: sound transmission through a distended wire 1969 first known FIFO impl. Peter Alfke at Fairchild Semiconductor Eavesdropping was developed by “bad children” 1987 “Exploiting virtual synchrony in distributed systems” by Birman and Joseph
  13. 13. 1664 Robert Hooke: sound transmission through a distended wire 1969 first known FIFO impl. Peter Alfke at Fairchild Semiconductor Eavesdropping was developed by “bad children” 1987 “Exploiting virtual synchrony in distributed systems” by Birman and Joseph
  14. 14. 1664 Robert Hooke: sound transmission through a distended wire 1969 first known FIFO impl. Peter Alfke at Fairchild Semiconductor Eavesdropping was developed by “bad children” 1987 “Exploiting virtual synchrony in distributed systems” by Birman and Joseph
  15. 15. 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  16. 16. 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  17. 17. 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  18. 18. 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  19. 19. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  20. 20. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  21. 21. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  22. 22. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  23. 23. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  24. 24. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  25. 25. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  26. 26. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  27. 27. RFC1901 1854 Ignazio Porro patented this binocular design mailboxes Paris in the 18th century, Britian in the 19th century 1987 the International Organization for Standardization set for the OSI model 16th century, popularized in the Elizabethan era for hangings 1956 IBM introduced the harddrive Graphs? perhaps cavemen. Pressure measurement: Evangelista Torricelli in 1643 and te U-Tube by Christian Huygens in 1661 SMS concept was developed in the Franco-German GSM cooperation in 1984 Harold Black made the first negative feedback amplifier in the 1930s... inspiring at least me. the construction of monitoring systems
  28. 28. Keep on keepin’ on. Eventually we’ll get it (more) right We need to keep improving. Internet engineering is still very young.
  29. 29. Ten Commandments of Web Scale @postwait: the “Webscale! Wow!” campaign
  30. 30. I. Thou shalt not claim hugeness falsely 1TB is not a lot of data 100 million tuples is not a large table 1000 write transactions/second is not a lot A distributed system must contain 3 nodes A geographically distributed system must exceed one of: 1+ node in each of 3+ locations 3+ nodes in each of 2+ locations
  31. 31. I. Thou shalt not claim hugeness falsely 1TB is not a lot of data 100 million tuples is not a large table 1000 write transactions/second is not a lot A distributed system must contain 3 nodes A geographically distributed system must exceed one of: 1+ node in each of 3+ locations 3+ nodes in each of 2+ locations
  32. 32. II. Thou shalt not worship shininess shiny does not mean good good means good cope
  33. 33. II. Thou shalt not worship shininess shiny does not mean good good means good cope
  34. 34. III. Thou shalt always remain recoverable. data is precious map acceptable business liability to technical tolerances in systems never trust your systems test failures; force failures this is particularly true in large scale systems recoverability of petabytes is “non-trivial”
  35. 35. III. Thou shalt always remain recoverable. data is precious map acceptable business liability to technical tolerances in systems never trust your systems test failures; force failures this is particularly true in large scale systems recoverability of petabytes is “non-trivial”
  36. 36. IV. Respect that different problems may need different solutions a set P of problems is being solved by technology T. Pnew is added to P and T is found insufficient. a new technology S solves Pnew well so now P must be solved by S. This only proves: flawed logic Results in: poorly fitted solutions
  37. 37. IV. Respect that different problems may need different solutions a set P of problems is being solved by technology T. Pnew is added to P and T is found insufficient. a new technology S solves Pnew well so now P must be solved by S. This only proves: flawed logic Results in: poorly fitted solutions
  38. 38. V. Thou shalt not covet thy neighbor’s technology. unless one wish to covet their problems and failures to. do not learn by copying technologies and implementations. learn by adapting designs and sharing experiences.
  39. 39. V. Thou shalt not covet thy neighbor’s technology. unless one wish to covet their problems and failures to. do not learn by copying technologies and implementations. learn by adapting designs and sharing experiences.
  40. 40. VI. That which cannot be measured cannot be scaled. in a world where nothing is implemented, only designed, empirical measurement is the closest thing to reality without measurement you are blind measure everything forget nothing measurement is a prerequisite of success
  41. 41. VI. That which cannot be measured cannot be scaled. in a world where nothing is implemented, only designed, empirical measurement is the closest thing to reality without measurement you are blind measure everything forget nothing measurement is a prerequisite of success
  42. 42. VII. Operate even in the greatest of storms overall robustness is achieved through individual fragility the architecture should be highly componentized each component should perform well or not at all the overall system should operate despite removal of as many components as possible
  43. 43. VII. Operate even in the greatest of storms overall robustness is achieved through individual fragility the architecture should be highly componentized each component should perform well or not at all the overall system should operate despite removal of as many components as possible
  44. 44. VIII. Embrace the math of engineers 89 + 23 = 110 89 + 23 != 11.2 validate all your models, calculations and extrapolations with back-of-the- envelope math thou shalt have adequate quantity of envelopes.
  45. 45. VIII. Embrace the math of engineers 89 + 23 = 110 89 + 23 != 11.2 validate all your models, calculations and extrapolations with back-of-the-envelope math thou shalt have adequate quantity of envelopes.
  46. 46. IX. Thou shalt perform most favourably scaling out is required in some problem spaces poor performance is not required in any scaling out (or the ability to do so) is no excuse for poor performance
  47. 47. IX. Thou shalt perform most favourably scaling out is required in some problem spaces poor performance is not required in any scaling out (or the ability to do so) is no excuse for poor performance
  48. 48. X. Thou shalt be consistent... eventually eventual consistency only works when it is eventually consistent. building these systems is hard. thou shalt not use those yet to be proved correct.
  49. 49. X. Thou shalt be consistent... eventually eventual consistency only works when it is eventually consistent. building these systems is hard. thou shalt not use those yet to be proved correct.
  50. 50. XI. Thou shalt share with your brothers and sisters. (one for the road) we can learn by failures we can learn from the experiences of our peers I suggest a healthy mix
  51. 51. XI. Thou shalt share with your brothers and sisters. we can learn by failures we can learn from the experiences of our peers I suggest a healthy mix
  52. 52. Baltimore,MD

×