Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Distributed Systems In One Lesson

2,220 views

Published on

Normally simple tasks like running a program or storing and retrieving data become much more complicated when we start to do them on collections of computers, rather than single machines. Distributed systems has become a key architectural concern, and affects everything a program would normally do. Using a series of examples all set in a coffee shop, we’ll explore distributed storage, computation, timing, communication, consensus, and even some distributed programming paradigms.

Published in: Software
  • Don't forget another good way of simplifying your writing is using external resources (such as ⇒ www.HelpWriting.net ⇐ ). This will definitely make your life more easier
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Distributed Systems In One Lesson

  1. 1. Distributed Systems One Lesson in @tlberglund
  2. 2. What  are ?Distributed  Systems Any  system  too  large  to  fit   on  one  computer.
  3. 3. What  are • Storage   • Transac6ons   • Computa6on   • Coordina6on ?Distributed  Systems
  4. 4. Distributed  Storage
  5. 5. Distributed  Storage • Read  replica6on   • Sharding   • Consistent  hashing   • Distributed  filesystems
  6. 6. Distributed  Storage • MongoDB   • Cassandra   • HDFS
  7. 7. When  Life  is  Good
  8. 8. When  Life  is  Good
  9. 9. When  Life  is  Good Tim:Americano
  10. 10. When  Life  is  Good Americano Tim?
  11. 11. When  Life  is  Good
  12. 12. When  Life  is  Busier
  13. 13. When  Life  is  Busier Tim:Triple  Skinny
  14. 14. When  Life  is  Busier Tim:Triple  SkinnyTim:Triple  Skinny
  15. 15. When  Life  is  Busier Tim:Triple  Skinny Tim:Triple  Skinny
  16. 16. When  Life  is  Busier Tim?
  17. 17. ReplicaCon  Problems • Complexity   • Consistency
  18. 18. When  Life  is  REALLY  busy Aaron-­‐   Faye
  19. 19. When  Life  is  REALLY  busy Aaron-­‐   Faye Faye-­‐   Nancy
  20. 20. When  Life  is  REALLY  busy Aaron-­‐   Faye Faye-­‐   Nancy Nancy-­‐   Zed
  21. 21. Sharding  Problems • More  Complexity   • Broken  data  model   • Limited  data  access  paEerns
  22. 22. Consistent  Hashing 2000 4000 6000 8000 A000 C000 E000 0000
  23. 23. Consistent  Hashing 2000 4000 6000 8000 A000 C000 E000 0000 Tim:Americano
  24. 24. Consistent  Hashing 2000 4000 6000 8000 A000 C000 E000 0000 9F72:Americano
  25. 25. Consistent  Hashing 2000 4000 6000 8000 A000 C000 E000 0000 Tim?
  26. 26. Consistent  Hashing 2000 4000 6000 8000 A000 C000 E000 0000 9F72?
  27. 27. Consistent  Hashing 2000 4000 6000 8000 A000 C000 E000 0000 9F72:Americano
  28. 28. Consistent  Hashing 2000 4000 6000 8000 A000 C000 E000 0000
  29. 29. CAP  Theorem C PA
  30. 30. CAP  Theorem • Consistency   • Availability   • Par66on  Tolerance
  31. 31. CAP  Theorem • Shared  wri6ng  project   • Coffee  shop  closes   • Synchronizing  over  the  phone   • BaEery  dies   • Status  report!
  32. 32. CAP  Theorem C PA
  33. 33. Distributed  TransacCons
  34. 34. Distributed  TransacCons
  35. 35. ACID  TransacCons • Atomic   • Consistent   • Isolated   • Durable   • One  barista   • Mul6ple  baristas
  36. 36. Ordering  Coffee • Receive  order   • Process  payment   • Enqueue  order   • Make  coffee   • Deliver  drink
  37. 37. Ordering  Coffee • Receive  order   • Process  payment   • Enqueue  order   • Make  coffee   • Deliver  drink {Different Actors
  38. 38. Ordering  Coffee • Why  split  up  order  processing?   • What  can  fail?   • What  are  the  consequences  of  failure?   • How  do  we  repair  failure?
  39. 39. Ordering  Coffee • How  can  we  design  a  coffee  shop  with   atomic  transac6ons?   • How  does  that  limit  the  business?   • Why  give  up  atomicity?
  40. 40. Distributed  ComputaCon
  41. 41. Distributed  ComputaCon • ScaEer/Gather   • MapReduce   • Hadoop   • Spark
  42. 42. MapReduce • All  computa6on  in  two  func6ons:  Map  and  Reduce   • Keep  data  (mostly)  where  it  is   • Move  compute  to  data
  43. 43. MapReduce Mapper Reducer (k ,v) [(k ,v), (k ,v), (k ,v), …] Shuffle(k , [v, v, v, …]) [(k ,v), (k ,v), (k ,v), …]
  44. 44. MapReduce Once upon a midnight dreary, while I pondered, weak and weary, Over many a quaint and curious volume of forgotten lore, While I nodded, nearly napping, suddenly there came a tapping, As of some one gently rapping, rapping at my chamber door. "'Tis some visitor," I muttered, "tapping at my chamber door- Only this, and nothing more." Ah, distinctly I remember it was in the bleak December, And each separate dying ember wrought its ghost upon the floor. Eagerly I wished the morrow;- vainly I had sought to borrow poems/raven.txt:
  45. 45. MapReduce It shall clasp a sainted maiden whom the angels name Lenore- Clasp a rare and radiant maiden whom the angels name Lenore." Quoth the Raven, "Nevermore." "Be that word our sign in parting, bird or fiend," I shrieked, upstarting- "Get thee back into the tempest and the Night's Plutonian shore! Leave no black plume as a token of that lie thy soul hath spoken! Leave my loneliness unbroken!- quit the bust above my door! Take thy beak from out my heart, and take thy form from off my door!" Quoth the Raven, "Nevermore." And the Raven, never flitting, still is sitting, still is sitting On the pallid bust of Pallas just above my chamber door; And his eyes have all the seeming of a demon's that is dreaming, And the lamp-light o'er him streaming throws his shadow on the floor; And my soul from out that shadow that lies floating on the floor Shall be lifted- nevermore! poems/raven.txt:
  46. 46. MapReduce Once upon a midnightdrearywhile pondered suddenly there came a tapping gently rapping rapping tapping at my chamber door Map
  47. 47. MapReduce Once:1 upon:1 a:1 midnight:1dreary:1while:1 pondered:1 suddenly:1 there:1 came:1 a:1 tapping:1 gently:1 rapping:1 rapping:1 tapping:1 at:1 my:1 chamber:1 door:1 Map
  48. 48. MapReduce Once:1 upon:1a:1 midnight:1 dreary:1while:1 pondered:1suddenly:1 there:1came:1a:1 tapping:1gently:1 rapping:1 rapping:1 tapping:1 at:1 my:1chamber:1 door:1 Shuffle
  49. 49. MapReduce Once:[1] upon:[1] midnight:[1] dreary:[1]while:[1] pondered:[1]suddenly:[1] there:[1]came:[1]a:[1,1] gently:[1] rapping:[1,1] tapping:[1,1] at:[1] my:[1]chamber:[1] door:[1] Reduce
  50. 50. MapReduce Once:1 upon:1 midnight:1 dreary:1while:1 pondered:1suddenly:1 there:1came:1a:2 gently:1 rapping:2 tapping:2 at:1 my:1chamber:1 door:1 Reduce
  51. 51. Real-­‐World  MapReduce • Hadoop   • Cloudera,  Hortonworks,  MapR   • Nobody  write  map,  reduce  func6ons   • Hive  (SQL-­‐like  interface)   • Integra6on  with  BI  front-­‐ends
  52. 52. Hadoop • MapReduce  API   • Job  Tracker,  Task  Tracker   • Distributed  Filesystem  (HDFS)   • Enormous  ecosystem
  53. 53. Spark
  54. 54. Spark • ScaEer/gather  paradigm  (similar  to  MapReduce)   • More  general  data  model  (RDDs)   • More  general  programming  model  (transform/ac6on)   • Storage  agnos6c
  55. 55. Spark  Architecture 2000 4000 6000 8000 A000 C000 E000 0000 Spark Client Spark Context Driver
  56. 56. Spark Client Spark ContextJob Spark  Architecture
  57. 57. Spark Client Spark Context Job Cluster Manager Spark  Architecture
  58. 58. Spark Client Spark Context Cluster Manager JobTaskTaskTaskTask Spark  Architecture
  59. 59. Spark Client Spark Context Job Task Task Task Task Spark  Architecture Cluster Manager
  60. 60. Executor Task Task Cache Spark  Worker  Node
  61. 61. Executor Task Task Cache Executor Task Task CacheRDD RDD RDD RDD RDD RDD Spark  Worker  Node
  62. 62. What’s  an  RDD?
  63. 63. What’s  an  RDD? This is
  64. 64. What’s  an  RDD? • Bigger  than  a  computer   • Read  from  an  input  source   • Output  of  a  pure  func6on   • Immutable   • Typed   • Ordered   • Lazily  evaluated   • Par66oned   • Collec6on  of  things
  65. 65. Distributed,  how?
  66. 66. 5444 5676 0686 8389: List(xn-1, xn-2, xn-3) 4532 4569 7030 1191: List(xn-4, xn-5, xn-6) 5444 5607 6517 6027: List(xn-7, xn-8, xn-9) 4532 4577 0122 2189: Hash these Distributed,  how?
  67. 67. Distributed  CoordinaCon
  68. 68. Distributed  CoordinaCon • NTP   • Epidemic  Protocols   • Paxos
  69. 69. NTP • Distributed  6me  synchroniza6on   • A  network  of  6ered  clocks   • Synchroniza6on  algorithm   • O`en  accurate  to  ±10ms
  70. 70. Gossip 2000 4000 6000 8000 A000 C000 E000 0000
  71. 71. Paxos • Distributed  log  protocol   • Assumes  distributed,  unreliable  nodes   • Performs  leader  elec6on
  72. 72. Review • Storage   • Transac6ons   • Computa6on   • Coordina6on
  73. 73. Thank You! @tlberglund

×