Download as PDF, PPTX
![MapReduce
Mapper
Reducer
(k ,v) [(k ,v), (k ,v), (k ,v), …]
Shuffle(k , [v, v, v, …])
[(k ,v), (k ,v), (k ,v), …]](https://image.slidesharecdn.com/distributedsystems-150224073002-conversion-gate01/75/Distributed-Systems-In-One-Lesson-43-2048.jpg)
![MapReduce
Once:[1]
upon:[1]
midnight:[1]
dreary:[1]while:[1]
pondered:[1]suddenly:[1] there:[1]came:[1]a:[1,1]
gently:[1]
rapping:[1,1]
tapping:[1,1]
at:[1]
my:[1]chamber:[1]
door:[1]
Reduce](https://image.slidesharecdn.com/distributedsystems-150224073002-conversion-gate01/75/Distributed-Systems-In-One-Lesson-49-2048.jpg)
Uploaded byTim Berglund
Distributed Systems In One Lesson
The document provides an overview of distributed systems concepts including distributed storage, transactions, computation, and coordination. It discusses various approaches to distributed storage including replication, sharding, consistent hashing, and distributed file systems. It also covers distributed transactions and the challenges of maintaining ACID properties in a distributed environment. The document introduces concepts of distributed computation frameworks like MapReduce and Spark and how they move computation towards data rather than moving large amounts of data. It discusses distributed coordination challenges and approaches like NTP for time synchronization, gossip protocols, and Paxos for reliable distributed logging.
In this document
Powered by AI
Overview of distributed systems including definition and key components such as storage, transactions, computation, and coordination.
Exploration of distributed storage techniques including read replication, sharding, and examples like MongoDB and HDFS.
Problems associated with replication and sharding, highlighting complexity and consistency issues therein.
Concept of consistent hashing illustrated through numerical examples, focusing on its relevance in distributed systems.Introduction to the CAP theorem emphasizing consistency, availability, and partition tolerance with real-world implications.Core concepts around distributed transactions using the ACID model, with practical scenarios like order processing in coffee shops.
Explanation of distributed computation including MapReduce paradigm, data handling techniques, and process flow.
Practical applications of MapReduce leveraging tools like Hadoop, including its ecosystem and user-friendly interfaces.
Introduction to Spark's architecture and its operational model, detailing concepts like RDDs and execution plans.
Discussion on methods for distributed coordination such as NTP, epidemic protocols, and the Paxos consensus algorithm.
Concise review of the core topics like storage, transactions, computation, and coordination within distributed systems.
More Related Content
Similar to Distributed Systems In One Lesson
Distributed Systems In One Lesson
- 1.
- 2. What are ?Distributed Systems Any system too large to fit on one computer.
- 3. What are • Storage • Transac6ons • Computa6on • Coordina6on ?Distributed Systems
- 4.
- 5. Distributed Storage • Read replica6on • Sharding • Consistent hashing • Distributed filesystems
- 6. Distributed Storage • MongoDB • Cassandra • HDFS
- 7. When Life is Good
- 8. When Life is Good
- 9. When Life is Good Tim:Americano
- 10. When Life is Good Americano Tim?
- 11. When Life is Good
- 12. When Life is Busier
- 13. When Life is Busier Tim:Triple Skinny
- 14. When Life is Busier Tim:Triple SkinnyTim:Triple Skinny
- 15. When Life is Busier Tim:Triple Skinny Tim:Triple Skinny
- 16. When Life is Busier Tim?
- 17. ReplicaCon Problems • Complexity • Consistency
- 18. When Life is REALLY busy Aaron-‐ Faye
- 19. When Life is REALLY busy Aaron-‐ Faye Faye-‐ Nancy
- 20. When Life is REALLY busy Aaron-‐ Faye Faye-‐ Nancy Nancy-‐ Zed
- 21. Sharding Problems • More Complexity • Broken data model • Limited data access paEerns
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. CAP Theorem • Consistency • Availability • Par66on Tolerance
- 31. CAP Theorem • Shared wri6ng project • Coffee shop closes • Synchronizing over the phone • BaEery dies • Status report!
- 32.
- 33.
- 34.
- 35. ACID TransacCons • Atomic • Consistent • Isolated • Durable • One barista • Mul6ple baristas
- 36. Ordering Coffee • Receive order • Process payment • Enqueue order • Make coffee • Deliver drink
- 37. Ordering Coffee • Receive order • Process payment • Enqueue order • Make coffee • Deliver drink {Different Actors
- 38. Ordering Coffee • Why split up order processing? • What can fail? • What are the consequences of failure? • How do we repair failure?
- 39. Ordering Coffee • How can we design a coffee shop with atomic transac6ons? • How does that limit the business? • Why give up atomicity?
- 40.
- 41. Distributed ComputaCon • ScaEer/Gather • MapReduce • Hadoop • Spark
- 42. MapReduce • All computa6on in two func6ons: Map and Reduce • Keep data (mostly) where it is • Move compute to data
- 43. MapReduce Mapper Reducer (k ,v) [(k,v), (k ,v), (k ,v), …] Shuffle(k , [v, v, v, …]) [(k ,v), (k ,v), (k ,v), …]
- 44. MapReduce Once upon amidnight dreary, while I pondered, weak and weary, Over many a quaint and curious volume of forgotten lore, While I nodded, nearly napping, suddenly there came a tapping, As of some one gently rapping, rapping at my chamber door. "'Tis some visitor," I muttered, "tapping at my chamber door- Only this, and nothing more." Ah, distinctly I remember it was in the bleak December, And each separate dying ember wrought its ghost upon the floor. Eagerly I wished the morrow;- vainly I had sought to borrow poems/raven.txt:
- 45. MapReduce It shall claspa sainted maiden whom the angels name Lenore- Clasp a rare and radiant maiden whom the angels name Lenore." Quoth the Raven, "Nevermore." "Be that word our sign in parting, bird or fiend," I shrieked, upstarting- "Get thee back into the tempest and the Night's Plutonian shore! Leave no black plume as a token of that lie thy soul hath spoken! Leave my loneliness unbroken!- quit the bust above my door! Take thy beak from out my heart, and take thy form from off my door!" Quoth the Raven, "Nevermore." And the Raven, never flitting, still is sitting, still is sitting On the pallid bust of Pallas just above my chamber door; And his eyes have all the seeming of a demon's that is dreaming, And the lamp-light o'er him streaming throws his shadow on the floor; And my soul from out that shadow that lies floating on the floor Shall be lifted- nevermore! poems/raven.txt:
- 46.
- 47.
- 48.
- 49.
- 50.
- 51. Real-‐World MapReduce • Hadoop • Cloudera, Hortonworks, MapR • Nobody write map, reduce func6ons • Hive (SQL-‐like interface) • Integra6on with BI front-‐ends
- 52. Hadoop • MapReduce API • Job Tracker, Task Tracker • Distributed Filesystem (HDFS) • Enormous ecosystem
- 53.
- 54. Spark • ScaEer/gather paradigm (similar to MapReduce) • More general data model (RDDs) • More general programming model (transform/ac6on) • Storage agnos6c
- 55.
- 56.
- 57.
- 58.
- 59.
- 60.
- 61. Executor Task Task Cache Executor Task Task CacheRDDRDD RDD RDD RDD RDD Spark Worker Node
- 62.
- 63.
- 64. What’s an RDD? •Bigger than a computer • Read from an input source • Output of a pure func6on • Immutable • Typed • Ordered • Lazily evaluated • Par66oned • Collec6on of things
- 65.
- 66. 5444 5676 06868389: List(xn-1, xn-2, xn-3) 4532 4569 7030 1191: List(xn-4, xn-5, xn-6) 5444 5607 6517 6027: List(xn-7, xn-8, xn-9) 4532 4577 0122 2189: Hash these Distributed, how?
- 67.
- 68. Distributed CoordinaCon • NTP • Epidemic Protocols • Paxos
- 69. NTP • Distributed 6me synchroniza6on • A network of 6ered clocks • Synchroniza6on algorithm • O`en accurate to ±10ms
- 70.
- 71. Paxos • Distributed log protocol • Assumes distributed, unreliable nodes • Performs leader elec6on
- 74. Review • Storage •Transac6ons • Computa6on • Coordina6on
- 75.