Uploaded byChicago Hadoop Users Group
Map Reduce v2 and YARN - CHUG - 20120604
The document introduces MapReduce 2 and YARN, which were designed to address limitations in MapReduce 1. YARN allows for decoupling of MapReduce processing from cluster resource management, enabling better resource utilization and support for additional applications beyond MapReduce. It separates resource management from job scheduling and processing, with a centralized ResourceManager and per-node NodeManagers. This improves scalability and high availability. The new architecture also allows for containers with variable resource limits rather than fixed slot types.
More Related Content
![[225]yarn 기반의 deep learning application cluster 구축 김제민](https://cdn.slidesharecdn.com/ss_thumbnails/225yarndeeplearningapplicationcluster-161025031031-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Map Reduce v2 and YARN - CHUG - 20120604
More from Chicago Hadoop Users Group
![[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-themodernstackbuildingwebaiapplicationswithserverless-251124030844-388cf04f-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-agenticaiarchitectureredefiningsystemcommunication-251124030838-e6c70cc2-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels](https://cdn.slidesharecdn.com/ss_thumbnails/md-exploringappleson-devicefoundationmodels-251124030840-d690542c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DevFest Strasbourg 2025] - NodeJs Can do that !!](https://cdn.slidesharecdn.com/ss_thumbnails/devfeststrasbourg2025-nodejscandothat-251127142731-da65b6fd-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-phpinaiagethelaravelway-251125012602-ef9d330e-thumbnail.jpg?width=640&height=640&fit=bounds)
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Map Reduce v2 and YARN - CHUG - 20120604
- 1. An Introduction to MapReduce 2 and YARN Tom White, Cloudera @tom_e_white June 4, 2012 Chicago HUG Tuesday, June 5, 2012
- 2. Road Trip Tuesday, June5, 2012
- 3. About me • Apache Hadoop Committer, PMC Member, Apache Member • Engineer at Cloudera working on core Hadoop • Founder of Apache Whirr • Author of “Hadoop: The Definitive Guide” • http://hadoopbook.com Tuesday, June 5, 2012
- 4. First, whatʼs MapReduce 1? Tuesday, June 5, 2012
- 5. Tuesday, June 5,2012
- 6. Whatʼs wrong with MR1? Tuesday, June 5, 2012
- 7. Motivation 1 • Scaling >4000 nodes • Fewer, larger clusters Tuesday, June 5, 2012
- 8. Motivation 2 • HA of Job Tracker • Large, complex state Tuesday, June 5, 2012
- 9. Motivation 3 • Poor resource utilization • Slots in MR1 are for either map or reduce Tuesday, June 5, 2012
- 10. Yet Another ResourceNegotiator Tuesday, June 5, 2012
- 11. Tuesday, June 5,2012
- 12. Tuesday, June 5,2012
- 13. Node Manager is a generalized Task Tracker • Task Tracker • fixed number of map or reduce slots • Node Manager • containers with variable resource limits Tuesday, June 5, 2012
- 14. Tuesday, June 5,2012
- 15. Tuesday, June 5,2012
- 16. MR is userspace YARN is kernel Tuesday, June 5, 2012
- 17. Bonus Apps • Distributed shell • MPI (MAPREDUCE-2911) • Master-worker (MAPREDUCE-3315) • Apache Giraph, Hama Tuesday, June 5, 2012
- 18. Tuesday, June 5,2012
- 19. Tuesday, June 5,2012
- 20. Old API ≠MR1 New API ≠ MR2 Tuesday, June 5, 2012
- 21. Old API New API o.a.h.mapred o.a.h.mapreduce MR1 ✓ ✓ MR2 ✓ ✓ Tuesday, June 5, 2012
- 22. Tuesday, June 5,2012
- 23. Try out MR2 • Apache Hadoop 2.0.0-alpha • hadoop.apache.org • CDH4 and Cloudera Manager • cloudera.com • Cloud - Apache Whirr Tuesday, June 5, 2012
- 24. MR1 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>1.0.3</version> </dependency> MR2 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.0.0-alpha</version> </dependency> Tuesday, June 5, 2012
- 25. TODO • Still alpha status • Performance tuning • Usability bug fixes • RM recovery • Security in MR2 not complete Tuesday, June 5, 2012
- 26.