Map Reduce v2 and YARN - CHUG - 20120604

3,610
-1

Published on

View the accompanying video on vimeo: https://vimeo.com/43474797

Published in: Technology
1 Comment
23 Likes
Statistics
Notes
  • http://dbmanagement.info/Tutorials/MapReduce.htm
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
3,610
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
1
Likes
23
Embeds 0
No embeds

No notes for slide

Map Reduce v2 and YARN - CHUG - 20120604

  1. 1. An Introduction to MapReduce 2 and YARN Tom White, Cloudera @tom_e_white June 4, 2012 Chicago HUGTuesday, June 5, 2012
  2. 2. Road TripTuesday, June 5, 2012
  3. 3. About me • Apache Hadoop Committer, PMC Member, Apache Member • Engineer at Cloudera working on core Hadoop • Founder of Apache Whirr • Author of “Hadoop: The Definitive Guide” • http://hadoopbook.comTuesday, June 5, 2012
  4. 4. First, whatʼs MapReduce 1?Tuesday, June 5, 2012
  5. 5. Tuesday, June 5, 2012
  6. 6. Whatʼs wrong with MR1?Tuesday, June 5, 2012
  7. 7. Motivation 1 • Scaling >4000 nodes • Fewer, larger clustersTuesday, June 5, 2012
  8. 8. Motivation 2 • HA of Job Tracker • Large, complex stateTuesday, June 5, 2012
  9. 9. Motivation 3 • Poor resource utilization • Slots in MR1 are for either map or reduceTuesday, June 5, 2012
  10. 10. Yet Another Resource NegotiatorTuesday, June 5, 2012
  11. 11. Tuesday, June 5, 2012
  12. 12. Tuesday, June 5, 2012
  13. 13. Node Manager is a generalized Task Tracker • Task Tracker • fixed number of map or reduce slots • Node Manager • containers with variable resource limitsTuesday, June 5, 2012
  14. 14. Tuesday, June 5, 2012
  15. 15. Tuesday, June 5, 2012
  16. 16. MR is user space YARN is kernelTuesday, June 5, 2012
  17. 17. Bonus Apps • Distributed shell • MPI (MAPREDUCE-2911) • Master-worker (MAPREDUCE-3315) • Apache Giraph, HamaTuesday, June 5, 2012
  18. 18. Tuesday, June 5, 2012
  19. 19. Tuesday, June 5, 2012
  20. 20. Old API ≠ MR1 New API ≠ MR2Tuesday, June 5, 2012
  21. 21. Old API New API o.a.h.mapred o.a.h.mapreduce MR1 ✓ ✓ MR2 ✓ ✓Tuesday, June 5, 2012
  22. 22. Tuesday, June 5, 2012
  23. 23. Try out MR2 • Apache Hadoop 2.0.0-alpha • hadoop.apache.org • CDH4 and Cloudera Manager • cloudera.com • Cloud - Apache WhirrTuesday, June 5, 2012
  24. 24. MR1 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>1.0.3</version> </dependency> MR2 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.0.0-alpha</version> </dependency>Tuesday, June 5, 2012
  25. 25. TODO • Still alpha status • Performance tuning • Usability bug fixes • RM recovery • Security in MR2 not completeTuesday, June 5, 2012
  26. 26. Questions?Tuesday, June 5, 2012

×