An Introduction to MapReduce 2 and YARN

4,079 views

Published on

Published in: Technology, Education
1 Comment
7 Likes
Statistics
Notes
  • http://dbmanagement.info/Tutorials/MapReduce.htm
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
4,079
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
167
Comments
1
Likes
7
Embeds 0
No embeds

No notes for slide

An Introduction to MapReduce 2 and YARN

  1. 1. An Introduction toMapReduce 2 andYARNTom WhiteApril 25, 2012Seattle Hadoop / Scalability / NoSQL MeetupWednesday, April 25, 2012
  2. 2. First, whatʼsMapReduce 1?Wednesday, April 25, 2012
  3. 3. Wednesday, April 25, 2012
  4. 4. Whatʼs wrong withMR1?Wednesday, April 25, 2012
  5. 5. Motivation•Scaling >4000 nodes•HA of Job Tracker•Poor resource utilizationWednesday, April 25, 2012
  6. 6. Yet Another Resource NegotiatorWednesday, April 25, 2012
  7. 7. Wednesday, April 25, 2012
  8. 8. Wednesday, April 25, 2012
  9. 9. Node Manageris a generalized Task Tracker• Task Tracker• fixed number of map or reduceslots• Node Manager• containers with variable resourcelimitsWednesday, April 25, 2012
  10. 10. Wednesday, April 25, 2012
  11. 11. Wednesday, April 25, 2012
  12. 12. MR is user spaceYARN is kernelWednesday, April 25, 2012
  13. 13. Bonus Apps•Distributed shell•MPI (MAPREDUCE-2911)•Master-worker(MAPREDUCE-3315)•Apache Giraph, HamaWednesday, April 25, 2012
  14. 14. Wednesday, April 25, 2012
  15. 15. Wednesday, April 25, 2012
  16. 16. Old API ≠ MR1New API ≠ MR2Wednesday, April 25, 2012
  17. 17. Old APIo.a.h.mapredNew APIo.a.h.mapreduceMR1 ✓ ✓MR2 ✓ ✓Wednesday, April 25, 2012
  18. 18. Wednesday, April 25, 2012
  19. 19. Try out MR2•Apache Hadoop 0.23.1•CDH4 Beta 2Wednesday, April 25, 2012
  20. 20. <dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>1.0.2</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>0.23.1</version></dependency>MR1MR2Wednesday, April 25, 2012
  21. 21. TODO• Still alpha status• Performance tuning• Usability bug fixes• RM recovery• Security in MR2 not completeWednesday, April 25, 2012
  22. 22. Further ReadingWednesday, April 25, 2012
  23. 23. Thank You!Wednesday, April 25, 2012

×