Your SlideShare is downloading. ×
0
An Introduction toMapReduce 2 andYARNTom WhiteApril 25, 2012Seattle Hadoop / Scalability / NoSQL MeetupWednesday, April 25...
First, whatʼsMapReduce 1?Wednesday, April 25, 2012
Wednesday, April 25, 2012
Whatʼs wrong withMR1?Wednesday, April 25, 2012
Motivation•Scaling >4000 nodes•HA of Job Tracker•Poor resource utilizationWednesday, April 25, 2012
Yet Another Resource NegotiatorWednesday, April 25, 2012
Wednesday, April 25, 2012
Wednesday, April 25, 2012
Node Manageris a generalized Task Tracker• Task Tracker• fixed number of map or reduceslots• Node Manager• containers with ...
Wednesday, April 25, 2012
Wednesday, April 25, 2012
MR is user spaceYARN is kernelWednesday, April 25, 2012
Bonus Apps•Distributed shell•MPI (MAPREDUCE-2911)•Master-worker(MAPREDUCE-3315)•Apache Giraph, HamaWednesday, April 25, 2012
Wednesday, April 25, 2012
Wednesday, April 25, 2012
Old API ≠ MR1New API ≠ MR2Wednesday, April 25, 2012
Old APIo.a.h.mapredNew APIo.a.h.mapreduceMR1 ✓ ✓MR2 ✓ ✓Wednesday, April 25, 2012
Wednesday, April 25, 2012
Try out MR2•Apache Hadoop 0.23.1•CDH4 Beta 2Wednesday, April 25, 2012
<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>1.0.2</version></dependency...
TODO• Still alpha status• Performance tuning• Usability bug fixes• RM recovery• Security in MR2 not completeWednesday, Apri...
Further ReadingWednesday, April 25, 2012
Thank You!Wednesday, April 25, 2012
Upcoming SlideShare
Loading in...5
×

An Introduction to MapReduce 2 and YARN

3,025

Published on

Published in: Technology, Education
1 Comment
7 Likes
Statistics
Notes
  • http://dbmanagement.info/Tutorials/MapReduce.htm
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
3,025
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
131
Comments
1
Likes
7
Embeds 0
No embeds

No notes for slide

Transcript of "An Introduction to MapReduce 2 and YARN"

  1. 1. An Introduction toMapReduce 2 andYARNTom WhiteApril 25, 2012Seattle Hadoop / Scalability / NoSQL MeetupWednesday, April 25, 2012
  2. 2. First, whatʼsMapReduce 1?Wednesday, April 25, 2012
  3. 3. Wednesday, April 25, 2012
  4. 4. Whatʼs wrong withMR1?Wednesday, April 25, 2012
  5. 5. Motivation•Scaling >4000 nodes•HA of Job Tracker•Poor resource utilizationWednesday, April 25, 2012
  6. 6. Yet Another Resource NegotiatorWednesday, April 25, 2012
  7. 7. Wednesday, April 25, 2012
  8. 8. Wednesday, April 25, 2012
  9. 9. Node Manageris a generalized Task Tracker• Task Tracker• fixed number of map or reduceslots• Node Manager• containers with variable resourcelimitsWednesday, April 25, 2012
  10. 10. Wednesday, April 25, 2012
  11. 11. Wednesday, April 25, 2012
  12. 12. MR is user spaceYARN is kernelWednesday, April 25, 2012
  13. 13. Bonus Apps•Distributed shell•MPI (MAPREDUCE-2911)•Master-worker(MAPREDUCE-3315)•Apache Giraph, HamaWednesday, April 25, 2012
  14. 14. Wednesday, April 25, 2012
  15. 15. Wednesday, April 25, 2012
  16. 16. Old API ≠ MR1New API ≠ MR2Wednesday, April 25, 2012
  17. 17. Old APIo.a.h.mapredNew APIo.a.h.mapreduceMR1 ✓ ✓MR2 ✓ ✓Wednesday, April 25, 2012
  18. 18. Wednesday, April 25, 2012
  19. 19. Try out MR2•Apache Hadoop 0.23.1•CDH4 Beta 2Wednesday, April 25, 2012
  20. 20. <dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>1.0.2</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>0.23.1</version></dependency>MR1MR2Wednesday, April 25, 2012
  21. 21. TODO• Still alpha status• Performance tuning• Usability bug fixes• RM recovery• Security in MR2 not completeWednesday, April 25, 2012
  22. 22. Further ReadingWednesday, April 25, 2012
  23. 23. Thank You!Wednesday, April 25, 2012
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×