![Next Generation of Apache Hadoop MapReduce,[object Object],Arun C. Murthy - Hortonworks Founder and Architect,[object Object],@acmurthy (@hortonworks),[object Object],Formerly Architect, MapReduce @ Yahoo!,[object Object],8 years @ Yahoo!,[object Object],© Hortonworks Inc. 2011,[object Object],June 29, 2011,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-1-2048.jpg?cb=1668006294)
![Hello! I’m Arun…,[object Object],Architect & Lead, Apache Hadoop MapReduce Development Team at Hortonworks (formerly at Yahoo!),[object Object],Apache Hadoop Committer and Member of PMC,[object Object],Full-time contributor to Apache Hadoop since early 2006,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-2-2048.jpg?cb=1668006294)
![Hadoop MapReduce Today,[object Object],JobTracker,[object Object],Manages cluster resources and job scheduling,[object Object],TaskTracker,[object Object],Per-node agent,[object Object],Manage tasks,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-3-2048.jpg?cb=1668006294)
![Current Limitations,[object Object],Scalability,[object Object],Maximum Cluster size – 4,000 nodes,[object Object],Maximum concurrent tasks – 40,000,[object Object],Coarse synchronization in JobTracker,[object Object],Single point of failure ,[object Object],Failure kills all queued and running jobs,[object Object],Jobs need to be re-submitted by users,[object Object],Restart is very tricky due to complex state,[object Object],Hard partition of resources into map and reduce slots,[object Object],© Hortonworks Inc. 2011,[object Object],5,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-4-2048.jpg?cb=1668006294)
![Current Limitations,[object Object],Lacks support for alternate paradigms,[object Object],Iterative applications implemented using MapReduce are 10x slower. ,[object Object],Example: K-Means, PageRank,[object Object],Lack of wire-compatible protocols ,[object Object],Client and cluster must be of same version,[object Object],Applications and workflows cannot migrate to different clusters,[object Object],© Hortonworks Inc. 2011,[object Object],6,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-5-2048.jpg?cb=1668006294)
![Requirements,[object Object],Reliability,[object Object],Availability,[object Object],Scalability - Clusters of 6,000-10,000 machines,[object Object],Each machine with 16 cores, 48G/96G RAM, 24TB/36TB disks,[object Object],100,000+ concurrent tasks,[object Object],10,000 concurrent jobs,[object Object],Wire Compatibility,[object Object],Agility & Evolution – Ability for customers to control upgrades to the grid software stack.,[object Object],© Hortonworks Inc. 2011,[object Object],7,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-6-2048.jpg?cb=1668006294)
![Design Centre,[object Object],Split up the two major functions of JobTracker,[object Object],Cluster resource management,[object Object],Application life-cycle management,[object Object],MapReduce becomes user-land library,[object Object],© Hortonworks Inc. 2011,[object Object],8,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-7-2048.jpg?cb=1668006294)
![Architecture,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-8-2048.jpg?cb=1668006294)
![Architecture,[object Object],Resource Manager,[object Object],Global resource scheduler,[object Object],Hierarchical queues,[object Object],Node Manager,[object Object],Per-machine agent,[object Object],Manages the life-cycle of container,[object Object],Container resource monitoring,[object Object],Application Master,[object Object],Per-application,[object Object],Manages application scheduling and task execution,[object Object],E.g. MapReduce Application Master,[object Object],© Hortonworks Inc. 2011,[object Object],10,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-9-2048.jpg?cb=1668006294)
![Improvements vis-à-vis current MapReduce,[object Object],Scalability ,[object Object],Application life-cycle management is very expensive,[object Object],Partition resource management and application life-cycle management,[object Object],Application management is distributed,[object Object],Hardware trends - Currently run clusters of 4,000 machines,[object Object],6,000 2012 machines > 12,000 2009 machines,[object Object],<16+ cores, 48/96G, 24TB> v/s <8 cores, 16G, 4TB>,[object Object],© Hortonworks Inc. 2011,[object Object],11,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-10-2048.jpg?cb=1668006294)
![Improvments vis-à-vis current MapReduce,[object Object],Fault Tolerance and Availability ,[object Object],Resource Manager,[object Object],No single point of failure – state saved in ZooKeeper,[object Object],Application Masters are restarted automatically on RM restart,[object Object],Applications continue to progress with existing resources during restart, new resources aren’t allocated,[object Object],Application Master,[object Object],Optional failover via application-specific checkpoint,[object Object],MapReduce applications pick up where they left off via state saved in HDFS,[object Object],© Hortonworks Inc. 2011,[object Object],12,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-11-2048.jpg?cb=1668006294)
![Improvements vis-à-vis current MapReduce,[object Object],Wire Compatibility ,[object Object],Protocols are wire-compatible,[object Object],Old clients can talk to new servers,[object Object],Rolling upgrades,[object Object],© Hortonworks Inc. 2011,[object Object],13,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-12-2048.jpg?cb=1668006294)
![Improvements vis-à-vis current MapReduce,[object Object],Innovation and Agility,[object Object],MapReduce now becomes a user-land library,[object Object],Multiple versions of MapReduce can run in the same cluster (a la Apache Pig),[object Object],Faster deployment cycles for improvements,[object Object],Customers upgrade MapReduce versions on their schedule,[object Object],Users can customize MapReduce e.g. HOP without affecting everyone!,[object Object],© Hortonworks Inc. 2011,[object Object],14,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-13-2048.jpg?cb=1668006294)
![Improvements vis-à-vis current MapReduce,[object Object],Utilization,[object Object],Generic resource model ,[object Object],Memory,[object Object],CPU,[object Object],Disk b/w,[object Object],Network b/w,[object Object],Remove fixed partition of map and reduce slots,[object Object],© Hortonworks Inc. 2011,[object Object],15,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-14-2048.jpg?cb=1668006294)
![Improvements vis-à-vis current MapReduce,[object Object],Support for programming paradigms other than MapReduce,[object Object],MPI,[object Object],Master-Worker,[object Object],Machine Learning,[object Object],Iterative processing,[object Object],Enabled by allowing use of paradigm-specific Application Master,[object Object],Run all on the same Hadoop cluster,[object Object],© Hortonworks Inc. 2011,[object Object],16,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-15-2048.jpg?cb=1668006294)
![Summary,[object Object],MapReduce .Next takes Hadoop to the next level,[object Object],Scale-out even further,[object Object],High availability,[object Object],Cluster Utilization ,[object Object],Support for paradigms other than MapReduce,[object Object],© Hortonworks Inc. 2011,[object Object],17,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-16-2048.jpg?cb=1668006294)
![Status – June, 2011,[object Object],Feature complete,[object Object],Rigorous testing cycle underway,[object Object],Scale testing at ~500 nodes,[object Object],Sort/Scan/Shuffle benchmarks,[object Object],GridMixV3!,[object Object],Integration testing,[object Object],Pig integration complete!,[object Object],Coming in the next release of Apache Hadoop!,[object Object],Beta deployments of next release of Apache Hadoop at Yahoo! in Q4, 2011,[object Object],© Hortonworks Inc. 2011,[object Object],18,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-17-2048.jpg?cb=1668006294)
![Questions?,[object Object],http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/,[object Object],© Hortonworks Inc. 2011,[object Object],19,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-18-2048.jpg?cb=1668006294)
![Thank You.,[object Object],© Hortonworks Inc. 2011,[object Object]](https://image.slidesharecdn.com/apachehadoopmapreducenextgen-110630154552-phpapp01/75/nextgen-apache-hadoop-mapreduce-19-2048.jpg?cb=1668006294)
NextGen Apache Hadoop MapReduce
•49 likes•5,413 views
Report
Share
Arun C Murthy, Founder and Architect at Hortonworks Inc., talks about the upcoming Next Generation Apache Hadoop MapReduce framework at the Hadoop Summit, 2011.
Arun C Murthy, Founder and Architect at Hortonworks Inc., talks about the upcoming Next Generation Apache Hadoop MapReduce framework at the Hadoop Summit, 2011.
