Apache Hadoop MapReduce: What's Next

Apache Hadoop MapReduceWhat is next?Arun C. MurthyFounder & Architect@acmurthy (@hortonworks) Page 1

Hello! I’m Arun• Founder/Architect at Hortonworks Inc. – Lead, Map-Reduce – Formerly, Architect Hadoop MapReduce, Yahoo – Responsible for running Hadoop MR as a service for all of Yahoo (50k nodes footprint)• Apache Hadoop, ASF – VP, Apache Hadoop, ASF (Chair of Apache Hadoop PMC) – Long-term Committer/PMC member (full time >6 years) – Release Manager for hadoop-2 Page 2

Agenda• Yesterday: Hadoop MapReduce, circa 2011• Today: Hadoop YARN – Overview – State of the art•  Art of the possible – YARN Runtime – MapReduce Framework• Q&A Page 3

Hadoop MapReducecirca 2011 Page 4

Hadoop MapReduce Classic•  JobTracker –  Manages cluster resources and job scheduling•  TaskTracker –  Per-node agent –  Manage tasks

Current Limitations•  Utilization•  Scalability –  Maximum Cluster size – 4,000 nodes –  Maximum concurrent tasks – 40,000 –  Coarse synchronization in JobTracker•  Single point of failure –  Failure kills all queued and running jobs –  Jobs restarted on bounce 6

Current Limitations•  Hard partition of resources into map and reduce slots –  Low resource utilization•  Lacks support for alternate paradigms –  Iterative applications implemented using MapReduce are 10x slower –  Hacks for the likes of MPI/Graph Processing•  Lack of wire-compatible protocols –  Client and cluster must be of same version –  Applications and workflows cannot migrate to different clusters 7

Hadoop YARNOverview Page 8

Requirements•  Reliability•  Availability•  Utilization•  Wire Compatibility•  Agility & Evolution – Ability for customers to control upgrades to the grid software stack.•  Scalability - Clusters of 6,000-10,000 machines –  Each machine with 16 cores, 48G/96G RAM, 24TB/36TB disks –  100,000+ concurrent tasks –  10,000 concurrent jobs 9

Design Centre•  Split up the two major functions of JobTracker –  Cluster resource management –  Application life-cycle management•  MapReduce becomes user-land library 10

Architecture•  Application –  Application is a job submitted to the framework –  Example – Map Reduce Job•  Container –  Basic unit of allocation –  Example – container A = 2GB, 1CPU –  Replaces the fixed map/reduce slots 11

Architecture•  Resource Manager –  Global resource scheduler –  Hierarchical queues•  Node Manager –  Per-machine agent –  Manages the life-cycle of container –  Container resource monitoring•  Application Master –  Per-application –  Manages application scheduling and task execution –  E.g. MapReduce Application Master 12

Architecture Node Node Manager Manager Container App Mstr App Mstr Client Resource Node Node Resource Manager Manager Manager Manager Client Client App Mstr Container Container MapReduce Status Node Node MapReduce Status Manager Manager Job Submission Job Submission Node Status Node Status Resource Request Resource Request Container Container

How do I get it?•  Available in hadoop-2.0.0-alpha release 14

Performance• 2x+ across the board• MapReduce – Unlock lots of improvements from Terasort record (Owen/Arun, 2009) – Shuffle 30%+ – Merge improvements – Small Jobs – Uber AM – Re-use task slots (containers)More details: http://hortonworks.com/delivering-on-hadoop-next-benchmarking-performance/ Page 15

Resourceshadoop-2.0.0 (alpha release):http://hadoop.apache.org/common/releases.htmlRelease Documentation:http://hadoop.apache.org/common/docs/r2.0.0-alpha/ Page 16

Art of the possibleYARN RuntimeMapReduce Framework Page 17

Looking ahead• YARN – Runtime Improvements – Alternate programming models – Long(er) running services• MapReduce – Framework enhancements – Unpack! Page 18

YARN - Roadmap• Scheduler – Multi-dimensional resource scheduling (MAPREDUCE-4327) – Preemption (MAPREDUCE-3938) – Gang scheduling• Runtime improvements – Container Isolation (MAPREDUCE-4334) Page 19

YARN - Data Processing Applications• OpenMPI on Hadoop• Spark (UC Berkeley) – Shark is Hive-on-Spark• Real-time data processing – Storm (Twitter) – Apache S4•  Graph processing – Apache Giraph Page 20

YARN - Beyond Data Processing Apps• Apache Hbase – Deployment via YARN (HBASE-4329) – Co-processors via YARN (HBASE-4047)• Simple deployment for cluster services Page 21

MapReduce – Way Forward• MapReduce Framework Runtime – Monolithic software• MR Runtime? – Sort, Merge, Shuffle et al• Unpack into smaller building blocks! – Allow applications and Pig/Hive to ‘plug-n-play’ – MR framework, as we know today, becomes a particular configuration of the building blocks Page 22

MapReduce – Pluggable Sort• Pig & Hive benefit from hash-based aggregation – Several queries don’t need full-sort of map-outputs – Aggregation suffices – Allow for pluggable MapOutputBuffer in MapTask – Sort Avoidance - MAPREDUCE-4039 – External sort plugin – MAPREDUCE-2454 Page 23

MapReduce – Pluggable Shuffle• Push v/s Pull shuffle• Plug shuffle implementation (already in hadoop-2) – E.g. RDMA for shuffle – MAPREDUCE-4049• Collation tasks – Sailfish - Yahoo Research (includes auto-tuning of reduces) Page 24

MapReduce – More ideas• Allow for Map-Reduce-Reduce – Allow for reduce output to be sorted/shuffled – JOIN followed by ORDER BY – Really big deal for Pig/Hive Page 25

MapReduce – How do we get there?• Multiple, concurrent implementations of MapReduce – YARN is a really big deal… – Allows for safe experiments, much less risky! – Exposure surface is highly limited Page 26

Questions?Thank You.@acmurthy Page 27

http://www.scoop.it	1141
http://marilson.pbworks.com	78
http://eventifier.co	9
https://hwtest.uservoice.com	3
http://webcache.googleusercontent.com	2
http://translate.googleusercontent.com	1

Apache Hadoop MapReduce: What's Next

by Hadoop_Summit on Jun 19, 2012

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 1,234

Statistics

Apache Hadoop MapReduce: What’s Next Presentation Transcript