• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Hortonworks Yarn Code Walk Through January 2014
 

Hortonworks Yarn Code Walk Through January 2014

on

  • 2,406 views

This slide deck accompanies the Webinar recording YARN Code Walk through on Jan. 22, 2014, on Hortonworks.com/webinars under Past Webinars, or

This slide deck accompanies the Webinar recording YARN Code Walk through on Jan. 22, 2014, on Hortonworks.com/webinars under Past Webinars, or
https://hortonworks.webex.com/hortonworks/lsr.php?AT=pb&SP=EC&rID=129468197&rKey=b645044305775657

Statistics

Views

Total Views
2,406
Views on SlideShare
2,403
Embed Views
3

Actions

Likes
9
Downloads
155
Comments
1

1 Embed 3

https://twitter.com 3

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Have more introduce about hdfs1 upgrade to yarn?
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • So while Hadoop 1.x had its uses this is really about turning Hadoop into the next generation platform. So what does that mean? A platform should be able to do multiple things, ergo more then just batch processing. Need Batch, Interactive, Online, and Streaming capabilities to really turn Hadoop into a Next Gen Platform.SCALES! Yahoo plans to move into a 10k node cluster
  • Now we have a concept of deploying applications into the hadoop clusterThese applications run in containers of set resources
  • RM takes place of JT and still has scheduling ques and such like the fair, capacity and hierarchical ques

Hortonworks Yarn Code Walk Through January 2014 Hortonworks Yarn Code Walk Through January 2014 Presentation Transcript

  • YARN Code Overview Ocular bleeding is no reason to stop programing! © Hortonworks Inc. 2013 Page 1
  • Quick Bio – Joseph Niemiec • Hadoop user for 2+ years • 1 of 5 Author’s for Apache Hadoop YARN • Originally used Hadoop for location based services (March 2014) – Destination Prediction – Traffic Analysis – Effects of weather at client locations on call center call types • Pending Patent in Automotive/Telematics domain • Defensive Paper on M2M Validation • Started on analytics to be better at an MMORPG © Hortonworks Inc. 2013
  • Agenda • What Is YARN • YARN Concepts & Architecture • Code and more Code • Q&A © Hortonworks Inc. 2013 Page 3
  • From Batch To Anything Single Use System Multi Purpose Platform Batch Apps Batch, Interactive, Online, Streaming, … HADOOP 1.0 HADOOP 2.0 MapReduce (data processing) MapReduce Others (data processing) YARN (cluster resource management & data processing) (cluster resource management) HDFS HDFS2 (redundant, reliable storage) (redundant, reliable storage) © Hortonworks Inc. 2013 Page 4
  • Concepts • Application –Application is a job submitted to the framework –Examples – Map Reduce Job – MoYa Cluster • Container –Basic unit of allocation –Fine-grained resource allocation across multiple resource types (memory, cpu, disk, network, gpu etc.) – container_0 = 2GB, 1CPU – container_1 = 1GB, 6 CPU –Replaces the fixed map/reduce slots © Hortonworks Inc. 2013 5
  • Architecture • Resource Manager –Global resource scheduler –Hierarchical queues • Node Manager –Per-machine agent –Manages the life-cycle of container –Container resource monitoring • Application Master –Per-application –Manages application scheduling and task execution –E.g. MapReduce Application Master © Hortonworks Inc. 2013 6
  • To the code! © Hortonworks Inc. 2013 Page 7
  • Q&A © Hortonworks Inc. 2013 Page 8
  • YARN - ApplicationMaster • ApplicationMaster – ApplicationSubmissionContext is the complete specification of the ApplicationMaster, provided by Client – ResourceManager responsible for allocating and launching ApplicationMaster container ApplicationSubmissionContext resourceRequest containerLaunchContext appName queue © Hortonworks Inc. 2013 Page 9
  • YARN – Resource Allocation & Usage • ContainerLaunchContext – The context provided by ApplicationMaster to NodeManager to launch the Container – Complete specification for a process – LocalResource used to specify container binary and dependencies – NodeManager responsible for downloading from shared namespace (typically HDFS) ContainerLaunchContext container commands environment localResources LocalResource uri type © Hortonworks Inc. 2013 Page 10
  • YARN – Resource Allocation & Usage • ResourceRequest priority 1 © Hortonworks Inc. 2013 <4gb, 1 core> numContainers 1 rack0 1 * <2gb, 1 core> resourceName host01 0 capability 1 * 1 Page 11
  • YARN – Resource Allocation & Usage • Container – The basic unit of allocation in YARN – The result of the ResourceRequest provided by ResourceManager to the ApplicationMaster – A specific amount of resources (cpu, memory etc.) on a specific machine Container containerId resourceName capability tokens © Hortonworks Inc. 2013 Page 12