The document provides an overview of YARN (Yet Another Resource Negotiator), highlighting its architecture and operational concepts. It discusses the transition from Hadoop 1.0 to Hadoop 2.0, introducing key components such as the Resource Manager, Node Manager, and Application Master. The document emphasizes the container as the fundamental unit of resource allocation in YARN, supporting diverse application types.

1. YARN Code Overview
Ocular bleeding is no reason to stop programing!
© Hortonworks Inc. 2013
Page 1

2. Quick Bio – Joseph Niemiec
• Hadoop user for 2+ years
• 1 of 5 Author’s for Apache Hadoop YARN
• Originally used Hadoop for location based services
(March 2014)
– Destination Prediction
– Traffic Analysis
– Effects of weather at client locations on call center call types
• Pending Patent in Automotive/Telematics domain
• Defensive Paper on M2M Validation
• Started on analytics to be better at an MMORPG
© Hortonworks Inc. 2013

3. Agenda
• What Is YARN
• YARN Concepts & Architecture
• Code and more Code
• Q&A
© Hortonworks Inc. 2013
Page 3

4. From Batch To Anything
Single Use System
Multi Purpose Platform
Batch Apps
Batch, Interactive, Online, Streaming, …
HADOOP 1.0
HADOOP 2.0
MapReduce
(data processing)
MapReduce
Others
(data processing)
YARN
(cluster resource management
& data processing)
(cluster resource management)
HDFS
HDFS2
(redundant, reliable storage)
(redundant, reliable storage)
© Hortonworks Inc. 2013
Page 4

5. Concepts
• Application
–Application is a job submitted to the framework
–Examples
– Map Reduce Job
– MoYa Cluster
• Container
–Basic unit of allocation
–Fine-grained resource allocation across multiple resource
types (memory, cpu, disk, network, gpu etc.)
– container_0 = 2GB, 1CPU
– container_1 = 1GB, 6 CPU
–Replaces the fixed map/reduce slots
© Hortonworks Inc. 2013
5

6. Architecture
• Resource Manager
–Global resource scheduler
–Hierarchical queues
• Node Manager
–Per-machine agent
–Manages the life-cycle of container
–Container resource monitoring
• Application Master
–Per-application
–Manages application scheduling and task execution
–E.g. MapReduce Application Master
© Hortonworks Inc. 2013
6

7. To the code!
© Hortonworks Inc. 2013
Page 7

8. Q&A
© Hortonworks Inc. 2013
Page 8

9. YARN - ApplicationMaster
• ApplicationMaster
– ApplicationSubmissionContext is the complete specification of the
ApplicationMaster, provided by Client
– ResourceManager responsible for allocating and launching
ApplicationMaster container
ApplicationSubmissionContext
resourceRequest
containerLaunchContext
appName
queue
© Hortonworks Inc. 2013
Page 9

10. YARN – Resource Allocation & Usage
• ContainerLaunchContext
– The context provided by ApplicationMaster to NodeManager to
launch the Container
– Complete specification for a process
– LocalResource used to specify container binary and
dependencies
– NodeManager responsible for downloading from shared namespace
(typically HDFS)
ContainerLaunchContext
container
commands
environment
localResources
LocalResource
uri
type
© Hortonworks Inc. 2013
Page 10

11. YARN – Resource Allocation & Usage
• ResourceRequest
priority
1
© Hortonworks Inc. 2013
<4gb, 1 core>
numContainers
1
rack0
1
*
<2gb, 1 core>
resourceName
host01
0
capability
1
*
1
Page 11

12. YARN – Resource Allocation & Usage
• Container
– The basic unit of allocation in YARN
– The result of the ResourceRequest provided by ResourceManager
to the ApplicationMaster
– A specific amount of resources (cpu, memory etc.) on a specific
machine
Container
containerId
resourceName
capability
tokens
© Hortonworks Inc. 2013
Page 12