YARN is a resource manager for Hadoop that allows for more efficient resource utilization and supports non-MapReduce applications. It separates resource management from job scheduling and execution. Key components include the ResourceManager, NodeManagers, and Containers. Ambari can be used to monitor YARN components and applications, configure queues and capacity scheduling, and view metrics and alerts. Future work includes supporting more applications and improving Capacity Scheduler configuration and health checks.

1. © Hortonworks Inc. 2013
YARN and AmbariYARN service management using Ambari
Srimanth Gunturi
September 25th, 2013
Page 1

2. © Hortonworks Inc. 2013
Agenda
• YARN Overview
• Installing
• Monitoring
• Configuration
• Capacity Scheduler
• MapReduce2
• Future
Page 2
Architecting the Future of Big Data

3. © Hortonworks Inc. 2013
YARN Overview
Page 3
Architecting the Future of Big Data
• Yet Another Resource Negotiator (YARN)
• Purpose - Negotiating resources (Memory, CPU, Disk, etc.) on a cluster
• What was wrong with original MapReduce?
– Cluster not open to non-MapReduce paradigms
– Inefficient resource utilization (Map slots, Reduce slots)
– Upgrade rigidity
• MapReduce (Hadoop 1.0) -> YARN + MapReduce 2 (Hadoop 2.0)

4. © Hortonworks Inc. 2013
YARN Overview - Applications
Page 4
Architecting the Future of Big Data
• MapReduce1 applications are fully compatible with MapReduce2
– Same JARs can be used
– Binary compatibility with org.apache.hadoop.mapred API
– Source compatibility with org.apache.hadoop.mapreduce API

5. © Hortonworks Inc. 2013
YARN Overview - Architecture
Page 5
Architecting the Future of Big Data
• ResourceManager
• NodeManagers
• Containers
• Applications (ApplicationMasters)

6. © Hortonworks Inc. 2013
Installing
Page 6
Architecting the Future of Big Data

7. © Hortonworks Inc. 2013
Monitoring
Page 7
Architecting the Future of Big Data

8. © Hortonworks Inc. 2013
Monitoring – NodeManager Summary
Page 8
Architecting the Future of Big Data
NodeManagers Status
• Active – In communication with RM
• Lost – Not communicating with RM
• Unhealthy – Flagged by custom health
check script identified by property
yarn.nodemanager.health-checker.script.path
• Rebooted – Automatically restarted due
to internal problems
• Decommissioned – RM ignoring
communications from host. Host placed
in yarn.resourcemanager.nodes.exclude-path
file.

9. © Hortonworks Inc. 2013
Monitoring – Container Summary
Page 9
Architecting the Future of Big Data
Containers
• Allocated – Containers which have
been created with requested resources.
• Pending – Containers, whose resources
will become available and are pending
creation.
• Reserved – Containers, whose
resources are not yet available.
Examples
10 GB Cluster
• Request three 5GB containers
• 2 allocated, 1 pending.
• Request three 4GB containers
• 2 allocated, 1 reserved (2GB)

10. © Hortonworks Inc. 2013
Monitoring – Applications Summary
Page 10
Architecting the Future of Big Data
Applications
• Submitted – Application requests made
to YARN.
• Running – Application with Masters
which have been created and are running.
• Pending – Application requests which
are pending creation.
• Completed – Applications which have
completed running. They could have
been successful, killed or failed.
• Killed – Applications which have been
terminated by user
• Failed – Applications which have failed
to run due to internal failures.

11. © Hortonworks Inc. 2013
Monitoring – Memory Summary
Page 11
Architecting the Future of Big Data
Cluster Memory
• Used – Memory resource currently
being used across the cluster
• Reserved – Memory resources that
are set aside for being allocated.
• Total – Memory resource available
across entire cluster

12. © Hortonworks Inc. 2013
Monitoring – Alerts
Page 12
Architecting the Future of Big Data
Service Alerts
- ResourceManager health
- % NodeManagers alive
Host Alerts
- NodeManager health
- NodeManager process check

13. © Hortonworks Inc. 2013
Monitoring – Graphs
Page 13
Architecting the Future of Big Data

14. © Hortonworks Inc. 2013
Configuration
Page 14
Architecting the Future of Big Data

15. © Hortonworks Inc. 2013
Configuration – Capacity Scheduler
Page 15
Architecting the Future of Big Data
Queues
Root
o A
o B
o C
o C1
o C2
o default
• Hierarchical Queues
• Capacity Guarantees
• Capacity (%)
• Maximum-am-resource-percent (%)
• Elasticity
• Maximum-capacity (%)
• Access Control

16. © Hortonworks Inc. 2013
MapReduce2
Page 16
Architecting the Future of Big Data
YARN-321: Generic application history service

17. © Hortonworks Inc. 2013
Future
Page 17
Architecting the Future of Big Data
• Support more YARN applications
• Improve per application-type information
• Improve Capacity Scheduler configuration
• Better health checks