YARN is the resource manager platform in Hadoop clusters that allows for dynamic sharing of memory and CPU between frameworks like MapReduce, Spark, and others. It provides more predictable performance and better cluster utilization compared to earlier versions of MapReduce. YARN supports multi-tenancy and is backward compatible with MapReduce v1, allowing MapReduce and other frameworks to run on a single cluster.

1. Meet Hadoop Family: part 2
YARN

3. • What is it?
Resource manager platform in a Hadoop cluster, it allows dynamic memory
and CPU sharing between processing frameworks such as MapReduce,
Spark, and others
• Purpose
More predictable performance
Better cluster utilization
• Compared to MapReduce v1
MapReduce v1 starts to break on > 4000 nodes
YARN allows other frameworks to run on it also support multi tenancy
YARN is backward compatible with MapReduce V1

4. YARN Architecture

5. Scheduler types
• FIFO Scheduler
• Capacity scheduler
Fixed pools for resources
FIFO scheduling for each pools
• Fair scheduler
Weighted pools for resources
Fair sharing

6. Capacity Scheduler
• Capacity guaranted on each pool, with hard limits
and soft limits
• Hierarchical pool with a root pool
• Elasticity with preemptive option

7. Preemption Option
• T1: Time of App2’s submission
• T2: Time of App1 can finish
• T3: Time of App2 can finish

8. Fair Scheduler
• Each application assigned to a pool, a subpool is possible
• Excess capacity will be spreaded across all pools
• Pools with minimum resources defined received priority
during allocation
• Minimum resources are minimum amount of resources that
must be allocated to the pool before any fair allocation,
often used to satisfy SLA (service level agreement)
• Pools can be assigned a weight
• Preemption types, minimum and fair share

11. • Resource manager web interface,
port 8088
• Job history web interface,
port 19888

12. Log Aggregation
• Logs can be grouped by application
• Stored in HDFS (was not in Map Reduce v1)
• Gives better load balance when writting logs

13. • Show applications
yarn application -list
yarn application -list all
yarn application -status <application_id>
yarn application -list -appstates FINISHED
• Kill application
yarn application -kill <application_id>
• Show logs
yarn logs -applicationId <application_id>
• List YARN nodes
yarn node -list
Common Commands

14. Questions?
https://www.meetup.com/Jakarta-Hadoop-Big-Data/