An Introduction to Apache Hadoop Yarn, what is it and why is it important ? What does it improve in Apache Hadoop ?

1. Apache Hadoop Yarn
● What is Yarn
● Problems with Hadoop
● What does Yarn Do ?
● Old Architecture
● New Architecture
● Yarn Example
● Additions

2. Hadoop Yarn – What is it ?
● Next Generation MapReduce MRv2
● Split Job Tracker into
– Resource Manager
– Scheduling / Monitoring
● Improves scaling
● Improves resource management
● Already used by Yahoo

3. Problems with Hadoop 1.0
● Problems with large scaling
– > 4000 nodes
– > 40k concurrent tasks
● Problems with resource utilization
● Slots only for Map or Reduce
● Single NameNode, single point of failure
● Clients and Cluster must be at same version

4. What does Yarn do ?
● Provides a cluster level resource manager
● Adds application level resource management
● Provides slots for jobs other than Map / Reduce
● Improves resource utilization

5. Old Architecture
● Cluster level Job Tracker, Task Tracker on data node

6. New Architecture

7. New Architecture
● Resource Manager
– Cluster level resource manager
– Long life
● Node Manager
– One per data server
– Monitors resources on node
● Application Master
– One per application
– Short life
– Manages task / scheduling

8. Yarn Example

9. Yarn Example
● 1) Client -> Resource Manager
– Submit App Master
● 2) Resource Manager -> Node Manager
– Start App Master
● 3) Application Master -> Resource Manager
– Request and release containers
● 4) Resource Manager -> Node Manager
– Start tasks in containers

10. Additions
● Consider Weave
– Simplifies the use of Yarn
– Reduced development effort
– Simplified API

11. Contact Us
● Feel free to contact us at
– www.semtech-solutions.co.nz
– info@semtech-solutions.co.nz
● We offer IT project consultancy
● We are happy to hear about your problems
● You can just pay for those hours that you need
● To solve your problems