Apache Hadoop YARN is the modern Distributed Operating System. It enables the Hadoop compute layer to be a common resource-management platform that can host a wide variety of applications. Multiple organizations are able to leverage YARN in building their applications on top of Hadoop without themselves repeatedly worrying about resource management, isolation, multi-tenancy issues etc. In this talk, we’ll first hit the ground with the current status of Apache Hadoop YARN – how it is faring today in deployments large and small. We will cover different types of YARN deployments, in different environments and scale. We'll then move on to the exciting present & future of YARN – features that are further strengthening YARN as the first-class resource-management platform for datacenters running enterprise Hadoop. We’ll discuss the current status as well as the future promise of features and initiatives like – 10x scheduler throughput improvements, docker containers support on YARN, support for long running services (alongside applications) natively without any changes, seamless application upgrades, fine-grained isolation for multi-tenancy using CGroups on disk & network resources, powerful scheduling features like application priorities, intra-queue preemption across applications and operational enhancements including insights through Timeline Service V2, a new web UI and better queue management. Speaker: Sunil Govindan, Senior Software Engineer, Hortonworks Rohith Sharma K S, Senior Software Engineer, Hortonworks

1. Rohith Sharma K S & Sunil G
Apache Hadoop PMC
Hortonworks Inc
Apache Hadoop YARN:
Present and Future 2017

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About.html
Rohith Sharma K S
 Apache Hadoop PMC and Committer
 Sr.Software Engineer @ Hortonworks
 7+ years for being “Hadooper”
Sunil G
 Apache Hadoop PMC and Committer
 Sr.Software Engineer @ Hortonworks
 Overall 10+ experience in enterprises software
development (5+ years for being “Hadooper”)

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
 Introduction
 Past
 Present & Future

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hadoop Compute Platform – Today and Tomorrow
 It’s all about data!
 Layers that enable applications and
higher order frameworks that interact
with data
 Multi-colored YARN
– Apps
– Long running services
 Admins and admin tools (Ambari) for
cluster management and monitoring
https://www.flickr.com/photos/happyskrappy/15699919424

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why?
• Different asks from different actors
• On isolation, capacity allocations, scheduling
Faster!
More! Best for my cluster
Throughput
Utilization
Elasticity
Service uptime
Security
ROIEverything! Right now!
SLA!

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hadoop Compute Platform – Today and Tomorrow
Platform Services
Storage
Resource
Management
Service
Discovery Management
Monitoring
Alerts
Holiday Assembly
HBase
HTTP
IOT Assembly
Kafk
a
Storm HBase Solr
Security
Governance
MR Tez
Spark
Hive / Pig
LLAP
Flink
REEF

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Past: A quick history

8. Page8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
A brief Timeline: Pre GA
• Sub-project of Apache Hadoop
• Releases tied to Hadoop releases
• Alphas and betas
– In production at several large sites for MapReduce already by that time
June-July 2010 August 2011 May 2012 August 2013

9. Page9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
A brief Timeline: GA Releases 1/3
2.2 2.3 2.4 2.5 2.6 2.7
15 October 2013 24 February 2014 07 April 2014 11 August 2014
• 1st GA
• MR binary
compatibility
• YARN API
cleanup
• Testing!
• 1st Post GA
• Bug fixes
• Alpha features
• RM Fail-over
• CS
Preemption
• Timeline
Service V1
• Writable
REST APIs
• Timeline
Service V1
security
• Rolling
Upgrades
• Services
• Node labels
18 November 2014
• Moving to
JDK 7+
• Pluggable
YARN
authentication
21 Apr 2015

10. Page10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
A brief Timeline: GA Releases 2/3
2.7.2 2.7.3 2.6.5 2.7.4 2.8.0 2.8.1
25 January 2016 25 August 2016 18 October 2016 04 August 2017
• Stabilization • Stabilization
• All community
parties
brought
together
• Stabilization • Stabilization • Application
Priority
• Reservations
• Node labels
improvements
22 March 2017
• Stabilization
• Next major
release
08 June 2017

11. Page11 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
A brief Timeline: GA Releases 3/3
3.0.0-alpha1 3.0.0-alpha2 3.0.0-alpha3 3.0.0-alpha4
03 September 2016 25 January 2017 26 May 2017 04 August 2017
• First alpha! • Second alpha! • Third alpha! • Fourth alpha!

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Present & Future

13. Page13 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Page 13
Containerization
Containers
GPUs /
FPGAs
More
powerful
scheduling
Much
faster
scheduling
Scale
SLAs
Usability
Service
workloads

14. Page14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Last few Hadoop releases
Apache Hadoop 2.8.x
Apache Hadoop 3.0.x

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hadoop 2.8.x

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Application priorities
• YARN-1963
• Within a leaf-queue
FIFO Policy App 1 App 2 App 3 App 4
FIFO Policy
With priorities
App 1
P8
App 2
P4
App 3
P3
App 4
P2

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Queue priorities
 Today
– Give to the least satisfied queue first
 With priorities
– Give to the highest priority queue first
root
A
20% Configured Capacity
But 5% of used Capacity
Usage = 5/20 = 25%
B
80% Configured Capacity
But 8% of used Capacity
Usage 8/80 = 10%

18. Page18 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Preemption within a queue
• Between apps of different priorities
• Between apps of different users
App 1 P3 App 2 P2 App 3 P1
App 1 U1 App 2 U2 App 3 U3
App 1 P3
App 2 P2
Candidates for
preemption
Queue Full
App 2 U2
App 1 U1
Candidates for
preemption
Queue Full

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Per-queue Policy-driven scheduling
Previously Now
Ingestion
FIFO
Adhoc
User-fairness
Adhoc
FIFO
Ingestion
FIFO
• Coarse policies
• One scheduling algorithm in the cluster
• Rigid
• Difficult to experiment
• Fine grained policies
• One scheduling algorithm per queue
• Flexible
• Very easy to experiment!
Batch
FIFO
Batch
FIFO

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reservations
• “Run my workload tomorrow at 6AM”
• Persistence of the plans with RM failover: YARN-2573
Timeline
Resources
6:00AM
Block #1
Timeline
Resources
6:00AM
Block #1
Block #2

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hadoop 3.x

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Looking at the Scale!
 Tons of sites with clusters made up of multiple thousands of nodes
– Yahoo!, Twitter, LinkedIn, Microsoft
 Largest clusters the last couple of years
– 6K-8K
 Roadmap: To 100K thousands and beyond
 Current progress: 40K nodes!

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Moving towards Global & Fast Scheduling
 Effort led by Wangda Tan from Hortonworks.
 Problems
– Current design of one-node-at-a-time allocation cycle can lead to suboptimal decisions.
– Several coarse grained locks
 Current effort made us where we improved to
– Look at several nodes at a time
– Fine grained locks
– Multiple allocator threads
– YARN scheduler can allocate 3k+ containers per second ≈ 10 mil allocations / hour!
– 10X throughput gains with enhancement added recently
– Much better placement decisions

24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Capacities in numbers vs percentages
root
(100%)
Sales
(30%)
Engineering
(65%)
Default
(5%)
Dev
(50%)
QE
(50%)
root
[memory=100Gi,vcores=100]
Sales
[memory=30Gi,vcores
=30]
Engineering
[memory=65Gi, vcores=60]
Default
[memory=1Gi,
vcores=1]
Dev
[memory=30Gi,
vcores=50]
QE
[memory=35Gi,
vcores=10]
 Current model using percentages has worked well for various use-cases
 Few challenges
– percentage based capacity is resilient to any changes in the cluster resource
– one percentage value for all resource-types
 YARN-5881

25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Resource profiles and custom resource types
 Past
– Supports only Memory and CPU
 Now
– A generalized vector
– Admins can create custom Resource Types!
– Ease of resource requesting model using
profiles
NodeManager
Memory
CPU
GPU
NodeManager
Memory
CPU
GPU
ResourceManager
Small
Medium
Large
Profile Memory CPU GPU
Small 2 GB 4 Cores 1 Cores
Medium 4 GB 8 Cores 1 Cores
Large 16 GB 16 Cores 4 Cores Application Master
Small (3 containers)
Medium (2 containers)

26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
GPUs on YARN!
 GPU can speed up compute-intensive applications 10x - 300x times
 Different levels of support
– Take me to a machine where GPUs are available with Partitions / Node Labels
– Take me to a machine where GPUs are available
• give me a full device only to me for the lifetime of my container
• give me multiple full devices only to me for the lifetime of my container
• give me full device(s) only to me for a portion of the lifetime of my container
• give me a slice of device(s) to me for a full / portion of the lifetime of my
container
 More dimensions:
– CPUs and memory and GPUs and on-GPU memory
– Topology of multiple GPUs

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
YARN Federation!
 Enables applications to scale to 100k of thousands of nodes
 Federation divides a large (10-100k nodes) cluster into smaller units called sub-clusters
 Federation negotiates with sub-clusters RM’s and provide resources to the application
 Applications can schedule tasks on any node

28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Better placement strategies
 Affinity
 Anti-affinity
 Node Constraints
HBase
Storm
YARN

29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Packaging
 Containers
– Lightweight mechanism for packaging and resource isolation
– Popularized and made accessible by Docker
– Can replace VMs in some cases
– Or more accurately, VMs got used in places where they didn’t
need to be
 Native integration ++ in YARN
– Support for “Container Runtimes” in LCE: YARN-3611
– Process runtime
– Docker runtime

30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Simplified APIs for service definitions
 Applications need simple APIs
 Need to be deployable “easily”
 Simple REST API layer fronting YARN
– YARN-4793 Simplified API layer for services and beyond
 Spawn services & Manage them

31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Services support
 Application & Services upgrades
– ”Do an upgrade of my Spark / HBase apps with minimal impact to end-users”
– YARN-4726
 Simplified discovery of services via DNS mechanisms: YARN-4757
– regionserver30.hbase-app-3.0.hadoop.yarn.site

32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Services Framework
 Platform is only as good as the tools
 A native YARN services framework
– YARN-4692
– [Umbrella] Native YARN framework layer for services and
beyond
 Assembly: Supporting a DAG of apps:
– SLIDER-875

33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
User experience
API based queue management
Decentralized
Improved logs
management
Live application logs

34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
User experience
New web UI

35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
User experience
New web UI

36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Timeline Service
 Application History
– “Where did my containers run?”
– “Why is my application slow?”
– “Is it really slow?”
– “Why is my application failing?”
– “What happened with my application?
Succeeded?”
 Cluster History
– Run analytics on historical apps!
– “User with most resource utilization”
– “Largest application run”
– “Why is my cluster slow?”
– “Why is my cluster down?”
– “What happened in my clusters?”
 Collect and use past data
– To schedule “my application” better
– To do better capacity planning

37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Timeline Service 2.0
• Next generation
– Today’s solution helped us understand the space
– Limited scalability and availability
• “Analyzing Hadoop Clusters is becoming a big-data problem”
– Don’t want to throw away the Hadoop application metadata
– Large scale
– Enable near real-time analysis: “Find me the user who is hammering the FileSystem with rouge applications. Now.”
• Timeline data stored in HBase and accessible to queries

38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
BoF’s: Apache Hadoop – YARN, HDFS
Sanjay Radia
Hortonworks
Sunil G
Hortonworks
https://dataworkssummit.com/sydney-2017/birds-of-a-feather/apache-hadoop-yarn-hdfs
Thursday September 21st Room C2.1 6.00PM
-- Related Session --
Rohith Sharma K S
Hortonworks

39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank you!