Running Services on YARN

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Running Services on
YARN
Munich, April 2017
Varun Vasudev

About myself
⬢ Apache Hadoop contributor since 2014
⬢ Apache Hadoop committer and PMC member
⬢ Currently working for Hortonworks
⬢ vvasudev@apache.org

Introduction to Apache Hadoop YARN
⬢ Architectural center of big data workloads
⬢ Enterprise adoption
–Secure mode is popular
–Multi-tenant support
⬢ SLAs
–Tolerance for slow running jobs decreasing
–Consistent performance desired
⬢ Diverse workloads increasing
–LLAP on Slider

Introduction to Apache Hadoop YARN
YARN: Data Operating System
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
TezTez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
Others
ISV
Engines
HDFS
(Hadoop Distributed File System)
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider Slider
BATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark
YARN
The Architectural
Center of Hadoop
• Common data platform, many applications
• Support multi-tenant access & processing
• Batch, interactive & real-time use cases

Several important trends in age of Hadoop 3.0 +
YARN and Other Platform Services
Storage
Resource
Management Security
Service
Discovery Management
Monitoring
Alerts
IOT Assembly
Kafka Storm HBase Solr
Governance
MR Tez Spark …
Innovating
frameworks:
Flink,
DL(TensorFlow),
etc.
Various Environments
On Premise Private Cloud Public Cloud

Services workloads becoming more popular
⬢ Users using more and more long running services like LLAP, HiveServer, HBase, etc
⬢ Service workloads are gaining more importance
–Need a webserver to serve results from a MR job
–New YARN UI can be run in its own container
–ATSv2 would like to launch ATS reader containers as well
–Applications want to run their own shuffle service

Node 1
NodeManager128G, 16 vcores
Launch Applicaton 1 AMAM process
Launch AM process via
ContainerExecutor – DCE, LCE, WSCE.
Monitor/isolate memory and cpu
Application Lifecycle
ResourceManager
(active)
Request containers
Allocate containers
Container 1 process
Container 2 process
Launch containers on node using
DCE, LCE, WSCE. Monitor/isolate
memory and cpu
History Server(ATS – leveldb,
JHS - HDFS)
HDFS
Log aggregation

Application Lifecycle
⬢ Designed for batch jobs
–Jobs run for hours, days
–Jobs are using frameworks(like MR, Tez, Spark) which are aware of YARN
–Container failure is bad but frameworks have logic to handle it
•Local container state loss is handled
–Jobs are chained/pipelined using application ids
–Debugging is an offline event
⬢ Doesn’t carry over cleanly for services
–Services run for longer periods of time
–Services may or may not be aware of YARN
–Container loss is a bigger problem, can have really bad consequences
–Services would like to discover other services
–Debugging is an online event

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enabling Services on YARN

1
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enabling Services on YARN
⬢ AM to manage services
⬢ Service discovery
⬢ Container lifecycle
⬢ Scheduler changes
⬢ YARN UI
⬢ Application upgrades
⬢ Other issues
–Log collection
–Support for monitoring

1
1
AM to manage services
⬢ Any service/job on YARN requires an AM
–AM’s are hard to write
–Different services will re-implement the same functionalities
–AM has to keep up with changes in Apache Hadoop
⬢ Native YARN framework layer for services(YARN-5079)
–Provides an AM that ships as part of Apache Hadoop that can be used to manage services
–AM is from the Apache Slider project
–AM provides REST APIs to manage applications
–Has support for functionality such as port scheduling, flexing the number of containers
–Maintained by the Apache Hadoop developers so it’s kept up to date with the rest of YARN
–New YARN REST APIs to launch services

1
2
YARN REST API to launch services
{ "name": "vvasudev-druid-2017-03-16",
"resource": {
"cpus": 16,
"memory": "51200"
},
"components" :
[
{
"name": "vvasudev-druid",
"dependencies": [ ],
"artifact": {
"id": ”druid-image:0.1.0.0-25",
"type": "DOCKER"
},
"configuration": {
"properties": {
"env.CUSTOM_SERVICE_PROPERTIES": "true",
"env.ZK_HOSTS": ”zkhost1:2181,zkhost2:2181,zkhost3:2181"
}
}
}
],
"number_of_containers": 5,
"launch_command": "/opt/druid/start-druid.sh",
"queue" : ”yarn-services”
}

1
3
Service discovery
⬢ Long running services require a way to discover them
–Application ids are constant for the lifetime of the application
–Container ids are constant for the lifetime of the container but containers will come up and go
down
⬢ Add support for discovery of long running services using DNS and the Registry
Service(YARN-4757)
–DNS is well understood
–Registry service will have a record of the application to DNS name
–YARN has a DNS server but currently this is for testing and experimentation only
–YARN will need to add support for DNS updates to fit into existing DNS solutions

1
4
Service Discovery
NodeManager
NodeManager
NodeManager
ResourceManager
DNS Server Registry Service
ApplicationManager
Zookeeper
Zookeeper
Zookeeper
User

1
5
Container lifecycle
⬢ When the container exits, the NodeManager(NM) reclaims all the resources
immediately
–NM also cleans up any local state that the container maintained
⬢ AM may or may not be able to get a container back on the same node
–NM has to download any private resources again for the container leading to delays in restarts
⬢ Added support for first class container re-tries(YARN-4725)
–AM can specify retry policy when starting the container
–On process exit, the NM will not clean up any state or resources
–Instead it will attempt to retry the container
–AM can specify limits on the number of retries as well as the delay between retries

1
6
Container Lifecycle
NodeManager Container process
Disk 1 Disk 2
Disk 3
HDFS
Application
Container
Data

1
7
Scheduler improvements
⬢ In case of services, affinity and and anti-affinity become important
–Affinity and anti-affinity apply at a container and an application level – e.g. don’t schedule two
HBase region servers on the same node but schedule the Spark containers on the same nodes as
the region server
⬢ Support is being added for affinity and anti-affinity in the RM(YARN-5907)
–Slider AM already has some basic support for container affinity and anti-affinity via re-tries
–RM can do a better job of container placement if it has first class support
–AMs can specify affinity and anti-affinity policies to get the right placement they need

1
8
Scheduler improvements - Affinity and Anti-affinity
⬢ Anti-Affinity
–Some services don’t want their daemons run on the same host/rack for better fault recovering or
performance.
–For example, don’t run >1 HBase region server on the same fault zone.

1
9
Scheduler Improvements - Affinity and Anti-affinity
⬢ Affinity
–Some services want to run their daemons on the same host/rack, etc. for performance.
–For example, run Storm workers as close as possible for better data exchanging performance.
(SW = Storm Worker)

2
0
YARN UI(YARN-3368)

2
1
YARN UI - Services

2
2
Application upgrades
⬢ YARN has no support for container or application upgrades
–Container upgrade support support needs to be added in NM
–Application upgrade support has to be added in the RM
⬢ Support added for container upgrade and rollback(YARN-4726)
–Application upgrade support still to be carried out

2
3
Other issues
⬢ Log rotation
–Log rotation used to run on application completion
–Support has been added to fetch the logs for running containers
⬢ Support for container monitoring/health checks

2
4
In Conclusion
⬢ Services workloads becoming more and more popular on YARN
⬢ Fundamental pieces to add support for services are in place but few additional pieces
remain

Running Services on YARN

More Related Content

What's hot

Viewers also liked

Similar to Running Services on YARN

More from DataWorks Summit/Hadoop Summit

Recently uploaded

Running Services on YARN