Lessons Learned Running a Container Cloud on Apache Hadoop YARN

1 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Lessons Learned Running a Container
Cloud on Apache Hadoop YARN
Billie Rinaldi
Software Engineering YARN R&D - Hortonworks
Hadoop, YARN, HDFS, Ambari, Ranger, Atlas, and Apache
are trademarks of the Apache Software Foundation

Introduction
Building Blocks for the Container Cloud
Lessons Learned
Q&A
Overview:

Disclaimer
• Some features discussed today are still experimental and/or under current
development.
• Security is a work in progress and a security assessment should be performed
before implementing these features.
• Many features are released in Apache Hadoop 3.1.0 in April 2018, but some will
not come out until 3.2.0 or 3.1.1.

Introduction

We Build, Test, and Release Open Source Software
• The rapid pace of open source results in
• dozens of releases a year
• tens of thousands of tests per release.
• Many permutations of tests
• Over a dozen supported Linux operating systems.
• Multiple backend databases.
• Nearly thirty open source products in our stack.
• Multiple supported versions of HDP per release.

Addressing the Challenges
• How is the industry addressing the these challenges? What are customers asking for?
• How can we reduce overhead to achieve greater density and improve hardware
utilization?
• How can we improve the speed at which tests run?
• How can we reuse packaging and automation?

Solution: Container Cloud
• Containers eliminate a bulk of the virtualization overhead, improving density per
node.
• Containers help reduce image variance through composition and simplified
packaging.
• Container startup time is fast, no real boot sequence.
• Containers naturally fit into YARN and its container model.
• Allow us to “use what we ship and ship what we use.”

Container Cloud Architecture
Shared Services
Resource
Management
(YARN)
Management
and
Monitoring
(Ambari)
Jenkins
Worker
(Docker)
Testing HDP and HDF releases in container clusters
HDP
(Docker)
Worker
(Docker)
Storage
(HDFS)
Service
Discovery and
REST API
(YARN Services)
Security and
Governance
(Ranger and
Atlas)
SubmitTest
LaunchTest
Worker
(Docker)
HDP
(Docker)
HDP
(Docker)
HDP
(Docker)

Two years later …
5.8+ million containers and many lessons learned.

• YARN Docker Support – Enables additional container types to make it easier
to onboard new applications and services on YARN.
• YARN Services Framework – Provides AM implementation and NM
improvements that enable long running services on YARN.
• YARN Service Discovery – Allows services running on YARN to discover one
another.

Adding Docker on YARN
• Why Docker?
• Provides a lightweight mechanism for
packaging, distributing, and isolating
processes.
• Currently the most popular containerization
framework.
• Allows YARN developers to focus on
integration instead of container primitives.
• Mostly fits into the YARN Container model.
• Buzzword compliant.

YARN Containers
• What is a YARN container?
• Process.
• Local Resources (scripts, jars, security tokens).
• Resource Requirements (CPU, Memory, I/O).
• AM requests containers.
• RM allocates containers on NMs.
• NM runs containers.
• Container Executor encapsulates platform-
specific logic needed to start a YARN container.
https://www.pepperfry.com/tupperware-mini-rectangular-white-container-850ml-1109991.html

New Abstraction: Container Runtimes
• Challenge: Container Executor approach is cluster wide; need more flexibility while still
being able to leverage existing Container Executor features.
• Solution: Runtimes added to LinuxContainerExecutor in YARN-3611 – initially released in
Apache Hadoop 2.8.0, while improvements are ongoing).
DefaultLinuxContainerRuntime DockerLinuxContainerRuntime
Existing Linux process-
based execution.
Using Docker to run and
monitor a container.

Distributed Shell and MR on Docker Examples
Environment variables are currently used to set the Container Runtime options.*
*WARNING: might change.
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTaGXMdZCdR6_RUC235TdafDqURxk-KJIptwALUmg5ZmCb3YBW7
> yarn jar $YARN_EXAMPLES_JAR pi
-Dmapreduce.map.env="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos:7"
-Dmapreduce.reduce.env="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos:7"
1 40000
> yarn jar $DSHELL_JAR
-shell_env YARN_CONTAINER_RUNTIME_TYPE=docker
-shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos:7
-shell_command "sleep 120”
-jar $DSHELL_JAR
-num_containers 1

Recent Improvements: Container Lifecycle
• Improvements to stopping and cleaning up of containers (YARN-5366)
• Improvements to handling short lived containers (YARN-5366, YARN-7914)
• Container relaunch improvements to reuse existing container (YARN-7973)
• Data in the container’s root filesystem and workdir can be recovered on
the same node
• Support for sending specific signals to the container’s root process (YARN-
5366)
• Delayed deletion for debugging (YARN-5366)

Recent Improvements: Container Security
• ACLs for privileged containers, with the ability to disable privileged
containers system wide (YARN-6623)
• Sudo / group check for running privileged containers (YARN-7221)
• Default untrusted mode for running unmodified images out of the box
(YARN-7516)
• Username to UID/GID mapping to ensure privacy (YARN-4266)
• User supplied bind mounts validated against an admin supplied whitelist
(YARN-5534)
• More restrictive YARN mounts to limit host exposure (YARN-7815)

another.

YARN Services Goals
• Long Running – Simplify the deployment and management of long running
applications on YARN.
• Easily Bring New Applications – Remove tedious process of bringing new
applications to YARN.
• Easy to Manage Applications – REST API and Command Line tools.
• Declarative Configuration – Provide configuration to the applications,
declare resource needs, specify placement policies.

YARN Services Overview
• Apache Slider – incubating at Apache since 2014, designed to make it easier
to run long-running applications on YARN.
• Simplified and first-class support for services in YARN (YARN-4692) – initiated
in 2016.
• Container orchestrator to provision docker-based or native-process based containers
(YARN-5079), integrates Slider core into YARN.
• REST API for managing services on YARN (YARN-4793).
• Simplified discovery of services via DNS mechanisms (YARN-4757).
• Released in Apache Hadoop 3.1.0!

YARN Services Docker Httpd Example
{
"name": "simple-httpd-service",
"version": "1.0.0",
"lifetime": "3600",
"components": [
{
"name": "httpd",
"number_of_containers": 2,
"launch_command": "/usr/bin/run-httpd",
"artifact": {
"id": "centos/httpd-24-centos7:latest",
"type": "DOCKER"
},
"resource": {
"cpus": 1,
"memory": "1024"
},
...
> yarn app –launch simple-httpd-service
simple-httpd-service.json

YARN Services Docker Httpd Example continued
"readiness_check": {
"type": "HTTP",
"properties": {
"url": "http://${THIS_HOST}:8080"
}
},
"configuration": {
"files": [
{
"type": "TEMPLATE",
"dest_file": "/var/www/html/index.html",
"properties": {
"content": "<html><body>Hello from
${COMPONENT_INSTANCE_NAME}!</body></html>"
}
}
]
}

Service assembly
{
"name": "httpd-proxy-service",
"version": "1.0.0",
"components": [
{
"artifact": {
"id": "simple-httpd-service",
"type": "SERVICE"
}
},
{
"name": "httpd-proxy",
"number_of_containers": 1,
"dependencies": [ "httpd" ],
"artifact": {
"id": "centos/httpd-24-centos7",
"type": "DOCKER"
}, ...
> yarn app –save simple-httpd-service
simple-httpd-service.json
> yarn app –launch httpd-proxy-service
httpd-proxy-service.json

Other features in progress
• Container upgrade
– Localize new resource while container is running (YARN-1503) portions in 2.9.0
– Restart container with new resources using same container allocation (YARN-4726)
– Support in service AM and service REST API (YARN-7512) slated for release 3.2.0
• Placement policy support in YARN services (YARN-7142)
• User supplied Docker client configs in YARN services (YARN-7996)
• Entrypoint support (YARN-7654)
• And many more!

another.

YARN Service Registry
• The YARN Service Registry allows deployed applications to register
themselves to allow discovery by other applications (YARN-913).
• Entries are stored in Zookeeper as the default k/v store, providing HA and
consistency.
• Native Java clients, REST and CLI interfaces exist for access the YARN Service
Registry.
http://api-university.com/blog/let-developers-try-your-apis-without-registration/

Simplified Discovery via DNS
Challenge - Native Java clients and REST interfaces are not ideal
discovery mechanisms for existing applications.
Solution - Exposing YARN Service Registry entries via a more
generic and widely used discovery mechanism: DNS.
The YARN Registry DNS server (YARN-4757) meets these needs.
• Watches the YARN Service Registry for new application and container
registration/deregistration.
• Creates the appropriate DNS records for the container
componentInstanceName.serviceName.user.domain
ctr-e138-1518143905142-215498-01-000007.domain
• Supports zone transfers or zone forwarding.
http://www.idownloadblog.com/2016/03/05/how-to-use-custom-dns-settings/

Lessons Learned

Success!
5.8+ million containers, 1.1+ million testsHuge uptick in adoption

Successes continued
• First full HDP release tested and certified end to end on the container cloud!
• All supported operating systems (CentOS 6/7, SLES, Ubuntu 14/16, Debian)
running in containers on CentOS 7.3 hosts!
Density per node improved by 2.5x!
14
35Virtual Machines
Containers

…but not without some pain
“No gains without pains.”
- Benjamin Franklin, Poor Richard's Almanack

IP Management
Challenge: YARN does not manage IP addresses directly.
What we did:
• Allocate a pool of IP addresses to the cluster on the same VLAN.
• Use Docker’s bridge networking with fixed_cidr option.
• Each node in the cluster is allocated 64 IP addresses from the pool.
• Use docker inspect to get the container IP address and add it to the
YARN Service Registry
• YARN Registry DNS Server registers DNS records for easy lookup.

Docker Storage Driver and Filesystem
Challenge: Many Docker Storage drivers, lots of limitations.
What is the best option for us?
What we did:
• Extensive testing of create, stop, and delete operation timings.
• Eliminate options that require significant modifications to the Linux
OS to ease adoption in enterprises.
• If possible, use a driver deemed production ready by maintainers.
• Ultimately, we landed on DeviceMapper LVM thinpool with ext4
backing filesystem ... or so we thought.

DeviceMapper kernel oops and performance
Challenge: Heavy writes to the container’s root filesystem
causing kernel panics, uninterruptible processes, and high IO wait.
What we did:
• DeviceMapper is the only viable option due to our workloads, can’t
switch to a different storage driver.
• Install SSDs and configure Docker’s graph storage to use it to
eliminate high IO wait.
• Test various RAID controller firmware, Linux kernel, and backing
filesystems to find a stable combination that doesn’t result in panics
and Docker hangs (most recently testing upgrade to CentOS 7.4 with
4.15 kernel, so we could look at overlay2 now).

User Namespacing
Challenge: YARN provides features for localized resources and log aggregation.
Running containers as the submitting user presents challenges.
What we did:
• Run all containers as the nobody user so that user and application directories
configured by YARN are available to the container.
• Update images so that nobody UID/GID match in image and host.
• Allow for “vanilla containers” that do not bind mount the YARN directories,
allowing the process in the container to run as any user, but logs can no
longer be aggregated (Docker run override disabled).
• User namespacing in Docker is lacking, as it can only remap a single user. As
this restriction is removed, we expect to migrate to this feature. Recent
improvements allow setting UID:GID pair, but further support is needed.

Image Management
Challenge: The implicit pull of a large image can lead to task timeouts.
SSD space is a premium making image clean up important.
What we did:
• Run an internal private registry to reduce WAN load.
• Jenkins job that builds and distributes the image to all nodes in the
cluster, avoiding the implicit pull from “docker run”
• Jenkins job to clean up images that are no longer needed. Full
thinpool can also cause kernel panics – aggressively age off images.
• Reuse base images where possible to reduce bandwidth.
• Work being discussed to provide first-class YARN support for image
management (YARN-3854).

Summary

Summary
• Massive density improvements.
• Greatly improved ease of use.
• Many real world lessons learned.
• Widespread internal adoption.
• Improved self service capabilities.
• Internal use of long running services a reality!

Get Involved
• Still plenty of work to go!
• Improve docker image management
and user handling
• Networking plugins.
• Security/permissions models.
• Bring Your Own Image challenges.
• Follow Along.
• Try out Apache Hadoop 3.1.0 or checkout trunk
and build it! Ansible/vagrant setups available.

Questions?
billie@hortonworks.com
skumpf@hortonworks.com

Lessons Learned Running a Container Cloud on Apache Hadoop YARN

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lessons Learned Running a Container Cloud on Apache Hadoop YARN

Similar to Lessons Learned Running a Container Cloud on Apache Hadoop YARN (20)

Recently uploaded

Recently uploaded (20)

Lessons Learned Running a Container Cloud on Apache Hadoop YARN