This document discusses integrating Docker containers with YARN by introducing a Docker container runtime to the LinuxContainerExecutor in YARN. The DockerContainerRuntime allows YARN to leverage Docker for container lifecycle management and supports features like resource isolation, Linux capabilities, privileged containers, users, networking and images. It remains a work in progress to support additional features around networking, users and images fully.

1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
YARN and the Docker
Container Runtime
Dublin, April 2016
Sidharta S

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Context
LinuxContainerExecutor
Why docker with YARN?

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Container Runtimes in LinuxContainerExecutor
⬢ Support for Container Runtimes in YARN was added as part of – YARN-3611
(umbrella). Multiple container types are supported in the same executor.
⬢ The current mechanism of handling container lifecycle is moved into its own
runtime
⬢ A new docker container runtime is introduced that manages docker containers
⬢ LinuxContainerExecutor can delegate to either runtime on a per application basis
⬢ Clients specify which container type they want to use – currently via environment
variables but eventually through well-defined client APIs.
⬢ We could support more container types in the future.

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DockerContainerRuntime in LinuxContainerExecutor
⬢ Exposes a subset of Docker container lifecycle functionality
⬢ Docker v1.10.x required for some of the work being planned (e.g. user namespaces)
⬢ A recent linux kernel is required (3.10+) for basic functionality. Some features (e.g.
overlay fs) require an even more recent kernel.

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DockerContainerRuntime : Resource Isolation
⬢ Support added in YARN-4553
⬢ LinuxContainerExecutor still manages resource isolation and enforcement .
⬢ Docker uses the cgroup specified by LCE ( --cgroup-parent introduced in docker v 1.6 ,
net_cls support added to libcontainer recently – support added in v1.9)

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DockerContainerRuntime : Linux Capabilities
⬢ Support added in YARN-4258
⬢ Based on linux capabilities
⬢ Admin controlled – cluster administrator can control which capabilities docker
containers have on the cluster
(yarn.nodemanager.runtime.linux.docker.capabilities)
⬢ Default set is based on what docker uses by default.

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DockerContainerRuntime : Privileged Containers
⬢ Support added in YARN-4262
⬢ Allows certain applications to run in docker containers – e.g. oracle
⬢ This could be a security hazard so access to this needs to be controlled :
–Disabled by default (yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed)
–Admin controlled whitelist (yarn.nodemanager.runtime.linux.docker.privileged-
containers.acl)
–This whitelisted set of users allowed to launch privileged containers – but they must explicitly
request for it (YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER)

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DockerContainerRuntime : Users
⬢ Docker runs the container as the specified user (-u)
⬢ This user needs to be available in the image being used.
⬢ Depending on capabilities (CAP_SETUID), privileged escalation could occur.
⬢ Docker v1.10 added support for user namespaces – requires daemon re-
configuration. Support for this in DockerContainerRuntime needs to be planned.

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DockerContainerRuntime : Networking
⬢ Defaults to --net=host in the docker container runtime.
–This is not secure – but this is the only way some applications can run.
–We need to switch the default to bridged mode or an admin specified network
plugin
⬢ Network plug-in support was added in v1.9 .
–This should allow for more sophisticated networking scenarios
–YARN doesn’t have to do anything except delegate to the specified network
plugin when launching the container.
–Support for this is a work in progress (YARN-4007)

10. 1
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
DockerContainerRuntime : Images
⬢ Localization via HDFS? : We could localize images from HDFS and load them using
‘docker load’.
⬢ This approach has the advantage of using an existing HDFS instance for
storage/distribution at scale.
⬢ However :
–we lose some of the optimizations/functionality that using a full-fledged docker
registry might provide.
–We have to figure out security implications. What if users clobber each others
images when ‘loading’ ?

11. 1
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demos
Spark on Docker
YARN on YARN

12. 1
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Q&A