Christian Kniep from Docker Inc. gave this talk at the Stanford HPC Conference. "This talk will recap the history of and what constitutes Linux Containers, before laying out how the technology is employed by various engines and what problems these engines have to solve. Afterward, Christian will elaborate on why the advent of standards for images and runtimes moved the discussion from building and distributing containers to orchestrating containerized applications at scale. In conclusion, attendees will get an update on what problems still hinder the adoption of containers for distributed high performance workloads and how Docker is addressing these issues." Christian Kniep is a Technical Account Manager at Docker, Inc. With a 10 year journey rooted in the HPC parts of the german automotive industry, Christian Kniep started to support CAE applications and VR installations. When told at a conference that HPC can not learn anything from the emerging Cloud and BigData companies, he became curious and was leading the containerization effort of the cloud-stack at Playstation Now. Christian joined Docker Inc in 2017 to help push the adoption forward and be part of the innovation instead of an external bystander. During the day he helps Docker customers in the EMEA region to fully utilize the power of containers; at night he likes to explore new emerging trends by containerizing them first and seek application in the nebulous world of DevOps. Watch the video: https://wp.me/p3RLHQ-i4X Learn more: http://docker.com and http://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: http://insidehpc.com

1. Container Technology 101
The bits and pieces
Christian Kniep, v2018-01-18
Technical Account Manager

2. Container History
Back in the days...

3. Brief History of Container Technology
Jails Zones Namespaces Docker
VServer cgroups LXC
Container Runtime and Image Format Standards, Jeff Borek, Stephen Walli, KubeCon Dec/2017
FreeBSD Jails
expand on
Unix chroot to
isolate files
Linux-VServer
ports kernel
isolation, but
requires
recompilation
Solaris Zones
bring the
concept of
snapshots
Google introduces
Process
Containers, merged
as cgroups
RedHat adds user
namespaces,
limiting root access
in containers
IBM creates LXC
providing user tools
for cgroups and
namespaces
Docker provides
simple user tools
and images.
Containers go
mainstream
2000
2001
2004
2006
2008
2008
2013

4. Linux Namespaces 101
Short Recap

5. Example PID
Processes Isolation
● host sees all processes with real PID from the Kernels perspective
● first process within PID namespace gets PID=1
Host
cnt0
ps -ef
cnt1
start.sh
java -jar ..
cnt2
start.sh
java -jar ..
health.sh

6. Resource Isolation of Process Groups
7 as of Kernel 4.10
1. MNT: Controls mount points
2. PID: Individual process table
3. NET: Network resources (IPs, routing,...)
4. IPC: Prevents the use of shared memory between processes
5. UTS: Individual host- and domain name
6. USR: Maps container UID to a different UID of the host
7. CGRP: Hides system cgroup hierarchy from container

7. Container Namespaces
A starting container gets his own namespaces.
PIDMNT IPCNET USR
Host
UTS CGRP
cnt0 cnt1 cnt2
But can share namespaces with other containers or even the host

8. All In
When using all host namespaces - we are on the host (almost like ssh).
PIDMNT IPCNET USR
Host
UTS CGRP
cnt0
$ docker run -ti --rm
--privileged
--security-opt=seccomp=unconfined
--pid=host
--uts=host
--ipc=host
--net=host
-v /:/host
ubuntu bash
root@linuxkit-025000000001:/# chroot /host
/ # ash
/ #

9. CGroups
While namespaces isolate,
Control Groups constraint resources.

10. CGroups / Filesystem Layering
Short Recap [cont]

11. Overlay Filesystem
Compose a FS from multiple pieces
ubuntu:16.04
openjre:9-b114
appA.jar:1.1 appB.jar
ARG FROM openjre:9-b114
COPY appB.jar /usr/local/bin/
CMD [“java”, “-jar”, “/usr/local/bin/appB.jar”]
ARG FROM openjre:9-b114
COPY appA.jar /usr/local/bin/
CMD [“java”, “-jar”, “/usr/local/bin/appA.jar”]
FROM ubuntu:16.04
ARG JRE_VER=9~b114-0ubuntu1
RUN apt-get update
&& apt-get install -y openjdk-9-jre-headless=${JRE_VER}
&& java -version
openjre:9-b117

12. Stacked View
Hardware
Host Kernel
Userland
Services Hypervisor
Kernel
Userland
Services1 Services2
Userland
Kernel
Hardware
Host Kernel
Userland
Services
Userland
appB appC
Userland
Cnt1 Cnt2
VM1 VM2
Traditional Virtualization os-virtualization

13. Traditional Virtualization
hardware
kernel
hypervisor
kernel
Interface View
libs
From Application to Kernel
application
libs
application
lib-calls
102
syscalls
101
hypercalls
hardware
kernel
hw-calls
container
VM-shortcuts
(PVM,pci-passthrough)
os-virtualization

14. THANK YOU :)
@CQnib