Docker provides an integrated and opinionated toolset to build, ship and run distributed applications. Over the past year, the Docker codebase has been refactored extensively to extract infrastructure plumbing components that can be used independently, following the UNIX philosophy of small tools doing one thing well: runC, containerd, swarmkit, hyperkit, vpnkit, datakit and the newly introduced InfraKit. This talk will give an overview of these tools and how you can use them to build your own distributed systems without Docker. Patrick Chanezon & David Chung, Docker & Phil Estes, IBM

1. Patrick Chanezon, @chanezon, Docker Inc.
Building Distributed Systems without Docker
Using Docker Plumbing Projects
David Chung, @dchungsf, Docker Inc.
Phil Estes @estep, IBM

2. French
Polyglot
Platforms
Software Plumber
San Francisco
Developer Relations
@chanezon

3. The world needs
tools of mass innovation

4. A programmable Internet would be the ultimate
tool of mass innovation

5. A commercial product,
built on
a development platform,
built on
infrastructure,
built on
standards.
Docker is building a stack to program the Internet

6. Docker Platform

7. Isolation using Linux kernel features
namespaces
 pid
 mnt
 net
 uts
 ipc
 user
cgroups
 memory
 cpu
 blkio
 devices

8. Image layers

9. 1.
Developer experience

10. 1. Get out of the way
The best tools…
2. Adapt to you
3. Make the
powerful simple

11. Docker for Mac Docker for Windows

12. 2.
Orchestration

13. ng the best way to orchestrate Docke
Docker 1.12: now with orchestration built-in.

14. Swarm mode
Service API
Cryptographic node identity
Built-in routing mesh
Docker 1.12: now with orchestration built-in.

15. Using the beta? You already have 1.12 installed.
> docker swarm init
> docker service create

16. 3.
Ops experience

17. Deep integration with native load-balancers, templates,
SSH keys, ACLs, scaling groups, firewall rules…
beta.docker.com

18. Docker CaaS

19. Goals
+ +
Agility Portability Control

20. BUILD
Development Environments
SHIP
Registry: Secure Content &
Collaboration
RUN
Control Plane: Deploy,
Orchestrate, Manage, Scale
Networking Volumes MonitoringLoggingConfig MgtCI/CD
IT Operations
Developers IT Operations
Docker CaaS Workflow

21. Docker Containers as a Service platform
23
BUILD
Developer Workflows
SHIP
Registry Services
RUN
Management
Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane
Docker Cloud
Docker Container Engine
Ecosystem Plugins and Integrations

22. Plumbing

23. 2013-05
2013-06
2013-07
2013-08
2013-09
2013-10
2013-11
2013-12
2014-01
1,000,000
0
2014-02
2014-03
2014-04
2014-05
2014-06
2014-07
2014-08
2014-09
2014-10
2014-11
2014-12
2015-01
2015-02
2015-03
2015-04
2015-05
2015-06
2015-07
2015-08
2015-09
2015-10
2015-11
2015-12
2016-01
1,000,000,000
~
10,000,000
9,000,000
8,000,000
7,000,000
6,000,000
5,000,000
4,000,000
3,000,000
2,000,000
6,000,000,000
5,750,000,000
5,500,000,000
5,250,000,000
5,000,000,000
4,750,000,000
4.500,000,000
4,250,000,000
4,000,000,000
3,750,000,000
3,500,000,000
3,250,000,000
3,000,000,000
2,750,000,000
2,500,000,000
2,250,000,000
2,000,000,000
1,750,000,000
1,500,000,000
1,250,000,000

24. 2013-05
2013-06
2013-07
2013-08
2013-09
2013-10
2013-11
2013-12
2014-01
1,000,000
0
2014-02
2014-03
2014-04
2014-05
2014-06
2014-07
2014-08
2014-09
2014-10
2014-11
2014-12
2015-01
2015-02
2015-03
2015-04
2015-05
2015-06
2015-07
2015-08
2015-09
2015-10
2015-11
2015-12
2016-01
~
2016-09
1,000,000,000
~
10,000,000
9,000,000
8,000,000
7,000,000
6,000,000
5,000,000
4,000,000
3,000,000
2,000,000
6,000,000,000
5,750,000,000
5,500,000,000
5,250,000,000
5,000,000,000
4,750,000,000
4.500,000,000
4,250,000,000
4,000,000,000
3,750,000,000
3,500,000,000
3,250,000,000
3,000,000,000
2,750,000,000
2,500,000,000
2,250,000,000
2,000,000,000
1,750,000,000
1,500,000,000
1,250,000,000
Notary
runC •
containerd •
HyperKit , VPNKit, DataKit •
SwarmKit •
libcontainer •
libnetwork • • Docker 1.8 : Docker Content Trust
• Docker for Mac
Docker for Windows
• Docker 1.12
with built-in
orchestration
• Docker 0.9 : Pluggable execution
• Docker 1.7 : Multi-Host Networking
• Docker 1.11:
OCI support

25. Notary
“Let’s stop using curl|sh”
Trusted collections for any content
Transport-agnostic
Reliable updates, proof of origin, resistant to untrusted
transport, survivable key compromise
Build on industry-leading standards and research

26. RunC
The universal container runtime
https://runc.io

27. containerd
A daemon to control runC
built for performance and density
http://containerd.tools/

28. containerd

29. Docker 1.11

30. Docker for Mac architecture
(simplified)

31. Hypervisor
Framework
vmnet Framework
Docker Container Engine
Hypervisor
Linux
VPN
Data
Service
Interface
Client Libraries
Admin GUI
CLI
Security Sandbox
Docker for Mac internals

32. Unikernels
http://unikernel.org/

33. Hypervisor
Framework
vmnet Framework
Docker Container Engine
Hyperkit
Linux
VPNKit
DataKit
Client Libraries
Admin GUI
CLI
Security Sandbox
Improving Docker with unikernel tech

34. runC

35. Open Container Initiative (OCI)
An open governance
structure for creating
open industry
standards: a common
container runtime and
image format.
• A Linux Foundation Collaborative Project
• Free from control by any particular vendor’s
specific cloud stack or ecosystem
• Includes a specification, reference
runtime* and now, a specified image
format*seeded with runc + libcontainer by Docker

36. OCI Specs & Status
> Announced June
20th, 2015
> Charter signed on
December 8th, 2015
> 49 current member
companies
> Both specifications
nearing 1.0 release
targets
https://opencontainers.org
https://github.com/opencontainers
> Runtime specification: Release 1.0.0-rc2 / September
2016
https://github.com/opencontainers/runtime-spec/releases/tag/v1.0.0-rc2
1. Very close to an official 1.0 release of the runtime spec
2. Includes required core for Linux, Windows, and Solaris
> Image format specification: Release 0.5.0 / September
2016
https://github.com/opencontainers/image-spec/releases/tag/v0.5.0
1. Seeded with Docker registry v2.2 specification
2. v1.0.0-rc1 release being voted/approved on mailing list

37. Introduction to `runc`
> runc is a client wrapper around libcontainer
> Libcontainer is the OS level interface for containers
Other platforms and architectures can implement
the libcontainer API via their own
primitives/system-level container concepts
$ docker run -it --read-only -v /host:/hostpath alpine sh
/#
{
"ociVersion": "0.6.0-dev",
"platform": {
"os": "linux",
"arch": "amd64"
},
"process": {
"terminal": true,
"args": [
"sh"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/bin”
config.json

38. ● CloudFoundry Garden OCI
implementation
● https://github.com/cloudfoundry-incubator/guardian
● Uses runc as a backend for container execution
● Docker 1.11 (and above)
● Switched from direct libcontainer API linkage to
calling runc as container executor
● Uses containerd as a gRPC daemon to
disconnect Docker daemon (API/mgmt) from
container execution (allows daemon restart in
future without container runtime impact)
runc in the “Wild”
runv - Hyper.sh; small & lightweight hypervisor wraps contained process
runz - Solaris zones implementation
> Ports/Implementations:

39. runc: An open innovation platform for containers
Implement low-level container features
Operating system level features should be defined in the OCI runtime specification
New capabilities (PID cgroup controls, checkpoint/restore, seccomp) implemented in
runC
INTEREST
OCI compliance/pluggable execution engine
Implement a OS/environment for containers via an OCI spec compliant binary
Examples: runz (Solaris zones), runv (hypervisor-based), Intel Clear Containers
Iterative container configuration test/debug
Simple variant of “Docker-like” containers with less friction for quick modifications
Low bar for dependencies: single binary + physical rootfs bundle + JSON config
INTEREST
INTEREST

40. How does Docker use runc?
Docker engine
containerd
gRPC
ctr-shim ctr-shim
runc runc
https://github.com/docker/docker
https://github.com/docker/containerd
https://github.com/opencontainers/runc
Docker client/API
HTTP/RES
T

41. OCI & runc Futures
● Entry point for OS-level container technology
implementations and added enhancements
• Recent examples: seccomp, user namespaces, checkpoint/restore
• Many smaller examples (lots of changes required for fully unprivileged
containers)
● More users and contributed implementations (for
runtime and image)
● What will you do with runc?

42. @estesp
github.com/estesp
estesp@gmail.com
https://integratedcode.us
IRC: estesp
Phil Estes, IBM
DEMO
$ runc run alpine
# /

43. InfraKit

44. Problem:
Managing Docker on different infrastructure is
difficult and not portable.

45. Consistent User Experience
48
How do we handle updates to a cluster??

46. Docker for AWS
EBS ELB
Container Engine
Storage plugin
Infrastructure Management
Network plugin Orchestration
IAM
CloudFormation
EC2VPC
Admin interface
Linux
User Applications / Services

47. Docker for AWS
EBS ELB
Container Engine
Storage plugin
InfraKit
Network plugin Orchestration
IAM
CloudFormation
EC2VPC
Admin interface
Linux
User Applications / Services

48. InfraKit
A toolkit for building declarative, self-healing
infrastructure.

49. Declarative
• JSON configuration for desired infrastructure state:
• Specification of instances — vm image, instance type, etc.
• Group properties — size, logical identifiers, etc.
• Design patterns encourage
• encapsulation
• composition
• Config is input to all operations — system figures out what to do
52

50. Self-healing
• Composed of a set of active components / processes that
• monitor infrastructure state
• detect state divergence
• take actions
• Continuous monitoring and reconciliation — always on
• No downtime — rolling update
53

51. Toolkit
• Primitives for managing collections of resources
• create, scale, destroy
• rolling update
• Abstractions & Developer SPI
• Group - manages collection of resources
• Instance - describes the physical resource
• Flavor - extra semantics for handling instances
• A collection of executable, active components — plugins
• Initially, Go daemons in the toolkit
• Soon, easy management via Docker Plugins (runc)

52. Architecture

53. Instance Plugin
• Spec: specification / model of an instance (e.g. vagrant, EC2):
• Logical ID, Init, Tags, and attachment
• Platform-specific properties
• Methods:
• /Instance.Validate
• /Instance.Provision
• /Instance.Destroy
• /Instance.DescribeInstances
• Examples: instance plugins for EC2, Azure VM, Vagrant, …56

54. Flavor Plugin
• Gives more context about the group members:
• Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’)
• Application-specific notions of ‘health’
Is the node not only present but also joined a swarm?
• Methods:
• /Flavor.Validate
• /Flavor.Prepare
• /Flavor.Healthy
• Examples: flavor for Zookeeper members, Docker swarm
nodes57

55. Group Plugin
• Main entry point for user interaction:
• Create, describe update, update, destroy
• Config JSON is always the input
• Composed of Instance and Flavor — mix and match to
manage cattle (fungible) or pets (special)
• Methods:
• /Group.Watch
• /Group.Unwatch
• /Group.Inspect
58
• /Group.DescribeUpdate
• /Group.Update
• /Group.StopUpdate
• /Group.Destroy

56. Configuration
Example config file (zk.conf): Group configuration = Instance + Flavor
{
"Properties": {
/* raw configuration */
}
}
{
"groups" : {
"my_zookeeper_nodes" : {
"Properties" : {
"Instance" : {
"Plugin": "instance-vagrant",
"Properties": {
"Box": "bento/ubuntu-16.04"
}
},
"Flavor" : {
"Plugin": "flavor-zookeeper",
"Properties": {
"type": "member",
"IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"]
}
}
}
}
}
}

57. Operations
• Make sure the plugins are running:
• infrakit/group &; infrakit/zookeeper &; infrakit/vagrant &;
• “Watch” the group starts management:
• infrakit/cli group watch zk.conf
• Update the config, e.g. change size or add IP address
• Describe changes before committing —
infrakit/cli group describe zk.conf
• Begin update —
infrakit/cli group update zk.conf
60

58. Demo
61

59. Today
62
• InfraKit is just getting started… only
primitives for working with groups like
clusters of hosts
• But we have big plans
• Improve group management strategies
• More resource types — networking, load
balancers, storage…
• A cohesive framework for active
management of infrastructure — physical,
virtual, or containers

60. Get Involved
• Help define and implement new and interesting plugins
• Instance plugins for different infrastructure providers
• Flavor plugins for systems like etcd or mysql clusters
• Group controller plugins — metrics-driven auto scaling
and more
• Help define interfaces and implement new infrastructure
resource types — load balancers, networks and storage
volume provisioners
63

61. More Info
• Github:
https://github.com/docker/infrakit
• A quick tutorial:
https://github.com/docker/infrakit/blob/master/docs/tutorial.m
d
64

62. Booth D38 @ LinuxCon + ContainerCon
Tues Oct 4th
• Build Distributed Systems without Docker, using Docker Plumbing Projects - Patrick Chanezon, David Chung and Captain Phil
Estes
• Getting Started with Docker Services - Mike Goelzer
• Swarmkit: Docker’s Simplified Model for Complex Orchestration - Stephen Day
• User Namespace and Seccomp Support in Docker Engine - Paul Novarese
• Build Efficient Parallel Testing Systems with Docker - Docker Captain Laura Frank
Wed Oct 5th
• How Secure is your Container? A Docker Engine Security Update - Phil Estes
• Docker Orchestration: Beyond the Basics - Aaron Lehmann
• When the Going gets Tough, get TUF Going - Riyaz Faizullabhoy and Lily Guo
Thurs Oct 6th
• Orchestrating Linux Containers while Tolerating Failures - Drew Erny
• Unikernels: When you Should and When you Shouldn’t - Amir Chaudhry
• Berlin Docker Meetup
Friday Oct 7th
• Tutorial: Comparing Container Orchestration Tools - Neependra Khare
• Tutorial: Orchestrate Containers in Production at Scale with Docker Swarm - Jerome Petazzoni

63. THANK YOU