containerd is a widely used container runtime that is now a CNCF project. It is designed to be embedded in larger systems rather than used directly by developers. containerd provides core primitives for managing containers on a host, such as container execution, image distribution, and storage. It focuses on simplicity, robustness, and portability. containerd will serve as a core container runtime for the CNCF ecosystem and is being integrated with projects like Kubernetes.

1. container the universal
container runtime
Justin Cormack, Docker
@justincormack

2. containerd: now a CNCF project
“containerd is a widely used container runtime with an emphasis on
simplicity, robustness and portability”
“containerd is designed to be embedded into a larger system, rather than
being used directly by developers or end-users. containerd will serve as a
core container runtime for the CNCF ecosystem.”
2

3. containerd: cloud native
Uses
Can be used in
3

4. containerd: not a new project, an evolution
• Started in November 2015 to control runc, the OCI runtime
• Used by Docker since 1.11 in 2016 as a container runtime
• Relaunched in December 2016 with new scope
• Docker now using 0.2 branch
• 1.0 master branch is where the new work is taking place
entirely new scope, and donated to CNCF
4

5. containerd: What is a Core Container Runtime?
Component that provides core primitives to manage containers on a host
• Container execution and supervision
• Image distribution
• Network Interfaces and management
• Local storage
• Native plumbing level API
5

6. containerd: isn’t that what Docker does?
Docker provides much more than that, it is a platform
• user interface
• backward compatibility with existing applications
• clustering with Swarm mode
• opinionated workflows and defaults, such as Docker Hub
• commercial support
• product direction not entirely community led
Some people do not want those things, containerd is for them.
6

7. containerd: role in Container Ecosystem
7

8. containerd and Kubernetes
• currently Docker is the default and best supported runtime
• containerd is being written to replace the relevant code in Docker
• the CRI acts as an API for runtimes in Kubernetes
• work on integration in https://github.com/cri-containerd/kubernetes
• Kubernetes PR https://github.com/kubernetes/kubernetes/pull/43655
• containerd 1.0 milestone support in at least one runtime
• likely 1.0 will be shipped with at least Docker and Kubernetes support
• working with Kubernetes is an essential part of roadmap
8

9. containerd: view from Tim Hockin at Google
9

10. containerd: roadmap to 1.0
Timetable is aggressive for 1.0 in 2Q 2017 but the evolution helps:
• some code is reused from Docker
• some is rewritten and improved based on experience
• the runtime code is already in production with lots of users
• focus on getting APIs clear and clean
• extensible via plugins
• will be supported for one year
• can evolve in new directions for 2.0, ...
• limit scope...
10

11. What is not in Containerd...
11
Limit the scope to where there is agreement, and not to constrain
platforms.
• No networking
• No volumes
• No logging
• No build

12. networking in containerd...
12
• No networking in containerd
• https://github.com/docker/containerd/issues/362
• This is what the users of containerd wanted
• Networking varies too much between platforms
• continue to use CNI or other APIs as before

13. No networking in containerd...
13
• Provide a network namespace
– Join a pre populated network namespace
• Use OCI Hooks to initialize namespace
– Exec a command with the container’s state to initialize network
• Setup networking between create and start
– Create container
– Setup network interfaces
– Start user’s process

14. No volume management in containerd
• No consensus yet around interfaces
• Storage layer can be hooked in at OCI layer
• It is just a set of mounts at this level
“The Container Storage Interface (CSI) is a proposed new industry
standard for cluster-wide volume plugins. This is a joint proposal from a
group of us who work on Docker, Kubernetes, Mesosphere and Cloud
Foundry. CSI is currently in the early draft stage, and we are seeking
feedback from the community.”
14

15. No log management in containerd (yet)
• Output streams of containers can be handled as required
• Platform can arrange logging how it wishes
• https://github.com/docker/containerd/issues/603 discusses changes
• Possibly adding timestamps, formatting in the shim
• Different use cases, eg docker run can have binary, not log output
15

16. No build in containerd
• Use other tools for building containers
• Very different concerns from runtime
• The storage driver can create snapshots from a writeable filesystem
16

17. Ok what is in containerd then?
containerd/api/services
• content
• rootfs
• execution
• shim
Each of these are GRPC APIs
17

18. 18
Docker storage architecture
18
Graph Driver
“layers” “mounts”
Layer Store
“content addressable layers”
Image Store
“image configs”
Containers
“container configs”
Reference Store
“names to image”
Daemon

19. 19
containerd storage architecture
19
Snapshotter
“layer snapshots”
Content Store
“content addressed blobs”
Metadata Store
“references”
dist ctr
Config
Rootfs (mounts)

20. containerd: content
Content is a content addressed store
service Content {
rpc Info(InfoRequest) returns (InfoResponse);
rpc Delete(DeleteContentRequest) returns (google.protobuf.Empty);
rpc Read(ReadRequest) returns (stream ReadResponse);
rpc Status(StatusRequest) returns (stream StatusResponse);
rpc Write(stream WriteRequest) returns (stream WriteResponse);
}
Content is identified via a digest, ie content hash.
Status gives the status of an in progress write transaction.
20
Content Service
Write
Read
Content
Digested

21. containerd: rootfs
service RootFS {
rpc Unpack(UnpackRequest) returns (UnpackResponse);
rpc Prepare(PrepareRequest) returns (MountResponse);
rpc Mounts(MountsRequest) returns (MountResponse);
}
• Unpack a downloaded image
• Prepare the root filesystem from the set of layers
• Mounts returns a list of mounts to make, does not execute them
21

22. containerd layers
containerd has three built in ways of layering images:
• overlay
• btrfs
• “vfs”
These correspond to overlay and snapshotting drivers, which are the two
models. The aim is to make sure the API provides support for both types,
not to be comprehensive. Also a plain driver that does not use layers.
Plugins will provide additional mechanisms, eg ZFS
22

23. containerd: execution
service ContainerService {
rpc Create(CreateRequest) returns (CreateResponse);
rpc Start(StartRequest) returns (google.protobuf.Empty);
rpc Delete(DeleteRequest) returns (DeleteResponse);
rpc Info(InfoRequest) returns (containerd.v1.types.Container);
rpc List(ListRequest) returns (ListResponse);
rpc Events(EventsRequest) returns (stream containerd.v1.types.Event);
}
Fairly simple API for creating containers, and getting info about them
23

24. containerd: shim
The shim is a process that handles IO for a container.
service Shim {
rpc Create(CreateRequest) returns (CreateResponse);
rpc Start(StartRequest) returns (google.protobuf.Empty);
rpc Delete(DeleteRequest) returns (DeleteResponse);
rpc Exec(ExecRequest) returns (ExecResponse);
rpc Pty(PtyRequest) returns (google.protobuf.Empty);
rpc Events(EventsRequest) returns (stream containerd.v1.types.Event);
rpc State(StateRequest) returns (StateResponse);
rpc Pause(PauseRequest) returns (google.protobuf.Empty);
rpc Resume(ResumeRequest) returns (google.protobuf.Empty);
rpc Exit(ExitRequest) returns (google.protobuf.Empty);
}
24

25. containerd command line
The command line is not the API
There are some commands for testing
However their CLIs are unstable and may be incomplete
They can be useful for understanding what is going on, and writing tests,
and trying out low level operations, eg applying layers.
25

26. dist: everything for images
images list images known to containerd
pull pull an image from a remote
fetch fetch all content for an image into containerd
fetch-object retrieve objects from a remote
ingest accept content into the store
active display active transfers.
get get the data for an object
delete, del, remove, rm permanently delete one or more blobs.
list, ls list all blobs in the store.
apply apply layer from stdin to dir
rootfs setup a rootfs
26

27. ctr: everything for containers
run run a container
events display containerd events
delete delete an existing container
list list containers
info get info about a container
shim interact with a shim directly
pprof provides golang pprof outputs for containerd
27

28. containerd end to end
As of last week there is an end to and PoC of image pull and run
dist pull docker.io/library/redis:alpine
ctr run --id redis -t docker.io/library/redis:alpine
28

29. containerd Prometheus endpoint
GRPC and container metrics exposed via Prometheus endpoint
# HELP container_memory_usage_usage_bytes The memory usage
# TYPE container_memory_usage_usage_bytes gauge
container_memory_usage_usage_bytes{id="test"} 69632
# HELP container_memory_writeback_bytes The writeback amount
# TYPE container_memory_writeback_bytes gauge
container_memory_writeback_bytes{id="test"} 0
# HELP container_per_cpu_nanoseconds The total cpu time per cpu
# TYPE container_per_cpu_nanoseconds gauge
container_per_cpu_nanoseconds{cpu="0",id="test"} 180986
container_per_cpu_nanoseconds{cpu="1",id="test"} 669445
container_per_cpu_nanoseconds{cpu="2",id="test"} 793582
container_per_cpu_nanoseconds{cpu="3",id="test"} 2.288791e+06
# HELP container_pids_current The current number of pids
# TYPE container_pids_current gauge
container_pids_current{id="test"} 1
29

30. containerd: Windows
• Full parity for Microsoft Windows containers is planned
• Not in codebase yet
• Full Windows OCI support finally
And other platforms, via OCI eg runv, and some potentially direct
30

31. containerd
• easy to use
• simple design
• consistent GRPC interfaces
• clean design, fixes complexities from Docker
• smaller component with narrower focus
• easier to understand
• long term interface stability
• portable
31

32. Get involved
Currently at
• https://github.com/docker/containerd
Will probably move since CNCF donation but there will be a redirect...
New twitter account!
• @containerd on twitter
try in docker
docker run --privileged -d dmcgowan/containerd
32

33. containerd weekly report
33

34. Docker Internals Summit @ DockerCon
• containerd in the morning with the maintainers
• Other Docker projects in the afternoon (Libnetwork, Notary, SwarmKit,
InfraKit, VPNKit, DataKit, HyperKit, etc)
You don’t have to attend the conference to attend this summit on 20 April
34

35. THANK YOU