This talk was presented at the Linux Plumbers Conference in Dusseldorf, Germany (http://www.linuxplumbersconf.org/2014/ocw/sessions/1899).

1. An Experiment in
Checkpointing and Restoring
Docker Containers with CRIU
Linux Plumbers Conference
October 17, 2014
Saied Kazemi (saied@)

2. Motivation
● Container migration through native Checkpoint and Restore (C/R)
Docker Meetup 9/17/14
support in Docker using CRIU
host A
docker checkpoint <id>
host B
docker restore <id>

3. Docker Meetup 9/17/14
Docker Client, Server, and Containers
client server
docker run ... docker -d
init
grandchild
Global
Namespace
Private
Namespace
container 1 container 2

4. criu
container
# criu dump ...
# criu restore ...

5. Docker Meetup 9/17/14
External C/R Issues
● Manual Set Up
○ Filesystem, cgroups
● Container State
○ After checkpoint, Docker thinks the container has finished and exited
○ After restore, Docker doesn’t know container has resumed
● Process Tree Ownership
○ Restored process tree is a child of system-wide init, not the Docker
daemon
● Other “Plumbing” Issues
○ docker stop, docker kill, etc.

6. container
Native C/R
exec driver nsinit
libcontainer
criu
External C/R
# criu dump ...
# criu restore ...
docker
# docker checkpoint ...
# docker restore ...

7. Docker Meetup 9/17/14
CRIU and Docker Containers
● There were a number of issues C/R’ing Docker containers
○ See backup slides for details
● Excellent support from upstream CRIU developers and community
● With CRIU 1.3, now possible to C/R
○ Works with AUFS (default) as well as VFS and UnionFS
○ Device Mapper not tested
● No container migration yet

8. Docker Meetup 9/17/14
Checkpoint and Restore Demo
● Using docker_cr.sh helper script (external)
● Using nsinit binary (external)
● Using Docker (native, proof of concept)

9. Docker Meetup 9/17/14
Backup Slides

10. Docker Meetup 9/17/14
Docker C/R Options
● There are two options to checkpoint and restore:
A) The Docker daemon and (all) its containers
and
B) An individual container (without the Docker daemon)
● Option A isn’t currently possible with CRIU due to nested
namespaces
○ Option B is possible today on the same machine
○ Will look into adding migration support

11. Docker Meetup 9/17/14
Issues and Solutions
● Issue: nested PID namespaces
○ two ways to start a container: interactive ($ docker run -i ...) or
detached ($ docker run -d ...)
○ in both cases the process is a child of the docker daemon (not the
docker client) running in global PID namespace
○ CRIU does not support nested PID namespaces
● Solution: C/R is done on process tree without Docker

12. Docker Meetup 9/17/14
Issues and Solutions
● Issue: external bind mounts
○ /etc/{hosts,hostname} from container’s config dir
○ /etc/resolv.conf from container’s config dir (or /etc/resolv.conf in older
versions)
○ /.dockerinit from Docker’s init dir in older versions
○ bind mount paths for files in /etc can be obtained with docker inspect,
but not for /.dockerinit
● Solution: external bind mount support with --ext-mount-map

13. Docker Meetup 9/17/14
Issues and Solutions
● Issue: /dev/null bind mount over /proc/kcore
○ appeared in Docker 0.10.0, caused dump failure
● Solution: patch 494c044
● Issue: dumpable flag
○ appeared in Docker 0.11.1 (libcontainer dropping all capabilities,
keeping those specified in config)
○ value is set to 2 by which cannot be restored
● Solution: patch 8870aa1

14. Docker Meetup 9/17/14
Issues and Solutions
● Issue: restoring cgroups subdirs and properties
○ after checkpointing, Docker daemon would remove container’s
cgroups subdirs (because the container has “exited”)
○ after restoring subdirs, properties were not restored
● Solution: cgroups restoration support with --manage-cgroups

15. Docker Meetup 9/17/14
Issues and Solutions
● Issue: stdin in detached mode
○ container’s stdin set to the global /dev/null in detached mode
$ docker run -d …
● Solution: fixed in Docker
○ use --evasive-devices for older Docker versions

16. Docker Meetup 9/17/14
Issues and Solutions
● Issue: AUFS
○ /proc/<pid>/map_files symbolic link paths point inside AUFS branches
○ CRIU gets confused seeing the same file in its physical location (in the
branch) and its logical location (from the root of mount namespace)
○ fixing the kernel is the right solution but time-consuming to roll out
● Solution:
○ fixed in AUFS (but will take time to be available in all distros)
○ in the meantime, CRIU patch d8b41b6 will compensate for the
problem