This document discusses the challenges and developments in live container migration within the OpenStack ecosystem, highlighting the role of CRIU (Checkpoint and Restore In Userspace) in enabling this feature. It outlines the integration of CRIU with Docker, the steps for executing live migrations, and addresses difficulties such as managing container metadata and networking issues. Demos showcasing various migration scenarios are also mentioned, alongside potential integrations with OpenStack services.

1. Live Container Migration
Phil Estes, STSM Open Containers, IBM Cloud, @estesp
Shaun Murakami, Senior Engineer, IBM Cloud, @stmuraka

2. Background
• Live migration supported by most hypervisors in OpenStack
• Container use across OpenStack ecosystem growing
MAGNUM, MURANO, KOLLA, KURYR, KUBERNETES
• Why can’t I do live migration of containers?
http://docs.openstack.org/developer/nova/support-matrix.html

3. Why?
• A hypervisor is controlling an emulated (complete) machine
• System suspend/resume is a solved problem
• Hypervisor-based live migration can take advantage of this
• Containers are processes
• Can the Linux kernel “migrate” a process?
• Well, not as easily…
• Until CRIU
• “Checkpoint Restore In Userspace”
• Work began in 2011/2012 in the Linux kernel
• 3.11 kernel + CRIU in late 2013 had working release
• 2016: CRIU 2.0 released

4. CRIU: A Brief Overview
• Proposed and led by Pavel Emelyanov from OpenVZ team
• Key difference from other checkpoint/restore attempts in Linux
was that the C/R work was performed from userspace
• All major kernel changes required were completed by the 3.11 Linux
kernel
• CRIU handles “freezing” all the process state:
• process/thread info+caps+uid/gid info, application memory, open files,
pipes/fifos, UNIX sockets, network sockets [magic!], IPC, timers, signals,
terminals, events/inotify state
See https://criu.org/Main_Page for more information.

5. CRIU & Docker
• The Docker engine is bringing CRIU
support to the Docker API/client UI
• Support has been merged into the
engine as of September 2016
• Will be part of the Docker 1.13 release
• Hard work (and perseverance) by Ross
Boucher
• Support already existed in
libcontainer (and the runc) codebase
• Popular demo from DockerCon SF 2015
migrating Quake game backend
between hosts

6. Docker Implementation Overview
• Docker 1.13 use will require the experimental build to allow for
testing/validation of capabilities and use cases
• Simplest model implemented allows for checkpoint associated
with a specific container; restore re-starts same container at the
named checkpoint
• Live migration not an intended feature. Don’t try this at home!
https://criu.org/Live_migration
# docker checkpoint [OPTIONS] container checkpoint_id
OPTIONS: [--keep-running]
# docker start --checkpoint checkpoint_id [OPTIONS] container

7. Implementation Details
Two potential paths for live migration of container data:
memory
process state
filesystem
contents
memory
process state
filesystem
contents
1. Only in-memory
changes to process
state
2. Filesystem is
untouched (e.g. a
read-only root
filesystem)
1
1. In-memory changes
to process state
AND
2. Changes to root
filesystem
underneath
container process
2

8. In-memory Only Sequence
• Checkpoint the container
• Migrate checkpoint metadata to target
• Pull (or pre-pull) exact image on target
• Create container from same image at target
• Extract checkpoint metadata into new container path
• Start container with docker start --checkpoint
1
memory
process state
filesystem
contents

9. In-memory + Filesystem Sequence
• Checkpoint the container
• Migrate checkpoint metadata to target
• docker export stopped container, migrate to target
(docker import on target to re-create target image)
• Read container metadata from source container
• Create container from new image at target
• Replicate container metadata into new container
• Extract checkpoint metadata into new container path
• Start container with docker start --checkpoint
2
memory
process state
filesystem
contents

10. Other Challenges
> Handling Container Metadata
• docker export -> docker import only addresses filesystem
migration; does not save container runtime metadata (e.g.
docker inspect information)
> Networking across Docker Hosts
• IP conflict with getting same IP assignment across hosts
• Requires load balancer solution fronting migrating services
> Volume Support
• External volume data migration would require a solution that
supported volume drivers/data export (use clustered FS)

11. Demo time!
Demonstrate basic checkpoint restore on a single host1
2
3
In-memory database migration across hosts
Video streaming container migration

12. Demo 1
A counter on the same host with checkpoint -> restore
busybox
HOST A
busybox’
HOST A
# docker run busybox
1
2
3
4
<checkpoint>
# docker start --ch...
<restore>
5
6
7
8
9
10
.
.

13. Demo 2
A tweet producer/consumer application centered around
an in-memory database, redis.
redis
tweet2redis
load balancer
(nginx)
HOST A HOST B
redis’
tweetviewer
(node.js)
HOST C

14. Demo 3
A video streaming back-end migration
Video
Stream
load balancer
(nginx)
HOST A HOST B
Video
Stream’
HOST C
video
client

15. Integration with OpenStack
• Magnum community integration of migration “steps”
• Preconfigured nova instances with criu support/capabilities?
• Normalize networking aspect via Kuryr/overlay capabilities
• Use of SwarmKit/Docker with Swarm mode features for service
discovery (remove requirement on self-managed load balancer)

16. Thank You!