This document discusses providing persistent, stateful services with Docker clusters. It covers using Docker volumes and namespaces to manage storage, implementing "storage engines" to back up volumes for different clouds, and using supercontainers to control the host and peer containers. It summarizes setting up stateful Docker clusters using Mesos/Marathon and scheduling a supercontainer volume service for each host to support backups across multiple storage engines.

1. Persistent, stateful services with docker
clusters, namespaces and docker volume
magic
Michael Neale
Co-founder, CloudBees (that Jenkins
company)

2. Agenda
Supercontainers and storage
Privileges
It’s all files (part 2)
Controlling the host and peer containers
Storage engines
Stateful docker clusters
“off the shelf” cluster scheduling
The solution chosen
Other tools out there
Credits…
Background
Use-case for stateful services
Docker volumes
Quick namespaces revision
nsenter
Mounts and Volumes
It’s all files (part 1)
the mount namespace
creating bind mounts
docker volume api (use it!)

3. Background
The Need for Stateful Services

4. Basis of this presentation:
.. was learned while building an elastic and scalable
Jenkins based product for multiple cloud environments, on
docker

5. —Michael Neale
“No containers were hurt as
part of this production.”
5

6. My history with docker
6
Ex Red Hat where I heard about “control groups”
Starting CloudBees, looking at ways to fairly multi tenant
Later would discover (and with much help) use LXC
Saw a video of Solomon demoing docker and didn’t believe it
Still didn’t believe it
For the longest time didn’t believe it

7. CloudBees & Docker
7
Actually spoke about this at DockerCon 2014 (the first one!)
cgroups -> LXC -> LXC + ZFS copy-on-write
Like dotCloud - ran a PaaS (as well as CI/CD toolchain)
In 2014 moved to focus on CI/CD (dotCloud focussed on docker)
In 2014 moved to adopt docker over LXC (and ZFS)
Using: Docker Hub (private repos), Private Registry
Many of our customers are commercial users of docker
Docker Jenkins plugins: docker hub, build and publish and many more

8. 8
Put all the things (OSS and commercial) on docker hub

9. 9
I started the “official” jenkins image early on
updated now ~weekly (with LTS images also)

10. 10
one MEEELION ??

11. A stateless cluster of apps is the dream
11
But the reality is, many apps still need state, a disk
Databases for example
Hands up who would run Oracle on NFS?

12. Reality: local disk
Network filesystems are great*
But sometimes you need the data close to the processing
EBS, HDFS, GCP, OpenStack block storage…
BUT: how to balance this need for local state with “ephemeral” servers
Servers come and go, need to restore the data (fast)
Need to backup the data (delta/snapshots - fast)
Alternatives: SANs (reattach volumes to replacement nodes, some
clouds also support this)
Reason for backups: resilience. Volumes can disappear too.

13. Current product
13
Years of experience with containers
EC2, ZFS, EBS, LXC
learn from it to build something new and “turn key” installable,
powered by docker
I accidentally created a cluster scheduler (it happens.. please don’t)
An evolved “pre-docker” system

14. Aim: a new product
14
A distributed Jenkins cluster
10000s of “masters”, 100000s of elastic build workers
Utilise “Off The Shelf” expertise based around docker: Mesos, Docker
Swarm, Kubernetes
Work within existing constraints of a lively and evolving open source
project
(this means accepting local disk state… for now)

15. Additional Constraints
Only want to depend on docker being present on “worker nodes”
Off the shelf cluster scheduler
Use local disk*
Multiple target clouds to be supported
Multiple storage “engines” to be supported
* Would love to refactor to DB backed

16. “Storage engines?”
“The thing that backs up and restores local disks”
eg: EBS (snapshots), rsync, NFS, ZFS send …
Same cluster management, same api, different storage tech for different
clouds/needs.
Ensures volumes are backed up in a consistent state (using LVM
snapshot, xfs_freeze, as needed)

17. Docker volumes
17
Docker helpfully lets you bind mount to host
Giving you a choice of ways to get data to the host
Containers can remain ephemeral
However, you need to manage those underlying volumes
Note: you shouldn’t need to do what I did. Use something off the shelf if
you can. If you must, there is an excellent docker plugin api and volume
plugin api.

18. Solving local disk with docker
client cluster sched. docker host storage
r
u
n
n
request app
find free slot
ask for data
provide data
Container fully running with data

19. Using “trickery”
client cluster sched. docker host storage
r
u
n
n
request app
find free slot
request data
provide data,
bind mount
container starts,
asks for dynamic
bind mount,
waits

20. With docker volume plugin api
client cluster sched. docker host storage
r
u
n
n
request app
find free slot
json
provide data
docker calls
volume plugin
BEFORE
container starts,
launches with
bind mount

21. However: Docker plugin api did not exist yet!
21
I had to make do with “trickery”
Other choices like powerstrip existed, but wanted “standard” docker
And you are here for namespace trickery
So lets learn from it…

22. —Unknown
“Hard work pays off
eventually, but laziness pays
off right now.”
22

23. Namespaces - really quick…
23
Along with cgroups are “foundational tech” for containers
6 types: Mount, UTS, IPC, PID, Network and User
My favourites:
Mount: filesystem stuff (that I used)
PID, Network and the exciting User namespaces!
https://lwn.net/Articles/531114/

24. How do we access these namespaces?
24
nsenter - command line tool
nsenter allows you to “enter” a namespace and do something in the
context of it
Available out of the box in many linux distros now
https://github.com/karelzak/util-linux/blob/master/sys-utils/nsenter.c
https://blog.docker.com/tag/nsenter/

25. Mounts and Volumes
It’s all files in Linux - part 1

26. Mount namespace
26
Containers don’t see all mount points, all devices, just their own
Allows dockers “bind mount” to work
A “bind mount” in linux is really an “alternative view of an existing
directory tree”
A docker bind mount takes that “alternative view” and makes it visible
to the container (via its mount name space)
Magic? No. Linux.

27. It’s all files, part 1
27
Start any container
Access docker host and run this to get the pid of the whole container:
docker inspect --format {{.State.Pid}} <container id>
You can then see the 6 namespaces in /proc/<PID>/ns:
ls /proc/7865/ns/
ipc mnt net pid user uts

28. /proc virtual filesystem and nsenter
28
/proc is a virtual filesystem (http://www.tldp.org/LDP/Linux-
Filesystem-Hierarchy/html/proc.html)
Run a command inside a given containers namespace:
nsenter --mount=/proc/$PID/ns/mnt --
/usr/bin/command param
RUN A COMMAND FROM HOST AS IF YOU ARE IN THAT CONTAINER

29. —SpidermansUncle
“With great nsenter power,
comes great responsibility ”
29

30. Creating a bind mount on a running container
30
( -v /var/foo:/var/bar ) High level steps:
Get the underlying device from the host, into the container
mount the device in the container
bind mount in the container to the “directory you want”
unmount the device in container
remove the initial mount
What you are left with: a bind mount to the volume on the host you
wanted in the first place, and only that path. Not the whole device/volume
on host.

31. You don’t need to do all this yourself, ever!
31

32. 32
# Using a device’s numbers we can create the same device in container
# use nsenter to create a device file IN the container (using its $PID):
nsenter --mount=/proc/$PID/ns/mnt -- mknod --mode 0600 /dev/sda1 b 8
0
# Now we have the device ALSO in the container!
# We can mount it (normal linux)
# bind mount to the desired directory (also normal linux)!
# all from the host

33. I told you not to panic!
33

34. Now we have a dynamic bind mount
34
As if we used -v /var/foo:/var/bar on startup
Remember: DON’T DO THIS!
Really: you shouldn’t need to do this yourself.
Use the docker plugin volume api! (if you must)

35. Docker plugin API
35
Out of process JSON based api (but running on same host)
plugins are installed by putting a file in a directory, and referred by
name (minutes the extension)
Well defined JSON protocol
https://docs.docker.com/extend/plugin_api/

36. Docker volume plugin API
36
docker run -v volumename:/data --volume-driver=mydriver ..
“volumename” is passed to the registered volume-driver
(which is listening on http)
volume-driver then prepares the data somewhere on the host, returns
where it lives (via json)… docker then bind mounts it in as /data
All happens BEFORE container starts
https://docs.docker.com/extend/plugins_volume/

37. Docker volume plugin API
37
Would not require messing with namespaces
Still allow an out of process “volume service” to take care of messy
volume details
However - DOES require you to register the plugin with docker on the
host
And less terrifying fun than nsenter and namespaces

38. If you really must
38
https://github.com/michaelneale/bind-mount-supercontainer
Sample python code that I prototyped this with. Use with care!

39. Supercontainers
and storage engines
Like containers, only more… uh super…

40. Supercontainers - concept
40
Term came from Red Hat
http://developerblog.redhat.com/2014/11/06/introducing-a-super-
privileged-container-concept/
You have heard of privileged containers?
docker run --privileged ..
Drops all namespace restrictions
“Super privileged containers” add in more access to the underlying
host…

41. It’s all files (part 2)
41
Add in the host root filesystem, docker daemon, and all the rest:
docker run -v /var/run/docker.sock:/var/run/docker.sock
—privileged
-v /:/media/host
my-super-container
Brings in docker socket, and root as /media/host
/media/host then contains ALL devices, virtual files, /proc etc

42. It’s all files (part 2)
42
Why?
We can do everything we did with nsenter before but from WITHIN a
“peer container”

43. 43

44. It’s all files (part 2)
44
We can do everything we did with nsenter before but from WITHIN a
“peer container”
Remember requirements: vanilla docker, only docker installed on host
Use super-container as a “agent” container, do all the automation you
could want
No need for extra bits on the host box
Allows using “off the shelf” cluster scheduling (only docker need be
installed)

45. Controlling the host
45
Host can be accessed from super-container via nsenter
PID of host is 1!
eg, from super-container, get all mounts:
nsenter --mount=/media/host/proc/1/ns/mnt -- cat
/proc/mounts
Run a command, from container, on the host (stuff after “--")
/media/host lets us get to the host. Even devices.

46. Controlling the host
46
Host can be accessed from super-container via nsenter
Do all the steps as before, but with “nsenter —
mount=/media/host/proc/1/ns/mnt” prefixed

47. Controlling peer containers from supercontainer
47
Peers are other “ordinary” containers on the same host as the super
container
Peers can be accessed from super-container also via nsenter
Just like before, we use nsenter, with the peer containers $PID
But prefix it with the hosts filesystem:
nsenter --mount=/proc/$PID/ns/mnt -- ..
becomes:
nsenter --mount=/media/host/proc/$PID/ns/mnt -- ..

48. Controlling peer containers
48
Why?
Once again, use he super-container as the controlling agent on a host
Less bits to install on the host

49. Storage engines
49
My requirement: multiple implementations for different clouds
Different clouds have different storage engines
Super container great place to host volume service
Different implementations on service depending on what is on offer
EBS, NFS, openstack rsync and more
This “volume service super-container” is responsible for
backup/restore

50. Storage engines - eg an AWS region
50
zone-1 zone-2
serverA serverBserverA serverB
vol-1 vol-2
vol-1vol-1 vol-1vol-2
snapshots
request backup

51. Snapshots/backups
51
Snapshots a cheap and quick
Zone resilience
Volumes (ie: disks) are not as durable as snapshots/backups
Similar in other platforms: GCP, OpenStack, Azure.
Google compute persistent disks: does allow volumes read-only extra
mounts across instances for redundancy of compute nodes
In our case: failing over is “restoring from backup” - always test your
backups!

52. Supercontainers - summary
52
A useful tool for low level control
No need to install bits on the host
Can control peers directly
Could be a great place to host a docker volume plugin implementation
(not currently recommended in Docker plugin api docs)

53. Stateful clusters
Everyone wants to be stateless…

54. What we built…
.. an elastic and scalable Jenkins based product for
multiple cloud environments, on docker

55. Cluster schedulers/managers
55
Remember: I have build schedulers before, would rather not again
Docker Swarm, Mesos/Marathon, Kubernetes etc
Some have concepts of volumes
All can schedule “plain” docker containers
Super containers can give you a way to get lower level access

56. What we settled on
56
Super containers to implement volume service
Support for multiple storage engines for different clouds
Scheduled via mesos+marathon
Only docker (+ mesos in this case) required on the hosts
Why mesos: practical choice for us but not a tight coupling
(could mesos be in a super container? probably)
Using containers for all the things: elastic search nodes, builds, even
haproxy
For us, 5 minute or event based backups/snapshots are fine

57. Running supercontainers
57
Eg. marathon: schedule a super container to run on each host
Constraint on volume service: one per host, size: number of servers in
cluster (3 in this case):
vol service vol servicevol service
master master
elastic search
haproxy
(free)

58. Working with EBS (an example)
client container volume service EBS api
requests backup
freeze for snapshot
initiate snapshot
unfreeze backup delta,
copy to s3
optimisation: use LVM snapshot instead of freeze

59. Backups, backups
59
Servers are ephemeral
Servers come and go
Disks are fallible (even if cloud platforms call them “volumes”)
Workload moves around
Restore data when workload is moved to a new location
Delta backups are used to avoid full copies each time

60. Cluster schedulers/managers
60
Storage awareness is being built in increasingly
(Kubernetes volumes, mesos storage awareness)
Ideal world: your cluster manager will do all this for you.
If you live in that world: congrats.
Make yourself a cocktail:

61. My recipe for no-sugar old fashioned:
https://gist.github.com/michaelneale/60341
45
61

62. “off the shelf” stateful volume tools
62
Rexray: use volume plugin api for Amazon EBS, Rackspace and more
Flocker from ClusterHQ
Kubernetes volume support
Apache “Mysos”: MySQL service backed up to HDFS on mesos
Tutum from Docker! has support for persistent volumes
Watch this space… (changing constantly)
https://docs.clusterhq.com/en/1.4.0/labs/docker-plugin.html
https://github.com/emccode/rexray

63. Stateful volumes summary
63
It is possible with docker
Avoid doing it yourself is someone else already has
Using local filesystem directly does feel a bit like “legacy”
But it is a reality for some apps (especially database services)
Lovely to port everything to be stateless, database backed, blobstore
backed, but it takes time
Lean on the capabilities of the underlying platform where you can

64. Credits
64
Jérôme Petazzoni (@jpetazzo) - years of inspirational blog posts, hacks on
linux/docker/volumes. And great hair.
http://jpetazzo.github.io/2015/01/13/docker-mount-dynamic-volumes/ -
BTW Jerome - it works for real!
Red Hat for Super Container concepts: Daniel Walsh:
http://developerblog.redhat.com/2014/11/06/introducing-a-super-privileged-
container-concept/
Trevor Jay from Red Hat for some final namespace tips
https://securityblog.redhat.com/author/tjay/
I really just mashed up the above concepts:
https://michaelneale.blogspot.com.au/2015/02/mounting-devices-host-
from-super.html

65. @jpetazzo’s hair - imminent singularity?
0
45
90
135
180
225
2012 2013 2014 2015
Region 1
65

66. Thank you!
Michael Neale
@michaelneale
mneale@cloudbees.com