1. Perspectives on Docker
10 things not to forget before
deploying Docker in production
Docker Meetup @ RightScale
Raphael Simon & Thorsten von Eicken
October 21st 2014

2. Docker from Theory to Production
How to cruise …
… and what to avoid

3. Containers vs. Virtual Machines
Differences:
● Size
● Boot time
● Performance
● Isolation
● Compatibility

4. Containers vs. Processes
textbook process
regs PGM MEM
proc
real process
regs MEM
proc
/etc
/lib
/bin
container
regs MEM
proc
/etc
/lib
/bin
net
⇒Containers are processes with env, not mini-VMs

5. Docker Benefits @RightScale
1. Dev & ops contract
Dev responsible for app
containers contents
Ops responsible for container
surrounds
Not against devops, but ops
handles >30 apps
2. App portability
Make it easier for our customers
to run an app anywhere
VMs need to be customized for
each cloud (or bare metal)
Rather than installing apps each
time, just drop down containers

6. Containers in a VM
Container 1
Runs app 1
Tenant A
Container 2
Runs app 2
Tenant A
VM 1
Tenant A
Host 1
Container 3
Runs app 3
Tenant B
Container 4
Runs app 4
Tenant B
VM 1
Tenant B
● Containers are produced by
development
● VMs are produced and
managed by ops
● Hosts are managed by the
cloud provider
Do not trust containers to provide
a hard security boundary

7. 10 Things not to Forget
before deploying Docker in production

8. 1. Logging – how Docker does it
Docker captures stdout/stderr
docker logs command prints combined stdout/err
docker logs -f : running tail (from the beginning!)
$ docker run --name hello -d busybox echo hello Santa Barbara
a3c0caa675e106cc0cf208dade762afcc341ed5b9ea8f3d75b6e2092745a5faa
$ docker logs hello
hello Santa Barbara
$

9. 1: Logging – how not to do it
● Log to stdout/stderr and collect them in the VM
○ Not all apps log to stdout/err, many don’t add timestamps
○ No log rotation (can use logrotate with copytruncate)
○ No tailing to ship the logs
● Run syslog daemon inside the container
○ Containers ≠ VMs
○ Configuration hell

10. 1: Logging – solutions
● Bind-mount /tmp/dev -> /dev
○ Can’t bind-mount /dev/log!
○ Move /dev/log to /tmp/dev/log
○ See http://jpetazzo.github.io/2014/08/24/syslog-docker/
● Fix docker daemon to handle logging
○ Fixing stdout/err is happening (#7195)
○ Ready to add support for syslog source, but not active
container docker
syslog
file
stdout/err json

11. 2: Monitoring – how not to do it
● Monitoring daemon inside
each container
○ Container ≠ VM
○ Monitoring daemons require privs
○ Configuration/management hell

12. Monitoring – how to do it
● Collect stats in VM using container-aware monitoring
○ Stats are in /sys/fs/cgroup/…
See: Docker doc article on run-time metrics
○ Docker support: cAdvisor, DataDog, … ?
● Or just monitor at the process level
$ docker run --name hello -d busybox sleep 60
3a804b088b432035c5cee541f4baef3cc728d27dded3378fd253c6b4abeb077a
$ cat /sys/fs/cgroup/cpuacct/docker/3a804b088b432035c5cee541f4ba
ef3cc728d27dded3378fd253c6b4abeb077a/cpuacct.usage_percpu
630924 4774818 7494614 3622216

13. 3. Secrets – how not to do it
DEMO

14. 3. Secrets – Solutions
Setup context prior to build:
$ cat Makefile
# Setup context then build image
build: Dockerfile
git clone git@github.com:rightscale/docker_demo
docker build -t raphael/demo .
rm -rf docker_demo
$ cat Dockerfile
FROM rightscale/ruby-212
ADD docker_demo /docker_demo

15. 3. Secrets – Take away
● Each Dockerfile ADD and RUN command results in a
new committed layer
● All image layers (built or pulled) are readily
accessible
● For now: Make sure to remove any unnecessary
credential from the context prior to building
● In the future: Take advantage of “nested builds”,
see #7115

16. 4. Container access
● Launch image manually with
custom command to troubleshoot
● Inspect files inside running container
● Launch shell into running container
using docker exec (new in 1.3)
$ docker exec -it hopeful_shockley /bin/sh
# ps -ax
PID TTY STAT TIME COMMAND
1 ? Ss+ 0:00 /bin/bash ← Main container process
43 ? S 0:00 /bin/sh
49 ? R+ 0:00 ps -ax

17. 5. Aufs vs. btrfs
● aufs corruption of container filesystems,
scope unknown, issue #7229
● btrfs seems to work better (default in CoreOS)
● btrfs “requires” separate partition
$ mkfs.btrfs /dev/xvdb
$ mount /dev/xvdb /mnt
$ mkdir -p /mnt/docker
$ ln -sf /mnt/docker /var/lib/docker
$ sed -i -e '/DOCKER_OPTS/s/.*/DOCKER_OPTS="-s=btrfs"/'
/etc/default/docker
$ restart docker

18. 6. Got Infinite disk space?
● Container logs grow indefinitely
○ Use logrotate with copytruncate
● Containers accumulate indefinitely
○ Becomes an issue if containers are frequently
restarted due to upgrades or crashes
○ Use docker run --rm
■ but then how do you troubleshoot?
○ Write script to docker rm old unused containers?

19. 7. Huge Containers – how not to do it
Overlays don’t go away
FROM ubuntu:14.04
RUN apt-get update
RUN apt-get install -y libjpeg
RUN apt-get install -y libjpeg-dev build-essential gcc 109 MB
ADD source /build 5 MB?
WORKDIR /build -
RUN ./configure 0 MB
RUN make 100 MB?
RUN make install
CMD /usr/local/bin/myexe

20. 7. Huge Containers – solutions
Use a tools container, share build results via volume
In the future: “nested builds” #7115, “squash” #4232 ?
FROM ubuntu:14.04
VOLUME /opt/app
ADD src /build
WORKDIR /build
RUN apt-get update
RUN apt-get install -y libjpeg-dev build-essential gcc
RUN ./configure
RUN make
RUN make install
RUN mkdir -p /opt/app
RUN cp -r /build/out/* /opt/app/

21. 8. Very slow container downloads
● Downloading docker images is very slow
● The problem isn’t bandwidth… see #7291
● Caching can help depending on use-case
Boot time steps Docker RightScript
Launch and boot 53s 49s
Prep VM environment 36s 16s
Install & launch zookeeper, redis, kafka,
mariadb, graphite, statsd 4m57s 1m5s
Install ruby n/a 54s
Install & launch custom apps 2m23s 3m3s
TOTAL 8m50s 6m8s

22. 9. Backups
Userguide: backup-restore-or-migrate-data-volumes
● Create DB container with /data volume
● Backup /data “anytime” from the VM
● Or launch 2nd backup container with --volumes-from
➣ Simple in a 1-off server, but how to automate in general?

23. 10. Docker Clusters
● Does Docker Cluster software solve all these issues?
● Kubernetes, Mesos, Fleet, …
○ apparently not (yet?)
● But, they require an overlay network…
Container 1
Runs app 1
172.16.4.3
VM 1
Container 2
Runs app 1
172.16.4.6
VM 2
10.0.0.1 10.0.0.2

24. Wrapping up
Why docker?
● dev-to-CI-to-prod workflow
● portability: same container in different VMs
Putting it into production:
● simple for one-off apps
● still WIP for system-wide deployment
Overall very promising and great to work with
Most pain points are actively being worked on

25. Perspectives on Docker
10 things not to forget before
deploying Docker in production
— the end —
Raphael Simon & Thorsten von Eicken