Docker has taken the world of software by storm, offering the promise of a portable way to build and ship software - including software running in the cloud. The RightScale development team has been diving into Docker for several projects, and we'll share our lessons learned on using Docker for our cloud-based applications.

1. REAL WORLD DOCKER: TEN THINGS WE’VE LEARNED

2. Speakers
•Thorsten von Eicken
•CTO
•Raphael Simon
•Senior Systems Architect

3. •Benefits of Docker
•Phases of Docker adoption
•Ten Things We’ve Learned
Agenda
2

4. Who is RightScale?
Self-Service
Cloud Analytics
RightScale Cloud Portfolio Management
Cloud Management
Design
Virtualized
Environments
Public
Clouds
Other
Services
Private Clouds
Automate
Multi-Cloud Orchestration
Operate
Deploy
Report
Optimize

5. POLLS
4

6. The Route to Docker
Exciting …
… and scary

7. Benefits of Docker at RightScale
1.Ability to run integration systems “all-in-one”
2.More robust deployments: same code, language runtime and library dependencies in dev, staging and production
3.Separation of concerns: devs produce Docker images, ops runs them
4.Easier deployment thanks to isolation: no need to worry about cross-dependencies
5.Smaller set of base VM images to maintain
6.Optimization of resource utilization
7....
It’s a spectrum ranging from quick wins to requiring fundamental changes
●Dev - “more per laptops”
●QA - more consistent builds
●Prod - more decoupling from dev

8. Adopting Docker in Phases
●Define steps with clear goals
●Start with quick wins
●Learn as we go
●Progress towards the “holy grail”:
oGeneric cluster of VMs used to dynamically deploy web services
●4 identified phases - currently deploying phase 2, starting dev on phase 3

9. Phase 1: Dev / Test
●Setup CI to build docker images
●Use containers to:
oRun unit tests
oRun “distributed” integration tests
●Benefits:
oGuaranteed clean environment with every run
oAbility to run full integration environments on one box
oOnly one Jenkins slave image / configuration

10. Phase 2: App Deploy
●Push docker images built in CI to production
●Minimal changes to VM images (need recent kernel + docker daemon)
●Benefits:
oCode shipped runs exactly as tested
oStreamlined deployments

11. Phase 3: “Generic” Virtual Machines
●Run “everything” in containers e.g. use fig to start services
●Dynamic app configuration
●Benefits:
oOne base image (Docker + utilities)
oOn-ramp to Phase 4

12. Phase 4: Automated “Pod” Deploys
●Automated container cluster management
●Scheduler to deploy pods of containers on the “right” host
oSwarm, Kubernetes, CoreOS Fleet, Mesos ... ?
●Benefits:
oFast auto-scaling / auto-healing
oResource utilization optimization
oCompletely streamlined continuous delivery

13. 10 Things We’ve Learned
How to cruise …
… and what to avoid

14. Docker captures stdout/stderr
docker logs command prints combined stdout/err
docker logs -tail 0 -f : running tail
1.Logging – how Docker does it
$ docker run --name hello -d busybox echo hello Santa Barbara
a3c0caa675e106cc0cf208dade762afcc341ed5b9ea8f3d75b6e2092745a5faa
$ docker logs hello
hello Santa Barbara
$

15. 1: Logging – how not to do it
●Log to stdout/stderr and collect them in the VM
oNot all apps log to stdout/err, many don’t add timestamps
oNo log rotation (can use logrotate with copytruncate)
oLimited tailing to ship the logs
●Run syslog daemon inside the container
oContainers ≠ VMs
oConfiguration hell

16. 1: Logging – solutions
●Bind-mount /tmp/dev -> /dev
oCan’t bind-mount /dev/log!
oMove /dev/log to /tmp/dev/log
oSee http://jpetazzo.github.io/2014/08/24/syslog-docker/
●Fix docker daemon to forward to syslog
oFixing stdout/err is happening (#7195)
oReady to add support for syslog source, but not active
container
docker
syslog
file
stdout/err
json

17. 2: Monitoring – how not to do it
●Monitoring daemon inside each container
oContainer ≠ VM
oMonitoring daemons require privs
oConfiguration/management hell

18. Monitoring – how to do it
●Collect stats in VM using container-aware monitoring
oStats are in /sys/fs/cgroup/… See: Docker doc article on run-time metrics
oDocker support: cAdvisor, DataDog, … ?
●Just monitor at the process level
$ docker run --name hello -d busybox sleep 60
3a804b088b432035c5cee541f4baef3cc728d27dded3378fd253c6b4abeb077a
$ cat /sys/fs/cgroup/cpuacct/docker/3a804b088b432035c5cee541f4ba ef3cc728d27dded3378fd253c6b4abeb077a/cpuacct.usage_percpu
630924 4774818 7494614 3622216

19. 3.Secrets – How not do it
Add then remove creds using Dockerfile statements
FROM rightscale/ruby-215
ADD github.key /root/.ssh/id_rsa # OOPS
RUN git clone git@github.com:rs/app
...
RUN rm /root/.ssh/id_rsa # DOES NOT HELP

20. 3.Secrets – Take away
●Each Dockerfile ADD and RUN command results in a new committed layer
●All image layers (built or pulled) are readily accessible
●For now: Make sure to remove any unnecessary credential from the context prior to building (see #8)
●In the future: Take advantage of “nested builds”, see Docker github issue #7115

21. 4.Container access
●Launch image manually with custom command to troubleshoot
●Inspect files inside running container
●Launch shell into running container using docker exec (new in 1.3)
$ docker exec -it hopeful_shockley /bin/sh
# ps -ax
PID TTY STAT TIME COMMAND
1 ? Ss+ 0:00 /usr/bin/ruby ← Main container process
43 ? S 0:00 /bin/sh ← Current shell
49 ? R+ 0:00 ps -ax

22. 5.Aufs vs. btrfs
●aufs corruption of container filesystems, issue #7229
oUbuntu use kernel 3.13.0-37 or newer
oRHEL 6.x/CentOS 6/... use kernel 2.6.32-504 or newer
●btrfs seems to work better (default in CoreOS)
●btrfs “requires” separate partition
$ mkfs.btrfs /dev/xvdb
$ mount /dev/xvdb /mnt
$ mkdir -p /mnt/docker
$ ln -sf /mnt/docker /var/lib/docker
$ sed -i -e '/DOCKER_OPTS/s/.*/DOCKER_OPTS="-s=btrfs"/' /etc/default/docker
$ restart docker

23. 6.Got Infinite disk space?
●Container logs grow indefinitely
oUse logrotate with copytruncate
●Containers accumulate indefinitely
oBecomes an issue if containers are frequently restarted due to upgrades or crashes
oUse docker run --rm
but then how do you troubleshoot?
oWrite script to docker rm old unused containers?

24. 7.Huge Containers – how not to do it
Overlays don’t go away
FROM ubuntu:14.04
RUN apt-get update
RUN apt-get install -y libjpeg
RUN apt-get install -y libjpeg-dev build-essential gcc 109 MB
ADD source /build 5 MB?
WORKDIR /build -
RUN ./configure 0 MB
RUN make 100 MB?
RUN make install
CMD /usr/local/bin/myexe

25. Use a tools container, share build results via volume
In the future: “nested builds” #7115, “squash” #4232 ?
FROM ubuntu:14.04
VOLUME /opt/app
ADD src /build
WORKDIR /build
RUN apt-get update
RUN apt-get install -y libjpeg-dev build-essential gcc
RUN ./configure
RUN make
RUN make install
RUN mkdir -p /opt/app
RUN cp -r /build/out/* /opt/app/
7.Huge Containers – solutions

26. 8.Building Images - The issues
Build typically requires access to private repos
➔Need to pull from them without leaving passwords/keys around (see issue #3)
Build typically requires env-specific tools
➔Need to run the tools in a controlled environment
◆(Example: import dependency analysis)

27. 8.Building Images - Strategy #1
public repos
private repos
build server
docker build
image
image repository
$ cat Makefile
build: Dockerfile
git clone git@github.com:rs/app
cd app && rm -rf ./.git
bundle install
docker build -t rightscale/app .
cd .. && rm -rf app
$ cat Dockerfile
FROM rightscale/ruby-215
ADD app /app
➔Pull code onto build box using creds; copy into container image from local FS
➔But: Image build is at the mercy of the build box env

28. 8.Building Images - Strategy #2
public repos
private repos
build server
image repository
➔Image build runs within appropriate environment
base
image
base image
docker run
docker commit
$ cat Makefile
build:
docker run
--volume $SRC:"/src"
--name "tmp_app"
$BASE_IMAGE
sh -c $BUILD_SCRIPT &&
docker commit tmp_app tmp
$ cat Dockerfile
FROM tmp
RUN "rainbows -c config.rb"

29. 9.Backups
Userguide: backup-restore-or-migrate-data-volumes
●Create DB container with /data volume
●Backup /data “anytime” from the VM
●Or launch 2nd backup container with --volumes-from
➣ Simple in a 1-off server, but how to automate in general?
… or don’t back containers up

30. 10.Docker Clusters
●Does Docker Cluster software solve all these issues?
●Swarm, Kubernetes, Mesos, Fleet, …
oapparently not (yet?)
●And, they require an overlay network…
VM 1
Container 1
Runs app 1
172.16.4.3
VM 2
Container 2
Runs app 1
172.16.4.6
10.0.0.1
10.0.0.2

31. Wrapping up
A phased approach:
●Start with highest impact / least disruption
●Typically around dev & continuous integration
Beware of issues and pitfalls
●Good news is that you’re not alone
●Most become real issues only at scale
●Use pragmatic work-arounds for the time being
Overall very promising and great to work with

32. Q&A
Read our blogs on Docker
www.rightscale.com/blog
Containers vs VMs? Combining Both for Cloud Portability Nirvana
eng.rightscale.com
Dockerizing RightScale

33. Containers vs. Virtual Machines
Differences:
●Size
●Boot time
●Performance
●Isolation
●Compatibility

34. Containers vs. Processes
MEM
PGM
regs
proc
textbook process
MEM
regs
proc
/etc
/lib
/bin
real process
MEM
regs
proc
/etc
/lib
/bin
container
net
⇒Containers are processes with env, not mini-VMs

35. Host 1
VM 1
Tenant A
Containers in a VM
Container 1
Runs app 1
Tenant A
Container 2
Runs app 2
Tenant A
VM 1
Tenant B
Container 3
Runs app 3
Tenant B
Container 4
Runs app 4
Tenant B
●Containers are produced by development
●VMs are produced and managed by ops
●Hosts are managed by the cloud provider
Do not trust containers to provide a hard security boundary

36. THANK YOU.