Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Perspectives on Docker


Published on

Slides from Docker Meetup Santa Barbara

Published in: Technology
  • Be the first to comment

Perspectives on Docker

  1. 1. Perspectives on Docker 10 things not to forget before deploying Docker in production Docker Meetup @ RightScale Raphael Simon & Thorsten von Eicken October 21st 2014
  2. 2. Docker from Theory to Production How to cruise … … and what to avoid
  3. 3. Containers vs. Virtual Machines Differences: ● Size ● Boot time ● Performance ● Isolation ● Compatibility
  4. 4. Containers vs. Processes textbook process regs PGM MEM proc real process regs MEM proc /etc /lib /bin container regs MEM proc /etc /lib /bin net ⇒Containers are processes with env, not mini-VMs
  5. 5. Docker Benefits @RightScale 1. Dev & ops contract Dev responsible for app containers contents Ops responsible for container surrounds Not against devops, but ops handles >30 apps 2. App portability Make it easier for our customers to run an app anywhere VMs need to be customized for each cloud (or bare metal) Rather than installing apps each time, just drop down containers
  6. 6. Containers in a VM Container 1 Runs app 1 Tenant A Container 2 Runs app 2 Tenant A VM 1 Tenant A Host 1 Container 3 Runs app 3 Tenant B Container 4 Runs app 4 Tenant B VM 1 Tenant B ● Containers are produced by development ● VMs are produced and managed by ops ● Hosts are managed by the cloud provider Do not trust containers to provide a hard security boundary
  7. 7. 10 Things not to Forget before deploying Docker in production
  8. 8. 1. Logging – how Docker does it Docker captures stdout/stderr docker logs command prints combined stdout/err docker logs -f : running tail (from the beginning!) $ docker run --name hello -d busybox echo hello Santa Barbara a3c0caa675e106cc0cf208dade762afcc341ed5b9ea8f3d75b6e2092745a5faa $ docker logs hello hello Santa Barbara $
  9. 9. 1: Logging – how not to do it ● Log to stdout/stderr and collect them in the VM ○ Not all apps log to stdout/err, many don’t add timestamps ○ No log rotation (can use logrotate with copytruncate) ○ No tailing to ship the logs ● Run syslog daemon inside the container ○ Containers ≠ VMs ○ Configuration hell
  10. 10. 1: Logging – solutions ● Bind-mount /tmp/dev -> /dev ○ Can’t bind-mount /dev/log! ○ Move /dev/log to /tmp/dev/log ○ See ● Fix docker daemon to handle logging ○ Fixing stdout/err is happening (#7195) ○ Ready to add support for syslog source, but not active container docker syslog file stdout/err json
  11. 11. 2: Monitoring – how not to do it ● Monitoring daemon inside each container ○ Container ≠ VM ○ Monitoring daemons require privs ○ Configuration/management hell
  12. 12. Monitoring – how to do it ● Collect stats in VM using container-aware monitoring ○ Stats are in /sys/fs/cgroup/… See: Docker doc article on run-time metrics ○ Docker support: cAdvisor, DataDog, … ? ● Or just monitor at the process level $ docker run --name hello -d busybox sleep 60 3a804b088b432035c5cee541f4baef3cc728d27dded3378fd253c6b4abeb077a $ cat /sys/fs/cgroup/cpuacct/docker/3a804b088b432035c5cee541f4ba ef3cc728d27dded3378fd253c6b4abeb077a/cpuacct.usage_percpu 630924 4774818 7494614 3622216
  13. 13. 3. Secrets – how not to do it DEMO
  14. 14. 3. Secrets – Solutions Setup context prior to build: $ cat Makefile # Setup context then build image build: Dockerfile git clone docker build -t raphael/demo . rm -rf docker_demo $ cat Dockerfile FROM rightscale/ruby-212 ADD docker_demo /docker_demo
  15. 15. 3. Secrets – Take away ● Each Dockerfile ADD and RUN command results in a new committed layer ● All image layers (built or pulled) are readily accessible ● For now: Make sure to remove any unnecessary credential from the context prior to building ● In the future: Take advantage of “nested builds”, see #7115
  16. 16. 4. Container access ● Launch image manually with custom command to troubleshoot ● Inspect files inside running container ● Launch shell into running container using docker exec (new in 1.3) $ docker exec -it hopeful_shockley /bin/sh # ps -ax PID TTY STAT TIME COMMAND 1 ? Ss+ 0:00 /bin/bash ← Main container process 43 ? S 0:00 /bin/sh 49 ? R+ 0:00 ps -ax
  17. 17. 5. Aufs vs. btrfs ● aufs corruption of container filesystems, scope unknown, issue #7229 ● btrfs seems to work better (default in CoreOS) ● btrfs “requires” separate partition $ mkfs.btrfs /dev/xvdb $ mount /dev/xvdb /mnt $ mkdir -p /mnt/docker $ ln -sf /mnt/docker /var/lib/docker $ sed -i -e '/DOCKER_OPTS/s/.*/DOCKER_OPTS="-s=btrfs"/' /etc/default/docker $ restart docker
  18. 18. 6. Got Infinite disk space? ● Container logs grow indefinitely ○ Use logrotate with copytruncate ● Containers accumulate indefinitely ○ Becomes an issue if containers are frequently restarted due to upgrades or crashes ○ Use docker run --rm ■ but then how do you troubleshoot? ○ Write script to docker rm old unused containers?
  19. 19. 7. Huge Containers – how not to do it Overlays don’t go away FROM ubuntu:14.04 RUN apt-get update RUN apt-get install -y libjpeg RUN apt-get install -y libjpeg-dev build-essential gcc 109 MB ADD source /build 5 MB? WORKDIR /build - RUN ./configure 0 MB RUN make 100 MB? RUN make install CMD /usr/local/bin/myexe
  20. 20. 7. Huge Containers – solutions Use a tools container, share build results via volume In the future: “nested builds” #7115, “squash” #4232 ? FROM ubuntu:14.04 VOLUME /opt/app ADD src /build WORKDIR /build RUN apt-get update RUN apt-get install -y libjpeg-dev build-essential gcc RUN ./configure RUN make RUN make install RUN mkdir -p /opt/app RUN cp -r /build/out/* /opt/app/
  21. 21. 8. Very slow container downloads ● Downloading docker images is very slow ● The problem isn’t bandwidth… see #7291 ● Caching can help depending on use-case Boot time steps Docker RightScript Launch and boot 53s 49s Prep VM environment 36s 16s Install & launch zookeeper, redis, kafka, mariadb, graphite, statsd 4m57s 1m5s Install ruby n/a 54s Install & launch custom apps 2m23s 3m3s TOTAL 8m50s 6m8s
  22. 22. 9. Backups Userguide: backup-restore-or-migrate-data-volumes ● Create DB container with /data volume ● Backup /data “anytime” from the VM ● Or launch 2nd backup container with --volumes-from ➣ Simple in a 1-off server, but how to automate in general?
  23. 23. 10. Docker Clusters ● Does Docker Cluster software solve all these issues? ● Kubernetes, Mesos, Fleet, … ○ apparently not (yet?) ● But, they require an overlay network… Container 1 Runs app 1 VM 1 Container 2 Runs app 1 VM 2
  24. 24. Wrapping up Why docker? ● dev-to-CI-to-prod workflow ● portability: same container in different VMs Putting it into production: ● simple for one-off apps ● still WIP for system-wide deployment Overall very promising and great to work with Most pain points are actively being worked on
  25. 25. Perspectives on Docker 10 things not to forget before deploying Docker in production — the end — Raphael Simon & Thorsten von Eicken