0
Continuous Delivery
with Docker
Tobias Schwab
Myself
• Tobias Schwab
• tobias.schwab@dynport.de
• www.dynport.de
• twitter.com/tobstarr
• github.com/tobstarr
Philosophie
• continuous delivery: deploy multiple times a day
• canary releases
• “never touch a running system”
• “Immut...
Theory
• AWS
• AMI based deployments
• Elastic Load Balancer
• AutoScaling Groups
• S3, RDS, …
Reality
• privacy concerns: AWS not an option
• hoster we could not pick
• first no, then proprietary and unreliable API
• ...
Docker
• build, distribute and deploy container based
applications
• creator: dotcloud
• initial release: March 13, 2013
•...
Container Virtualization
• os level
• shared kernel
• cgroups: isolate CPU, Memory, Block IO, Network
• lxc: cgroups + app...
Images
• blueprints for containers
• tarball of os installation/packages
• read only
• stateless
• layered
Containers
• instances of images
• copy on write / union file system
• running or exited
• goal: stateless and immutable
• ...
Containers and images
Source: http://docs.docker.io/en/latest/terms/container/
Demo
Build
• manual
• start and attach container
• install required packages
• checkout application code
• run build management...
Build
• chef/puppet/…
• start an attach container
• run chef/puppet/… client
• good: automated and documented
• bad: does ...
Dockerfile
• simple, plain text script to create images
• commands:
• FROM: base image to use
• RUN: execute shell command
...
Dockerfile
Dockerfile
Caching
• statement based: each step creates a new image
• existing steps (command tree exists) are re-used
• tricky: “non...
Caching
Configuration Management
• “store config in the environment” (http://12factor.net/config)
• dependency injected with start of...
Don’ts
• full blown VMs
• ssh daemon inside containers
• syslog daemon inside containers (sometimes
needed)
• user managem...
Build Management Tools
• candidates: bundler, pip, mvn, carton, composer, …
• problem with caching: bmt are slow when star...
• Problems
• unicorn: Rack HTTP server for fast clients
• static assets
• logging: default ruby syslog library uses syscal...
Multi-Host
• image distribution via docker registry
• weighted load balancing via HAProxy
• SSL termination via nginx in f...
Registry
• push and pull images
• public
• private
• backends: local, S3, Elliptics, Google Cloud
Storage, hosted
Load Balancing
• HAProxy
• license: GPL v2
• pool configuration stored in redis/etcd
• config update
• compile config files fr...
HAProxy
HAProxy
Deployment Pipeline
• commit triggers new image build
• build suite executed with image
• image is pushed to registry if t...
Deployment Pipeline
Nginx
HAProxy
Nginx
HAProxy
Docker
Container
Container
Container
Docker
Container
Container
Container
...
Logging
• host: docker host, container_id
• code: image_id, revision
• request: request_id, action, status_code, etag, tim...
Metrics
• OpenTSDB
• “distributed, scalable Time Series Database”
• license: LGPLv2.1+
• HBase
• Tags / Dimensions
• from ...
OpenTSDB
Metrics
Metrics
request counts by revision
Metrics
Metrics
Metrics
Docker reduces
• external dependencies (“rubygems/github slow/unreliable/down”)
after image is built
• “did work on my mac...
VS. AWS
• HAProxy much more flexible
• multiple containers per host
• balancing weights
• faster build process
• faster dep...
Resources
• docker.io
• opentsdb.net
• haproxy.1wt.eu
• continuousdelivery.com
• chadfowler.com/blog/2013/06/23/immutable-...
Questions?!?
Thank you!
Upcoming SlideShare
Loading in...5
×

OSDC 2014: Tobias Schwab - Continuous Delivery with Docker

797

Published on

Docker took the ops world by storm in 2013. Based on the same technology that powers Heroku (container virtualization) docker makes it easy to create private and data center agnostic PAAS architectures.

Container images created with docker contain the full application stack and enable rapid deployments and fast auto-scaling without any external dependencies at deploy time. They allow running the exact same configuration of OS, package dependencies, application code and configuration files in all environments and on all servers.

In this talk I want to present how we implement continuous delivery of a Ruby on Rails Application (1414.de) using docker. I will give a short introduction to docker and talk about best practices for production usage. Other topics which will be covered in the docker context are:

- image distribution with private registries
- multi docker host orchestration
- configuration management
- logging and metrics
- load balancing and failover

Published in: Software, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
797
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
29
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "OSDC 2014: Tobias Schwab - Continuous Delivery with Docker "

  1. 1. Continuous Delivery with Docker Tobias Schwab
  2. 2. Myself • Tobias Schwab • tobias.schwab@dynport.de • www.dynport.de • twitter.com/tobstarr • github.com/tobstarr
  3. 3. Philosophie • continuous delivery: deploy multiple times a day • canary releases • “never touch a running system” • “Immutable Infrastructure and Disposable Components" • don’t fix it, if it can be replaced
  4. 4. Theory • AWS • AMI based deployments • Elastic Load Balancer • AutoScaling Groups • S3, RDS, …
  5. 5. Reality • privacy concerns: AWS not an option • hoster we could not pick • first no, then proprietary and unreliable API • flash based infrastructure management • limited capacity • we were the biggest customer
  6. 6. Docker • build, distribute and deploy container based applications • creator: dotcloud • initial release: March 13, 2013 • license: Apache 2.0 • 11k stars on Github (top 50) • golang client/server
  7. 7. Container Virtualization • os level • shared kernel • cgroups: isolate CPU, Memory, Block IO, Network • lxc: cgroups + application namespaces • lightweight and fast
  8. 8. Images • blueprints for containers • tarball of os installation/packages • read only • stateless • layered
  9. 9. Containers • instances of images • copy on write / union file system • running or exited • goal: stateless and immutable • can be “saved” (docker commit) as images • created to be thrown away
  10. 10. Containers and images Source: http://docs.docker.io/en/latest/terms/container/
  11. 11. Demo
  12. 12. Build • manual • start and attach container • install required packages • checkout application code • run build management tool • bad: not reproducible • bad: does not utilise caching
  13. 13. Build • chef/puppet/… • start an attach container • run chef/puppet/… client • good: automated and documented • bad: does not utilise caching
  14. 14. Dockerfile • simple, plain text script to create images • commands: • FROM: base image to use • RUN: execute shell command • ENV: set environment variable • ADD: write local file to image • ENTRYPOINT: start command for containers • others: MAINTAINER, EXPOSE, CMD, USER, VOLUME, WORKDIR, ONBUILD
  15. 15. Dockerfile
  16. 16. Dockerfile
  17. 17. Caching • statement based: each step creates a new image • existing steps (command tree exists) are re-used • tricky: “non functional” commands (e.g. apt-get update/upgrade) • use ENV or comments to break caching of non functional commands
  18. 18. Caching
  19. 19. Configuration Management • “store config in the environment” (http://12factor.net/config) • dependency injected with start of container • same image for • development • testing • staging • production
  20. 20. Don’ts • full blown VMs • ssh daemon inside containers • syslog daemon inside containers (sometimes needed) • user management: everything can run as root • chef/puppet/… => makes caching useless
  21. 21. Build Management Tools • candidates: bundler, pip, mvn, carton, composer, … • problem with caching: bmt are slow when started with “clean slate” • option 1: add bmt manifest before code • bmt needs to run only when manifest changes • option 2: use pre-bundled base images • bmt only needs to work the delta • re-build base images from time to time • option 3: combine option 1 and option 2
  22. 22. • Problems • unicorn: Rack HTTP server for fast clients • static assets • logging: default ruby syslog library uses syscall (needs local syslog daemon) • Solution • run 3 daemons in 1 container: unicorn, nginx and rsyslogd • upstart • ENTRYPOINT [“/sbin/init”] • load ENV from /proc/1/environ • foreman Use Case: Ruby on Rails
  23. 23. Multi-Host • image distribution via docker registry • weighted load balancing via HAProxy • SSL termination via nginx in front of HAProxy
  24. 24. Registry • push and pull images • public • private • backends: local, S3, Elliptics, Google Cloud Storage, hosted
  25. 25. Load Balancing • HAProxy • license: GPL v2 • pool configuration stored in redis/etcd • config update • compile config files from stored configuration • upload via ssh • verify on remote hosts • replace current config with verified one • reload
  26. 26. HAProxy
  27. 27. HAProxy
  28. 28. Deployment Pipeline • commit triggers new image build • build suite executed with image • image is pushed to registry if tests passed • optional: start image with staging ENV settings for manual testing • start image with production ENV for last pre-flight tests • deploy image to more hosts • update load balancer (canary or green/blue) • monitor new containers/image
  29. 29. Deployment Pipeline Nginx HAProxy Nginx HAProxy Docker Container Container Container Docker Container Container Container Docker Container Container Container Docker Container Container Container Docker Registry Docker Build 2 push 3 pull + run 1 build 4 update4 update Route 53
  30. 30. Logging • host: docker host, container_id • code: image_id, revision • request: request_id, action, status_code, etag, times, calls • NOT inside containers • remote syslog (when possible) • alternative: local syslog relay inside container
  31. 31. Metrics • OpenTSDB • “distributed, scalable Time Series Database” • license: LGPLv2.1+ • HBase • Tags / Dimensions • from syslog via udp (StatsD “like”) • rickshaw.js for graphs • compare status codes, counts and times between actions of two revisions
  32. 32. OpenTSDB
  33. 33. Metrics
  34. 34. Metrics request counts by revision
  35. 35. Metrics
  36. 36. Metrics
  37. 37. Metrics
  38. 38. Docker reduces • external dependencies (“rubygems/github slow/unreliable/down”) after image is built • “did work on my machine/staging”: same OS package versions, configuration and code in all stages • unused CPU cycles • number of hosts • feedback times • time to get new host online • bottlenecks: hosts are more flexible
  39. 39. VS. AWS • HAProxy much more flexible • multiple containers per host • balancing weights • faster build process • faster deployments • instance flexibility
  40. 40. Resources • docker.io • opentsdb.net • haproxy.1wt.eu • continuousdelivery.com • chadfowler.com/blog/2013/06/23/immutable- deployments/ • 12factor.net
  41. 41. Questions?!?
  42. 42. Thank you!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×