Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Tale of a Docker-based Continuous Delivery Pipeline by Rafe Colton (ModCloth)

9,994 views

Published on

The ModCloth Platform team has been building a Docker-based continuous delivery pipeline. This presentation discusses that project and how we build containers at ModCloth. The topics include what goes into our containers; how to optimize builds to use the Docker build cache effectively; useful development workflows (including using fig); and the key decision to treat containers as processes instead of mini-vms. This presentation will also discuss (and demo!) the workflow we’ve adopted for building containers and how we’ve integrated container builds with our CI.

Published in: Technology

The Tale of a Docker-based Continuous Delivery Pipeline by Rafe Colton (ModCloth)

  1. 1. doing the old thing the new way by @rafecolton
  2. 2. brief prologue “rafe” (rafecolton on the internets) software engineer, platform @ modcloth using docker in prod since v0.7.0 *todo: explain “doing the old thing the new way”
  3. 3. obligatory slide with a bunch of logos
  4. 4. the stack that was
  5. 5. the stack that was
  6. 6. the stack that was
  7. 7. the stack that would be
  8. 8. motivations • simplify application architecture • support a variety of application languages • make provisioning and deployment more accessible
  9. 9. motivations • simplify application architecture • support a variety of application languages • make provisioning and deployment more accessible goals • push-button provisioning and deployment • consolidated, pluggable platform • move to linux
  10. 10. motivations • simplify application architecture • support a variety of application languages • make provisioning and deployment more accessible goals • push-button provisioning and deployment • consolidated, pluggable platform • move to linux bonus points • chatops • actual button for provisioning and deployment
  11. 11. motivations goals bonus points • chatops • actual button for provisioning and deployment does docker facilitate such a solution? • simplify application architecture • support a variety of application languages • make provisioning and deployment more accessible • push-button provisioning and deployment • consolidated, pluggable platform • move to linux
  12. 12. the stack that would be
  13. 13. case study: modcloth.com/style-gallery Clear and Simple Statement.
  14. 14. case study: modcloth.com/style-gallery
  15. 15. case study: modcloth.com/style-gallery
  16. 16. case study: modcloth.com/style-gallery webserver nginx ruby smartos rails
  17. 17. case study: modcloth.com/style-gallery webserver sidekiq workers nginx ruby smartos rails cron ruby smartos rails
  18. 18. case study: modcloth.com/style-gallery webserver nginx ruby smartos rails sidekiq workers cron ruby smartos rails how complex could it be?
  19. 19. case study: modcloth.com/style-gallery webserver rails ruby nginx docker ubuntu
  20. 20. case study: modcloth.com/style-gallery webserver rails ruby nginx docker ubuntu cron
  21. 21. case study: modcloth.com/style-gallery webserver rails ruby nginx docker ubuntu cron supervisord
  22. 22. case study: modcloth.com/style-gallery webserver rails ruby nginx docker ubuntu cron supervisord sidekiq workers rails
  23. 23. case study: modcloth.com/style-gallery webserver rails ruby nginx cron supervisord docker ubuntu sidekiq workers rails nad nodejs rsyslogd sshd
  24. 24. challenges • overall complexity • maintainability • image consistency • container reliability • log aggregation • monitoring
  25. 25. challenges • overall complexity • maintainability • image consistency • container reliability • log aggregation • monitoring lessons • don’t do the new thing the old way • consider division of responsibility
  26. 26. case study: modcloth.com/style-gallery webserver rails ruby rails docker cron ubuntu nginx sidekiq workers ruby nad nodejs rsyslogd
  27. 27. the stack
  28. 28. observations docker is an excellent packaging and distribution system
  29. 29. observations docker is an excellent packaging and distribution system containers are the canonical building block for a continuous delivery pipeline
  30. 30. begin github search… projects for orchestrating containers: • docker/fig • deis/deis • flynn/flynn • coreos/fleet • ansible/ansible • opscode/chef • progrium/dokku • newrelic/centurion
  31. 31. begin github search… • docker/fig • deis/deis • flynn/flynn • coreos/fleet • ansible/ansible • opscode/chef • progrium/dokku • newrelic/centurion • mesosphere/marathon • airbnb/chronos • GoogleCloudPlatform/kubernetes • openshift/geard • VoltFramework/volt projects for orchestrating containers:
  32. 32. • docker/fig • deis/deis • flynn/flynn • coreos/fleet • ansible/ansible • opscode/chef • progrium/dokku • newrelic/centurion • mesosphere/marathon • airbnb/chronos • GoogleCloudPlatform/kubernetes • openshift/geard • VoltFramework/volt • octohost/octohost • makeusabrew/decking • signalfuse/maestro-ng • shipyard/shipyard • DevTable/gantryd • mcuadros/dockership • longshoreman/longshoreman • marmelab/gaudi • etc. begin github search… projects for orchestrating containers:
  33. 33. begin github search… projects for building containers: • rafecolton/docker-builder • mitchellh/packer • swipely/dockly • ???
  34. 34. observation everybody is building containers differently.
  35. 35. observation everybody is building containers differently. how do we build production-ready containers?
  36. 36. writing a good Dockerfile lesson 0: getting started
  37. 37. writing a good Dockerfile lesson 0: getting started use a docker hub base
  38. 38. writing a good Dockerfile lesson 0: getting started set your env
  39. 39. writing a good Dockerfile lesson 1: order matters deps before bundling
  40. 40. writing a good Dockerfile lesson 1: order matters ADD only Gemfile* first
  41. 41. writing a good Dockerfile lesson 1: order matters `ADD .` as late as possible
  42. 42. writing a good Dockerfile lesson 2: optimize for size, repeatability combine RUN commands whenever possible
  43. 43. writing a good Dockerfile lesson 2: optimize for size, repeatability RUN dependent steps together
  44. 44. writing a good Dockerfile lesson 3: use a standard entrypoint use a *simple* entrypoint script
  45. 45. writing a good Dockerfile lesson 3: use a standard entrypoint operate on docker-specific environment variables
  46. 46. writing a good Dockerfile lesson 3: use a standard entrypoint wrap verbose CMD options
  47. 47. writing a good Dockerfile lesson 3: use a standard entrypoint exec "$@" # give yourself a shell
  48. 48. the image development lifecycle build *type things* push tag
  49. 49. the image development lifecycle push > docker build -t myapp:latest . > export latest="$(docker images | grep myapp:latest | head -n 1 | awk '{print $3}’)" > docker tag $latest "$(git rev-parse -q HEAD)" # sha > docker tag $latest "$(git describe --always --dirty --tags)" # tag > docker tag $latest "$(git rev-parse -q --abbrev-ref HEAD)" # branch > for image in $(docker images | grep myapp | awk '{print $1 ":" $2}' | head -n 4) ; build tag do docker push $image ; done *type things*
  50. 50. the image development lifecycle: docker-builder push build tag > docker-builder build . *type things*
  51. 51. teh pipeline docker build server* app app app app *https://github.com/rafecolton/docker-builder
  52. 52. conclusion building containers is like writing ruby code:
  53. 53. conclusion building containers is like writing ruby code: it’s easy to do it’s hard to do correctly
  54. 54. so what did we learn? (or, through what did you sleep?) stuff: • the complexity will come naturally (so don’t force it) • be intentional about your Dockerfile
  55. 55. so what did we learn? (or, through what did you sleep?) stuff: • the complexity will come naturally (so don’t force it) • be intentional about your Dockerfile • docker is an excellent packaging and distribution system • containers are the canonical building blocks
  56. 56. so what did we learn? (or, through what did you sleep?) stuff: • the complexity will come naturally (so don’t force it) • be intentional about your Dockerfile • docker is an excellent packaging and distribution system • containers are the canonical building blocks • consider division of responsibility between the host and the container • don’t do the new thing the old way (do the old thing the new way!)
  57. 57. brief epilogue goal: move to linux => all apps (less one) now employing docker/ansible/linux
  58. 58. brief epilogue goal: move to linux => all apps (less one) now employing docker/ansible/linux goal: consolidated, pluggable platform => shared monitoring, log aggregation, & load balancing services
  59. 59. brief epilogue goal: move to linux => all apps (less one) now employing docker/ansible/linux goal: consolidated, pluggable platform => shared monitoring, log aggregation, & load balancing services goal: push-button provisioning and deployment => it works, minimal magic… and it’s well documented
  60. 60. brief epilogue goal: move to linux => all apps (less one) now employing docker/ansible/linux goal: consolidated, pluggable platform => shared monitoring, log aggregation, & load balancing services goal: push-button provisioning and deployment => it works, minimal magic… and it’s well documented could easily be maintained by only two people… dun dun dun
  61. 61. brief epilogue I’m job hunting…
  62. 62. brief epilogue I’m job hunting… …and I haven’t shot anything yet. so if you’re hiring, come talk to me after the show. twitter: @rafecolton github: rafecolton rafecolton.com
  63. 63. thank you

×