Taking Control of Chaos with Docker and Puppet

20,081 views

Published on

"Taking Control of Chaos (with Docker and Puppet)" by Tomas Doran (@bobtfish) of Yelp at Puppet Camp London 2014. Find the video here: http://puppetlabs.com/community/puppet-camp

Published in: Technology
0 Comments
25 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
20,081
On SlideShare
0
From Embeds
0
Number of Embeds
122
Actions
Shares
0
Downloads
139
Comments
0
Likes
25
Embeds 0
No embeds

No notes for slide

Taking Control of Chaos with Docker and Puppet

  1. 1. Taking control of chaos (with Docker and puppet) - Tomas Doran @bobtfish 18/11/2013
  2. 2. Docker - why should you care? • Isolation – Run each daemon in it’s own container • Security – Containers are fully independent – Build a new container for a new application version! • Ease of development – Same containers in Vagrant on laptop as in production! – Build pipeline for whole application environment • Ease of deployment – Deploy from your own private registry – Roll back by just deploying the last version
  3. 3. Taking control of chaos • My personal server • Lots of responsibilities – – – – – – Postfix Dovecot DNS Irc + tmux tor A zillion web apps (ruby, perl, php, python, nodejs!) • 0 time spent on it
  4. 4. My personal server
  5. 5. Several years later…. Label for your image (if needed) $864
  6. 6. Reality Label for your image (if needed) $864
  7. 7. This year’s model Label for your image (if needed) $864
  8. 8. Half the answer $864
  9. 9. Convergence and immutability • Exactly one run – Should be building machines clean every time – Doing exactly one puppet run • Always rebuild – Unless you rebuild regularly – You don’t know you can rebuild • Immutable instances – Never change config on a server, replace instance! • Hard if you only have 1 server!
  10. 10. The other half $864
  11. 11. Compartmentalization $864
  12. 12. Every service? $864
  13. 13. Yes $864 Postfix
  14. 14. Really $864 Postfix Dovecot
  15. 15. Every $864 Postfix Dovecot Postgrey
  16. 16. Single $864 Postfix Dovecot Postgrey Spamassasin
  17. 17. Service $864 Postfix Dovecot Postgrey Spamassasin Mysql
  18. 18. Independently! $864 Postfix Dovecot Postgrey Spamassasin Mysql irssi
  19. 19. For real. $864 Postfix Dovecot Postgrey Spamassasin Mysql irssi playground (sshd)
  20. 20. A cunning plan • Build puppet code for installing service on the old server • Run same puppet code inside a container to install packages / build config • Add shims to start service inside container
  21. 21. Data management • All mutable data is an lvm volume mounted from the host • All lvm volumes also get bind mounted read only • Share unix domain sockets this way • Server for socket creates • Clients mount ro version
  22. 22. Compartmentalization $864
  23. 23. Containers and volumes
  24. 24. What’s inside a container? supervisord: • The ‘real’ process • mcollective • sshd
  25. 25. Code structure • profile::dovecot – All the things needed to run dovecot – Parameterizeable as needed for two different deploy environments. • container::dovecot – Main entry point when building the container – Delegates most of real setup to profile::dovecot – Adds all the container specific overrides – Adds supervisor service(s) for this container • run_container::dovecot – Wraps docker::run {} – Manages the associated lvm volumes – Adds firewall rules
  26. 26. profile::dovecot ! – All the things needed to run dovecot – Parameterizeable as needed for two different deploy environments.
  27. 27. container::dovecot ! – Main entry point when building the container – Delegates most of real setup to profile::dovecot – Adds all the container specific overrides – Adds supervisor service(s) for this container
  28. 28. run_container::dovecot ! – Wraps docker::run {} – Manages the associated lvm volumes – Adds firewall rules
  29. 29. Building containers • profile::docker::build_container – define - writes out Dockerfile + support files – Runs docker build . – Manages dependencies so base containers get built first • profile::docker::with_socket::mysql – /socket/mysql – Symlink into /var/lib/mysql • profile::docker::with_supervisord – Setup supervisor with default (mcollective + ssh) tasks – Default /start script to invoke supervisord
  30. 30. profile::docker::build_container ! – define - writes out Dockerfile + support files – Runs docker build . – Manages dependencies so base containers get built first
  31. 31. profile::docker::build_container ! – define - writes out Dockerfile + support files – Runs docker build . – Manages dependencies so base containers get built first
  32. 32. profile::docker::build_container ! – define - writes out Dockerfile + support files – Runs docker build . – Manages dependencies so base containers get built first
  33. 33. container::with_socket::mysql – /socket/mysql – Symlink into /var/lib/mysql
  34. 34. container::with_supervisord ! – Setup supervisor with default (mcollective + ssh) tasks – Default /start script to invoke supervisord
  35. 35. The Dockerfile • Drop facts – /etc/facter/facts.d/is_container.txt – /etc/facter/facts.d/container_name • Copy in code – ADD support/puppet /etc/puppet • Run puppet – Masterless – No real manifest, just: if $::is_container {..
  36. 36. Drop facts – /etc/facter/facts.d/is_container.txt ! ! ! !
  37. 37. Drop facts – /etc/facter/facts.d/is_container.txt ! ! ! ! ! ! – /etc/facter/facts.d/container_name.txt
  38. 38. Puppet code – ADD support/puppet /etc/puppet – RUN bundle exec rake puppet
  39. 39. site.pp – Masterless – No node manifest, just: if $::is_container {..
  40. 40. Issues • Docker is kinda buggy – Just went 1.0, being fixed fast! – No sane exit status to docker build – AUFS 42 layer limit • Forge modules + packages assume service management – No upstart inside containers - fails everywhere • Debian packages are inconsistent – Lots of packages don’t use invoke-rc.d
  41. 41. It’s still awesome $864
  42. 42. Loads of TODOs • HAProxy all the things! – Currently just bind containers to local ports – Container replacement is not invisible – Run haproxy on the real host – Dynamically regenerate its config based on running containers • Registry – Build containers in Vagrant, push up to prod • PAAS web stuff – www all still lives on old server :( – Gonna fix this real soon
  43. 43. Open sores! • Open source all the things! – garethr++ – (I owe you patches) – Forked and changed a million modules – Will cleanup and upstream some changes • My profile::docker code – Don’t know how to make this really generically reusable – We need sub-modules? – Happy to share chunks
  44. 44. Sorry!
  45. 45. Questions? http://www.yelp.com/careers?jvi=ogVTXfwL $864 http://containercats.tumblr.com/

×