"Taking Control of Chaos (with Docker and Puppet)" by Tomas Doran (@bobtfish) of Yelp at Puppet Camp London 2014. Find the video here: http://puppetlabs.com/community/puppet-camp

2. Taking control of chaos
(with Docker and puppet)
- Tomas Doran
@bobtfish
18/11/2013

3. Docker - why should you care?
• Isolation
– Run each daemon in it’s own container
• Security
– Containers are fully independent
– Build a new container for a new application version!
• Ease of development
– Same containers in Vagrant on laptop as in production!
– Build pipeline for whole application environment
• Ease of deployment
– Deploy from your own private registry
– Roll back by just deploying the last version

4. Taking control of chaos
• My personal server
• Lots of responsibilities
–
–
–
–
–
–
Postfix
Dovecot
DNS
Irc + tmux
tor
A zillion web apps (ruby, perl, php, python,
nodejs!)
• 0 time spent on it

5. My personal server

6. Several years later….
Label for your image (if needed)
$864

7. Reality
Label for your image (if needed)
$864

8. This year’s model
Label for your image (if needed)
$864

9. Half the answer
$864

10. Convergence and immutability
• Exactly one run
– Should be building machines clean every time
– Doing exactly one puppet run
• Always rebuild
– Unless you rebuild regularly
– You don’t know you can rebuild
• Immutable instances
– Never change config on a server, replace instance!
• Hard if you only have 1 server!

11. The other half
$864

12. Compartmentalization
$864

13. Every service?
$864

14. Yes
$864
Postfix

15. Really
$864
Postfix
Dovecot

16. Every
$864
Postfix
Dovecot
Postgrey

17. Single
$864
Postfix
Dovecot
Postgrey Spamassasin

18. Service
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql

19. Independently!
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql
irssi

20. For real.
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql
irssi
playground
(sshd)

21. A cunning plan
• Build puppet code for installing
service on the old server
• Run same puppet code inside a
container to install packages / build
config
• Add shims to start service inside
container

22. Data management
• All mutable data is an lvm volume
mounted from the host
• All lvm volumes also get bind
mounted read only
• Share unix domain sockets this way
• Server for socket creates
• Clients mount ro version

23. Compartmentalization
$864

24. Containers and volumes

25. What’s inside a container?
supervisord:
• The ‘real’ process
• mcollective
• sshd

26. Code structure
• profile::dovecot
– All the things needed to run dovecot
– Parameterizeable as needed for two different deploy
environments.
• container::dovecot
– Main entry point when building the container
– Delegates most of real setup to profile::dovecot
– Adds all the container specific overrides
– Adds supervisor service(s) for this container
• run_container::dovecot
– Wraps docker::run {}
– Manages the associated lvm volumes
– Adds firewall rules

27. profile::dovecot
!
– All the things needed to run dovecot
– Parameterizeable as needed for two different deploy environments.

28. container::dovecot
!
– Main entry point when building the container
– Delegates most of real setup to profile::dovecot
– Adds all the container specific overrides
– Adds supervisor service(s) for this container

29. run_container::dovecot
!
– Wraps docker::run {}
– Manages the associated lvm volumes
– Adds firewall rules

30. Building containers
• profile::docker::build_container
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first
• profile::docker::with_socket::mysql
– /socket/mysql
– Symlink into /var/lib/mysql
• profile::docker::with_supervisord
– Setup supervisor with default (mcollective + ssh) tasks
– Default /start script to invoke supervisord

31. profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

32. profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

33. profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

34. container::with_socket::mysql
– /socket/mysql
– Symlink into /var/lib/mysql

35. container::with_supervisord
!
– Setup supervisor with default (mcollective + ssh) tasks
– Default /start script to invoke supervisord

36. The Dockerfile
• Drop facts
– /etc/facter/facts.d/is_container.txt
– /etc/facter/facts.d/container_name
• Copy in code
– ADD support/puppet /etc/puppet
• Run puppet
– Masterless
– No real manifest, just:
if $::is_container {..

37. Drop facts
– /etc/facter/facts.d/is_container.txt
!
!
!
!

38. Drop facts
– /etc/facter/facts.d/is_container.txt
!
!
!
!
!
!
– /etc/facter/facts.d/container_name.txt

39. Puppet code
– ADD support/puppet /etc/puppet
– RUN bundle exec rake puppet

40. site.pp
– Masterless
– No node manifest, just:
if $::is_container {..

41. Issues
• Docker is kinda buggy
– Just went 1.0, being fixed fast!
– No sane exit status to docker build
– AUFS 42 layer limit
• Forge modules + packages assume
service management
– No upstart inside containers - fails everywhere
• Debian packages are inconsistent
– Lots of packages don’t use invoke-rc.d

42. It’s still awesome
$864

43. Loads of TODOs
• HAProxy all the things!
– Currently just bind containers to local ports
– Container replacement is not invisible
– Run haproxy on the real host
– Dynamically regenerate its config based on running
containers
• Registry
– Build containers in Vagrant, push up to prod
• PAAS web stuff
– www all still lives on old server :(
– Gonna fix this real soon

44. Open sores!
• Open source all the things!
– garethr++
– (I owe you patches)
– Forked and changed a million modules
– Will cleanup and upstream some changes
• My profile::docker code
– Don’t know how to make this really generically
reusable
– We need sub-modules?
– Happy to share chunks

45. Sorry!

46. Questions?
http://www.yelp.com/careers?jvi=ogVTXfwL
$864
http://containercats.tumblr.com/