The document discusses using Docker containers and Puppet to compartmentalize services running on a personal server. Previously, many services like Postfix, Dovecot, DNS, etc. were running on the server with no isolation. The approach taken was to define Puppet profiles for each service, build Docker containers from those profiles, and run the containers independently with their own isolated environments. This improves security, ease of development and deployment, and allows immutable infrastructure by replacing containers instead of changing server configurations. Challenges discussed include Docker bugs, inconsistent Debian packages, and future plans to add HAProxy and a container registry.

2. Taking control of chaos
(with Docker and puppet)
- Tomas Doran
@bobtfish
18/11/2013

3. Docker - why should you care?
• Isolation
– Run each daemon in it’s own container
• Security
– Containers are fully independent
– Build a new container for a new application version!
• Ease of development
– Same containers in Vagrant on laptop as in production!
– Build pipeline for whole application environment
• Ease of deployment
– Deploy from your own private registry
– Roll back by just deploying the last version

4. Taking control of chaos
• My personal server
• Lots of responsibilities
–
–
–
–
–
–
Postfix
Dovecot
DNS
Irc + tmux
tor
A zillion web apps (ruby, perl, php, python,
nodejs!)
• 0 time spent on it

5. My personal server

6. Several years later….
Label for your image (if needed)
$864

7. Reality
Label for your image (if needed)
$864

8. This year’s model
Label for your image (if needed)
$864

9. Half the answer
$864

10. Convergence and immutability
• Exactly one run
– Should be building machines clean every time
– Doing exactly one puppet run
• Always rebuild
– Unless you rebuild regularly
– You don’t know you can rebuild
• Immutable instances
– Never change config on a server, replace instance!
• Hard if you only have 1 server!

11. The other half
$864

12. Compartmentalization
$864

13. Every service?
$864

14. Yes
$864
Postfix

15. Really
$864
Postfix
Dovecot

16. Every
$864
Postfix
Dovecot
Postgrey

17. Single
$864
Postfix
Dovecot
Postgrey Spamassasin

18. Service
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql

19. Independently!
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql
irssi

20. For real.
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql
irssi
playground
(sshd)

21. A cunning plan
• Build puppet code for installing
service on the old server
• Run same puppet code inside a
container to install packages / build
config
• Add shims to start service inside
container

22. Data management
• All mutable data is an lvm volume
mounted from the host
• All lvm volumes also get bind
mounted read only
• Share unix domain sockets this way
• Server for socket creates
• Clients mount ro version

23. Compartmentalization
$864

24. Containers and volumes

25. What’s inside a container?
supervisord:
• The ‘real’ process
• mcollective
• sshd

26. Code structure
• profile::dovecot
– All the things needed to run dovecot
– Parameterizeable as needed for two different deploy
environments.
• container::dovecot
– Main entry point when building the container
– Delegates most of real setup to profile::dovecot
– Adds all the container specific overrides
– Adds supervisor service(s) for this container
• run_container::dovecot
– Wraps docker::run {}
– Manages the associated lvm volumes
– Adds firewall rules

27. profile::dovecot
!
– All the things needed to run dovecot
– Parameterizeable as needed for two different deploy environments.

28. container::dovecot
!
– Main entry point when building the container
– Delegates most of real setup to profile::dovecot
– Adds all the container specific overrides
– Adds supervisor service(s) for this container

29. run_container::dovecot
!
– Wraps docker::run {}
– Manages the associated lvm volumes
– Adds firewall rules

30. Building containers
• profile::docker::build_container
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first
• profile::docker::with_socket::mysql
– /socket/mysql
– Symlink into /var/lib/mysql
• profile::docker::with_supervisord
– Setup supervisor with default (mcollective + ssh) tasks
– Default /start script to invoke supervisord

31. profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

32. profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

33. profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

34. container::with_socket::mysql
– /socket/mysql
– Symlink into /var/lib/mysql

35. container::with_supervisord
!
– Setup supervisor with default (mcollective + ssh) tasks
– Default /start script to invoke supervisord

36. The Dockerfile
• Drop facts
– /etc/facter/facts.d/is_container.txt
– /etc/facter/facts.d/container_name
• Copy in code
– ADD support/puppet /etc/puppet
• Run puppet
– Masterless
– No real manifest, just:
if $::is_container {..

37. Drop facts
– /etc/facter/facts.d/is_container.txt
!
!
!
!

38. Drop facts
– /etc/facter/facts.d/is_container.txt
!
!
!
!
!
!
– /etc/facter/facts.d/container_name.txt

39. Puppet code
– ADD support/puppet /etc/puppet
– RUN bundle exec rake puppet

40. site.pp
– Masterless
– No node manifest, just:
if $::is_container {..

41. Issues
• Docker is kinda buggy
– Just went 1.0, being fixed fast!
– No sane exit status to docker build
– AUFS 42 layer limit
• Forge modules + packages assume
service management
– No upstart inside containers - fails everywhere
• Debian packages are inconsistent
– Lots of packages don’t use invoke-rc.d

42. It’s still awesome
$864

43. Loads of TODOs
• HAProxy all the things!
– Currently just bind containers to local ports
– Container replacement is not invisible
– Run haproxy on the real host
– Dynamically regenerate its config based on running
containers
• Registry
– Build containers in Vagrant, push up to prod
• PAAS web stuff
– www all still lives on old server :(
– Gonna fix this real soon

44. Open sores!
• Open source all the things!
– garethr++
– (I owe you patches)
– Forked and changed a million modules
– Will cleanup and upstream some changes
• My profile::docker code
– Don’t know how to make this really generically
reusable
– We need sub-modules?
– Happy to share chunks

45. Sorry!

46. Questions?
http://www.yelp.com/careers?jvi=ogVTXfwL
$864
http://containercats.tumblr.com/