1.
Taking control of chaos
(with Docker and puppet)
- Tomas Doran
@bobtfish
18/11/2013

2.
Docker - why should you care?
• Isolation
– Run each daemon in it’s own container
• Security
– Containers are fully independent
– Build a new container for a new application version!
• Ease of development
– Same containers in Vagrant on laptop as in production!
– Build pipeline for whole application environment
• Ease of deployment
– Deploy from your own private registry
– Roll back by just deploying the last version

3.
Taking control of chaos
• My personal server
• Lots of responsibilities
–
–
–
–
–
–
Postfix
Dovecot
DNS
Irc + tmux
tor
A zillion web apps (ruby, perl, php, python,
nodejs!)
• 0 time spent on it

4.
My personal server

5.
Several years later….
Label for your image (if needed)
$864

6.
Reality
Label for your image (if needed)
$864

7.
This year’s model
Label for your image (if needed)
$864

8.
Half the answer
$864

9.
Convergence and immutability
• Exactly one run
– Should be building machines clean every time
– Doing exactly one puppet run
• Always rebuild
– Unless you rebuild regularly
– You don’t know you can rebuild
• Immutable instances
– Never change config on a server, replace instance!
• Hard if you only have 1 server!

10.
The other half
$864

11.
Compartmentalization
$864

12.
Every service?
$864

13.
Yes
$864
Postfix

14.
Really
$864
Postfix
Dovecot

15.
Every
$864
Postfix
Dovecot
Postgrey

16.
Single
$864
Postfix
Dovecot
Postgrey Spamassasin

17.
Service
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql

18.
Independently!
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql
irssi

19.
For real.
$864
Postfix
Dovecot
Postgrey Spamassasin Mysql
irssi
playground
(sshd)

20.
A cunning plan
• Build puppet code for installing
service on the old server
• Run same puppet code inside a
container to install packages / build
config
• Add shims to start service inside
container

21.
Data management
• All mutable data is an lvm volume
mounted from the host
• All lvm volumes also get bind
mounted read only
• Share unix domain sockets this way
• Server for socket creates
• Clients mount ro version

22.
Compartmentalization
$864

23.
Containers and volumes

24.
What’s inside a container?
supervisord:
• The ‘real’ process
• mcollective
• sshd

25.
Code structure
• profile::dovecot
– All the things needed to run dovecot
– Parameterizeable as needed for two different deploy
environments.
• container::dovecot
– Main entry point when building the container
– Delegates most of real setup to profile::dovecot
– Adds all the container specific overrides
– Adds supervisor service(s) for this container
• run_container::dovecot
– Wraps docker::run {}
– Manages the associated lvm volumes
– Adds firewall rules

26.
profile::dovecot
!
– All the things needed to run dovecot
– Parameterizeable as needed for two different deploy environments.

27.
container::dovecot
!
– Main entry point when building the container
– Delegates most of real setup to profile::dovecot
– Adds all the container specific overrides
– Adds supervisor service(s) for this container

28.
run_container::dovecot
!
– Wraps docker::run {}
– Manages the associated lvm volumes
– Adds firewall rules

29.
Building containers
• profile::docker::build_container
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first
• profile::docker::with_socket::mysql
– /socket/mysql
– Symlink into /var/lib/mysql
• profile::docker::with_supervisord
– Setup supervisor with default (mcollective + ssh) tasks
– Default /start script to invoke supervisord

30.
profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

31.
profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

32.
profile::docker::build_container
!
– define - writes out Dockerfile + support files
– Runs docker build .
– Manages dependencies so base containers get built first

33.
container::with_socket::mysql
– /socket/mysql
– Symlink into /var/lib/mysql

34.
container::with_supervisord
!
– Setup supervisor with default (mcollective + ssh) tasks
– Default /start script to invoke supervisord

35.
The Dockerfile
• Drop facts
– /etc/facter/facts.d/is_container.txt
– /etc/facter/facts.d/container_name
• Copy in code
– ADD support/puppet /etc/puppet
• Run puppet
– Masterless
– No real manifest, just:
if $::is_container {..

36.
Drop facts
– /etc/facter/facts.d/is_container.txt
!
!
!
!

37.
Drop facts
– /etc/facter/facts.d/is_container.txt
!
!
!
!
!
!
– /etc/facter/facts.d/container_name.txt

38.
Puppet code
– ADD support/puppet /etc/puppet
– RUN bundle exec rake puppet

39.
site.pp
– Masterless
– No node manifest, just:
if $::is_container {..

40.
Issues
• Docker is kinda buggy
– Just went 1.0, being fixed fast!
– No sane exit status to docker build
– AUFS 42 layer limit
• Forge modules + packages assume
service management
– No upstart inside containers - fails everywhere
• Debian packages are inconsistent
– Lots of packages don’t use invoke-rc.d

41.
It’s still awesome
$864

42.
Loads of TODOs
• HAProxy all the things!
– Currently just bind containers to local ports
– Container replacement is not invisible
– Run haproxy on the real host
– Dynamically regenerate its config based on running
containers
• Registry
– Build containers in Vagrant, push up to prod
• PAAS web stuff
– www all still lives on old server :(
– Gonna fix this real soon

43.
Open sores!
• Open source all the things!
– garethr++
– (I owe you patches)
– Forked and changed a million modules
– Will cleanup and upstream some changes
• My profile::docker code
– Don’t know how to make this really generically
reusable
– We need sub-modules?
– Happy to share chunks

44.
Sorry!

45.
Questions?
http://www.yelp.com/careers?jvi=ogVTXfwL
$864
http://containercats.tumblr.com/