This document summarizes lessons learned from deploying Puppet code globally at high speed. The key changes were moving from SVN to Git for version control, parallelizing deployments using MCollective instead of SSH loops, using MCollective policies instead of sudo, and switching to a pull model over push. These changes allowed deployments to be reduced from 4 minutes to 4 seconds. Environments were used to separate code for different teams and stages. A custom MCollective agent was created to deploy Git branches as Puppet environments. Cron jobs were used to pull updates to environments. Overall this approach improved the speed, consistency, and security of global Puppet deployments.

2. Deploying Puppet Code At Light Speed
Tomas Doran
@bobtfish
tdoran@yelp.com
2014-01-27

3. Key lessons
•Speed of light is (still) slow
– West coast => EU slow
– East coast => Asia slow
•git <3
– Shipping minimal changes = fast
•Eventual consistency FTW
– Centrally orchestrating a global network = LOL
– Parts of the network will be down
•mcollective <3
– Yes, it’s a chainsaw
– Decoupling good!

4. What we did
•New puppet deployment system
• svn => git
• ssh for loop => parallel mcollective
• All users sudo root => mcollective policies
• push => pull
•Winning!
– 4m deployment
– 4s deployment
– 32 puppetmasters globally

5. Why?
•Pull models > Push models
– Eventual consistency FTW
– In a large network, you will have failure - don’t rely on
100% up to ship new code
– Just cron updates to environments you care about
•Environments
•
•
•
•
Personal branches
Demo/test your code with —environment
Different teams can own different parts of the code
Staged rollouts by merging changes between
branches

6. How?
• Environments in puppet.conf
• For modules
• modulepath = /etc/puppet/environments/
$environment/modules
• For site.pp
• manifest = /etc/puppet/environments/
$environment/manifests/site.pp
• For hiera data
• datadir = /etc/puppet/environments
• %{::environment}/common.yaml
• For manifests/ - refactor!!!

7. Segue - ‘refactor’
•Originally it meant…
• Change form
• But not function
• incrementally
• Supported by tests
•I’m giving up on that meaning…
• So abused by everyone, all the time
• It just means ‘change shit’
• See also hacker/cracker…
• We won that one, right?

8. How?
•Custom mcollective agent
–Every git branch => puppet environment
– 205 lines of code
– 215 lines of tests ;)
•Cron job
–
–
–
–
mcollective agent also ships a CLI (local) client
Just cron updates to all the branches you care about
(or all branches)
Eventual consistency!

9. Detached work trees
•/etc/puppet/puppet.git
git clone —bare
git —git-dir=/etc/puppet/puppet.git
—worktree=/etc/puppet/environments/my
_example checkout --detach
--force master

10. Gitolite
•Allows role users
– Generate ‘puppetupdate’ ssh key.
– Allow this to READ the puppet code.
– Distribute to puppet masters
•All the access controls
–
–
–
–
Multiple puppetupdate ssh keys.
Allow different keys different branches
dev/stage/prod
I don’t need this _yet_, but it’s there!

11. Gotchas
•ssh concurrency limits
– Defaults are conservative
– Limit number of processes in ‘preauth’
•Scaling MOAR
– If you have 100s of puppet masters
– You’re gonna want to have multiple git servers
– Still easy, just 2 step orchestration:
– Pull to all slave git servers
– puppetupdate all the masters

12. TODOs
•Documentation not on slideshare
– I did fix the README, it’s still not awesome.
– Only 250 lines of code, just reading it isn’t hard ;)
•Better application
– Nicer user display of status
•Extend mcollective
– mco plugin package only bundles mco bits
– You need to ship /usr/local/sbin/puppetupdate
yourself

13. Link me?
• https://github.com/Yelp/puppetupdate
• Slides: http://slideshare.net/bobtfish
• Tweet me @bobtfish
• Guess what?
• We’re hiring!!!
• SF + Palo Alto
• (+ London, + Hamburg, + Dublin)