Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
Deploying Puppet Code At Light Speed
•Speed of light is (still) slow
– West coast => EU slow
– East coast => Asia slow
– Shipping minimal changes = fast
•Eventual consistency FTW
– Centrally orchestrating a global network = LOL
– Parts of the network will be down
– Yes, it’s a chainsaw
– Decoupling good!
What we did
•New puppet deployment system
• svn => git
• ssh for loop => parallel mcollective
• All users sudo root => mcollective policies
• push => pull
– 4m deployment
– 4s deployment
– 32 puppetmasters globally
•Pull models > Push models
– Eventual consistency FTW
– In a large network, you will have failure - don’t rely on
100% up to ship new code
– Just cron updates to environments you care about
Demo/test your code with —environment
Different teams can own different parts of the code
Staged rollouts by merging changes between
• Environments in puppet.conf
• For modules
• modulepath = /etc/puppet/environments/
• For site.pp
• manifest = /etc/puppet/environments/
• For hiera data
• datadir = /etc/puppet/environments
• For manifests/ - refactor!!!
Segue - ‘refactor’
•Originally it meant…
• Change form
• But not function
• Supported by tests
•I’m giving up on that meaning…
• So abused by everyone, all the time
• It just means ‘change shit’
• See also hacker/cracker…
• We won that one, right?
•Custom mcollective agent
–Every git branch => puppet environment
– 205 lines of code
– 215 lines of tests ;)
mcollective agent also ships a CLI (local) client
Just cron updates to all the branches you care about
(or all branches)
Detached work trees
git clone —bare
_example checkout --detach
•Allows role users
– Generate ‘puppetupdate’ ssh key.
– Allow this to READ the puppet code.
– Distribute to puppet masters
•All the access controls
Multiple puppetupdate ssh keys.
Allow different keys different branches
I don’t need this _yet_, but it’s there!
•ssh concurrency limits
– Defaults are conservative
– Limit number of processes in ‘preauth’
– If you have 100s of puppet masters
– You’re gonna want to have multiple git servers
– Still easy, just 2 step orchestration:
– Pull to all slave git servers
– puppetupdate all the masters
•Documentation not on slideshare
– I did fix the README, it’s still not awesome.
– Only 250 lines of code, just reading it isn’t hard ;)
– Nicer user display of status
– mco plugin package only bundles mco bits
– You need to ship /usr/local/sbin/puppetupdate
• Slides: http://slideshare.net/bobtfish
• Tweet me @bobtfish
• Guess what?
• We’re hiring!!!
• SF + Palo Alto
• (+ London, + Hamburg, + Dublin)
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.