Deploying Puppet Code At Light Speed
Tomas Doran
@bobtfish
tdoran@yelp.com
2014-01-27
Key lessons
•Speed of light is (still) slow
– West coast => EU slow
– East coast => Asia slow

•git <3
– Shipping minimal ...
What we did
•New puppet deployment system
• svn => git
• ssh for loop => parallel mcollective
• All users sudo root => mco...
Why?
•Pull models > Push models
– Eventual consistency FTW
– In a large network, you will have failure - don’t rely on
100...
How?

• Environments in puppet.conf
• For modules
• modulepath = /etc/puppet/environments/
$environment/modules
• For site...
Segue - ‘refactor’
•Originally it meant…
• Change form
• But not function
• incrementally
• Supported by tests

•I’m givin...
How?
•Custom mcollective agent
–Every git branch => puppet environment
– 205 lines of code
– 215 lines of tests ;)

•Cron ...
Detached work trees
•/etc/puppet/puppet.git
git clone —bare
git —git-dir=/etc/puppet/puppet.git
—worktree=/etc/puppet/envi...
Gitolite
•Allows role users
– Generate ‘puppetupdate’ ssh key.
– Allow this to READ the puppet code.
– Distribute to puppe...
Gotchas
•ssh concurrency limits
– Defaults are conservative
– Limit number of processes in ‘preauth’

•Scaling MOAR
– If y...
TODOs
•Documentation not on slideshare
– I did fix the README, it’s still not awesome.
– Only 250 lines of code, just read...
Link me?
• https://github.com/Yelp/puppetupdate
• Slides: http://slideshare.net/bobtfish
• Tweet me @bobtfish
• Guess what...
Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
Upcoming SlideShare
Loading in...5
×

Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley

1,286

Published on

Tomas Doran talks about Deploying Puppet Code at Light Speed at Puppet Camp Silicon Valley

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,286
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
27
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley

  1. 1. Deploying Puppet Code At Light Speed Tomas Doran @bobtfish tdoran@yelp.com 2014-01-27
  2. 2. Key lessons •Speed of light is (still) slow – West coast => EU slow – East coast => Asia slow •git <3 – Shipping minimal changes = fast •Eventual consistency FTW – Centrally orchestrating a global network = LOL – Parts of the network will be down •mcollective <3 – Yes, it’s a chainsaw – Decoupling good!
  3. 3. What we did •New puppet deployment system • svn => git • ssh for loop => parallel mcollective • All users sudo root => mcollective policies • push => pull •Winning! – 4m deployment – 4s deployment – 32 puppetmasters globally
  4. 4. Why? •Pull models > Push models – Eventual consistency FTW – In a large network, you will have failure - don’t rely on 100% up to ship new code – Just cron updates to environments you care about •Environments • • • • Personal branches Demo/test your code with —environment Different teams can own different parts of the code Staged rollouts by merging changes between branches
  5. 5. How? • Environments in puppet.conf • For modules • modulepath = /etc/puppet/environments/ $environment/modules • For site.pp • manifest = /etc/puppet/environments/ $environment/manifests/site.pp • For hiera data • datadir = /etc/puppet/environments • %{::environment}/common.yaml • For manifests/ - refactor!!!
  6. 6. Segue - ‘refactor’ •Originally it meant… • Change form • But not function • incrementally • Supported by tests •I’m giving up on that meaning… • So abused by everyone, all the time • It just means ‘change shit’ • See also hacker/cracker… • We won that one, right?
  7. 7. How? •Custom mcollective agent –Every git branch => puppet environment – 205 lines of code – 215 lines of tests ;) •Cron job – – – – mcollective agent also ships a CLI (local) client Just cron updates to all the branches you care about (or all branches) Eventual consistency!
  8. 8. Detached work trees •/etc/puppet/puppet.git git clone —bare git —git-dir=/etc/puppet/puppet.git —worktree=/etc/puppet/environments/my _example checkout --detach --force master
  9. 9. Gitolite •Allows role users – Generate ‘puppetupdate’ ssh key. – Allow this to READ the puppet code. – Distribute to puppet masters •All the access controls – – – – Multiple puppetupdate ssh keys. Allow different keys different branches dev/stage/prod I don’t need this _yet_, but it’s there!
  10. 10. Gotchas •ssh concurrency limits – Defaults are conservative – Limit number of processes in ‘preauth’ •Scaling MOAR – If you have 100s of puppet masters – You’re gonna want to have multiple git servers – Still easy, just 2 step orchestration: – Pull to all slave git servers – puppetupdate all the masters
  11. 11. TODOs •Documentation not on slideshare – I did fix the README, it’s still not awesome. – Only 250 lines of code, just reading it isn’t hard ;) •Better application – Nicer user display of status •Extend mcollective – mco plugin package only bundles mco bits – You need to ship /usr/local/sbin/puppetupdate yourself
  12. 12. Link me? • https://github.com/Yelp/puppetupdate • Slides: http://slideshare.net/bobtfish • Tweet me @bobtfish • Guess what? • We’re hiring!!! • SF + Palo Alto • (+ London, + Hamburg, + Dublin)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×