0
Sock Puppets: Growing yourpuppet codebaseFabio Lessa & Andrew Myers
Part 1
Empty server room
Project teams          teams
OPS team for app    support
The                                   codebase                                   weve                                   st...
StandardPuppet Master setup
Code was split by environments
Workflow:1. Push change2. Wait for it to propagate to thatenvironment3. Check it workedIf you need that applied to another...
The real Workflow:1. Push change1.1. Stop the puppet daemon in all the critical nodes for thatenvironment1.2. Run puppetd ...
ProductionUAT                   System Testing           CI
Making changeswas painful
Part 2
"dont duplicate your manifests in your tests" Focus  on the catalog problems he described earlier and test your logic. Don...
The goal...Before          After
So how do we test/refactor
Compile Puppet Catalogues
Compare Versions
Application specific tests
Tying it all together
Part 3
Continuous Delivery
We were moving application deploys, whynot do the same with infrastructure?
MCollective
The new setup
Picture of the pipelines
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/hostinfo/lib/facter/hostinfo.rbin...
info: Retrieving plugininfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbinfo: Loading facts in /var/lib/...
notice: Class[Hostinfo]: Would have triggered refresh from 1 eventsnotice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6....
We were moving application deploys, whynot do the same with infrastructure?
Our MCollective agent
The new action
At every push, all nodes connect to the noop     environment and run in noop mode
Picture of a report
Changemanagementprocess  ?
What changed?
What did you test?
Who will do it?
Part 4
New data centre is ready!
Should be easy to migrate....
Okay so our manifests are incomplete,but we need to be able to rebuild machines...        Virtualisation to the rescue...
Magnets?
What happens
Wrap Up
120 pushes toproduction in thelast 8 months: 2failures
MTTR FTW!       5 minutes!
Thank you!
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing Your Puppet Codebase
Upcoming SlideShare
Loading in...5
×

Sock Puppets: Growing Your Puppet Codebase

2,029

Published on

Presenters: Fabio Lessa & Andrew Myers

Published in: Technology, Spiritual
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,029
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
23
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Transcript of "Sock Puppets: Growing Your Puppet Codebase"

  1. 1. Sock Puppets: Growing yourpuppet codebaseFabio Lessa & Andrew Myers
  2. 2. Part 1
  3. 3. Empty server room
  4. 4. Project teams teams
  5. 5. OPS team for app support
  6. 6. The codebase weve started withSomething that represents a mess
  7. 7. StandardPuppet Master setup
  8. 8. Code was split by environments
  9. 9. Workflow:1. Push change2. Wait for it to propagate to thatenvironment3. Check it workedIf you need that applied to anotherenvironment, go to 1
  10. 10. The real Workflow:1. Push change1.1. Stop the puppet daemon in all the critical nodes for thatenvironment1.2. Run puppetd manually in one hostto ensure it works1.3. Start the daemon again (on all the hosts you remember)2. Wait for it to propagate to that environment3. Check it worked (meh)If you need that applied to another environment, go to 1 (if youremember, that is)
  11. 11. ProductionUAT System Testing CI
  12. 12. Making changeswas painful
  13. 13. Part 2
  14. 14. "dont duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Dont test if puppet is doing its job, test that your logic its doing its job. - Nikolay Sturm
  15. 15. The goal...Before After
  16. 16. So how do we test/refactor
  17. 17. Compile Puppet Catalogues
  18. 18. Compare Versions
  19. 19. Application specific tests
  20. 20. Tying it all together
  21. 21. Part 3
  22. 22. Continuous Delivery
  23. 23. We were moving application deploys, whynot do the same with infrastructure?
  24. 24. MCollective
  25. 25. The new setup
  26. 26. Picture of the pipelines
  27. 27. info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/hostinfo/lib/facter/hostinfo.rbinfo: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/concat/lib/facter/concat_basedir.rbinfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbinfo: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rbinfo: Loading facts in /var/lib/puppet/facts/zone.rbinfo: Loading facts in /var/lib/puppet/facts/read_facts.rbinfo: Loading facts in /var/lib/puppet/facts/stomp_host.rbinfo: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rbinfo: Caching catalog for somehost.nbndc.localerr: Failed to apply catalog: Could not find dependency File[/root/.hgrc] forMercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/puppetmaster/manifests/repos.pp:36
  28. 28. info: Retrieving plugininfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbinfo: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rbinfo: Loading facts in /var/lib/puppet/facts/zone.rbinfo: Loading facts in /var/lib/puppet/facts/read_facts.rbinfo: Loading facts in /var/lib/puppet/facts/stomp_host.rbinfo: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rbinfo: Caching catalog for somehost.nbndc.localerr: Failed to apply catalog: You cannot specify more than one ofcontent, source, target at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/sbs/manifests/init.pp:73
  29. 29. notice: Class[Hostinfo]: Would have triggered refresh from 1 eventsnotice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message:current_value absent, should be Setting java the following java as default:1.6.0.31 sun 1.6.0 (noop)notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered refreshfrom 1 eventsnotice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggeredrefresh from 1 eventsnotice: Class[Java::Sun]: Would have triggered refresh from 1 eventsnotice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_valueabsent, should be directory (noop)err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could notevaluate: Could not retrieve information from environment noop source(s)puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/manifests/init.pp:30notice: Class[Puppetagent]: Would have triggered refresh from 1 eventsnotice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_valuedirectory, should be absent (noop)notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure:current_value file, should be absent (noop)
  30. 30. We were moving application deploys, whynot do the same with infrastructure?
  31. 31. Our MCollective agent
  32. 32. The new action
  33. 33. At every push, all nodes connect to the noop environment and run in noop mode
  34. 34. Picture of a report
  35. 35. Changemanagementprocess ?
  36. 36. What changed?
  37. 37. What did you test?
  38. 38. Who will do it?
  39. 39. Part 4
  40. 40. New data centre is ready!
  41. 41. Should be easy to migrate....
  42. 42. Okay so our manifests are incomplete,but we need to be able to rebuild machines... Virtualisation to the rescue...
  43. 43. Magnets?
  44. 44. What happens
  45. 45. Wrap Up
  46. 46. 120 pushes toproduction in thelast 8 months: 2failures
  47. 47. MTTR FTW! 5 minutes!
  48. 48. Thank you!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×