Sock Puppets: Growing Your Puppet Codebase

2,576 views
2,324 views

Published on

Presenters: Fabio Lessa & Andrew Myers

Published in: Technology, Spiritual

Sock Puppets: Growing Your Puppet Codebase

  1. 1. Sock Puppets: Growing yourpuppet codebaseFabio Lessa & Andrew Myers
  2. 2. Part 1
  3. 3. Empty server room
  4. 4. Project teams teams
  5. 5. OPS team for app support
  6. 6. The codebase weve started withSomething that represents a mess
  7. 7. StandardPuppet Master setup
  8. 8. Code was split by environments
  9. 9. Workflow:1. Push change2. Wait for it to propagate to thatenvironment3. Check it workedIf you need that applied to anotherenvironment, go to 1
  10. 10. The real Workflow:1. Push change1.1. Stop the puppet daemon in all the critical nodes for thatenvironment1.2. Run puppetd manually in one hostto ensure it works1.3. Start the daemon again (on all the hosts you remember)2. Wait for it to propagate to that environment3. Check it worked (meh)If you need that applied to another environment, go to 1 (if youremember, that is)
  11. 11. ProductionUAT System Testing CI
  12. 12. Making changeswas painful
  13. 13. Part 2
  14. 14. "dont duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Dont test if puppet is doing its job, test that your logic its doing its job. - Nikolay Sturm
  15. 15. The goal...Before After
  16. 16. So how do we test/refactor
  17. 17. Compile Puppet Catalogues
  18. 18. Compare Versions
  19. 19. Application specific tests
  20. 20. Tying it all together
  21. 21. Part 3
  22. 22. Continuous Delivery
  23. 23. We were moving application deploys, whynot do the same with infrastructure?
  24. 24. MCollective
  25. 25. The new setup
  26. 26. Picture of the pipelines
  27. 27. info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/hostinfo/lib/facter/hostinfo.rbinfo: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/concat/lib/facter/concat_basedir.rbinfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbinfo: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rbinfo: Loading facts in /var/lib/puppet/facts/zone.rbinfo: Loading facts in /var/lib/puppet/facts/read_facts.rbinfo: Loading facts in /var/lib/puppet/facts/stomp_host.rbinfo: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rbinfo: Caching catalog for somehost.nbndc.localerr: Failed to apply catalog: Could not find dependency File[/root/.hgrc] forMercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/puppetmaster/manifests/repos.pp:36
  28. 28. info: Retrieving plugininfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbinfo: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rbinfo: Loading facts in /var/lib/puppet/facts/zone.rbinfo: Loading facts in /var/lib/puppet/facts/read_facts.rbinfo: Loading facts in /var/lib/puppet/facts/stomp_host.rbinfo: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rbinfo: Caching catalog for somehost.nbndc.localerr: Failed to apply catalog: You cannot specify more than one ofcontent, source, target at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/sbs/manifests/init.pp:73
  29. 29. notice: Class[Hostinfo]: Would have triggered refresh from 1 eventsnotice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message:current_value absent, should be Setting java the following java as default:1.6.0.31 sun 1.6.0 (noop)notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered refreshfrom 1 eventsnotice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggeredrefresh from 1 eventsnotice: Class[Java::Sun]: Would have triggered refresh from 1 eventsnotice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_valueabsent, should be directory (noop)err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could notevaluate: Could not retrieve information from environment noop source(s)puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/manifests/init.pp:30notice: Class[Puppetagent]: Would have triggered refresh from 1 eventsnotice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_valuedirectory, should be absent (noop)notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure:current_value file, should be absent (noop)
  30. 30. We were moving application deploys, whynot do the same with infrastructure?
  31. 31. Our MCollective agent
  32. 32. The new action
  33. 33. At every push, all nodes connect to the noop environment and run in noop mode
  34. 34. Picture of a report
  35. 35. Changemanagementprocess ?
  36. 36. What changed?
  37. 37. What did you test?
  38. 38. Who will do it?
  39. 39. Part 4
  40. 40. New data centre is ready!
  41. 41. Should be easy to migrate....
  42. 42. Okay so our manifests are incomplete,but we need to be able to rebuild machines... Virtualisation to the rescue...
  43. 43. Magnets?
  44. 44. What happens
  45. 45. Wrap Up
  46. 46. 120 pushes toproduction in thelast 8 months: 2failures
  47. 47. MTTR FTW! 5 minutes!
  48. 48. Thank you!

×