Your SlideShare is downloading. ×
Continuously Integrating Puppet
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Continuously Integrating Puppet


Published on

"Continuously Integrating Puppet" by Matthew Bar at Puppet Camp New York 2013.

"Continuously Integrating Puppet" by Matthew Bar at Puppet Camp New York 2013.

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Continuously Integrating PuppetSNAP Interactive, Inc.Presented by:Matthew BarrTechnical ArchitectWednesday, April 24, 13
  • 2. www.snap-interactive.comCompany Overview• SNAP Interactive, Inc.• Makes AYI: on Facebook, iPhone,Android,• One of the largest social dating platforms on the web(based on monthly active users)• More than 5 million monthly active users• Over 1 billion total pieces of structured data from its users• Synced to 20+ million Facebook profiles• Receives over 35,000 real-time updates per minute from Facebook• Subscription-based business model• SNAP is publicly traded - Ticker: STVIWednesday, April 24, 13
  • 3. www.snap-interactive.comAbout me• Team lead for the operations group• Sysadmin for over 15 years• Dev/ops,WebOps, etc.• LOPSA & USENIX member• @MatthewBarr - twitter, github• mbarr@mbarr.netWednesday, April 24, 13
  • 4. www.snap-interactive.comContinuous Integration• The practice of frequently integrating onesnew or changed code with the existing coderepository• Maintain acoderepository• Automatethe build• Make thebuild self-testing• Keep thebuild fast• Everyonecan see theresults of thelatest buildWednesday, April 24, 13
  • 5. www.snap-interactive.comPuppet• Puppet Labs•• You’re at PuppetCamp NYC :)Wednesday, April 24, 13
  • 6. www.snap-interactive.comJenkins• An extendable open source continuous integrationserver•, April 24, 13
  • 7. www.snap-interactive.comSo, let’s do this!• Setup - Dynamic Env & Jenkins• Static Analysis: Puppet Parsing & Linting• Module Testing• Catalog Testing• Dynamic Analysis:Test deployment &Integrations test• Nightly RebuildsWednesday, April 24, 13
  • 8. www.snap-interactive.comActual Deployment!• Canary servers• Production rollout!Wednesday, April 24, 13
  • 9. www.snap-interactive.comSetup• Puppet - Dynamic Environments• Jenkins• Github / Github Pull Request BuilderWednesday, April 24, 13
  • 10. www.snap-interactive.comPuppet Dynamic Env’s• google: puppet dynamic environments• Git based• Each branch is it’s own environment• Commit hook / web•, April 24, 13
  • 11. www.snap-interactive.comJenkins install• puppet module install rtyler-jenkins• Puppet forge module: rtyler/puppet-jenkins• But: module is out of date on forge :(• It’s in our github repos:••, April 24, 13
  • 12. www.snap-interactive.comJenkinsWednesday, April 24, 13
  • 13. www.snap-interactive.comGithub• Private repos for organizations• Pull Requests!Wednesday, April 24, 13
  • 14. www.snap-interactive.comGH PR Builder• Monitors github for a pull request• Triggers a Jenkins build• Updates merge statusWednesday, April 24, 13
  • 15. www.snap-interactive.comGithub Pull RequestWednesday, April 24, 13
  • 16. www.snap-interactive.comStatic Analysis• Puppet Parser / Syntax check• puppet  parser  validate  $file• ERB Syntax check• erb  -­‐P  -­‐x  -­‐T  -­‐  mytempl.erb  |  ruby  -­‐c• Puppet Lint•, April 24, 13
  • 17. www.snap-interactive.comStatic Analysis• Catches errors in syntax & stylingWednesday, April 24, 13
  • 18. www.snap-interactive.comModule Testing• Rspec-Puppet•• Works well for advanced testing of a singlemodule.• Especially useful for verifying complexinternal logic.Wednesday, April 24, 13
  • 19. www.snap-interactive.comCatalog Testing• Execute a catalog compile (on the puppetmaster) with a specific set of facts and anenvironment:• puppet  master  -­‐-­‐compile  HOST    -­‐-­‐environment  ENV  >  /dev/null• Another option is Ohad Levy’s manitest•, April 24, 13
  • 20. www.snap-interactive.comCatalog TestingWednesday, April 24, 13
  • 21. www.snap-interactive.comCatalog Testing• Catches dependency loops, unknownmodules, variables or facts.Wednesday, April 24, 13
  • 22. www.snap-interactive.comDynamic Analysis• Test deployments to all server types• See the changes to each type of server• catches: RPM isn’t in a repo, etc• Integration tests• catches: Config files that break a serviceWednesday, April 24, 13
  • 23. www.snap-interactive.comTest Deployments• ssh  root@testweb1  puppet  agent  -­‐t  -­‐-­‐environment  pullreq_env  -­‐-­‐color=false• ssh  root@testmysql1  puppet  agent  -­‐t  -­‐-­‐environment  pullreq_env  -­‐-­‐color=falseWednesday, April 24, 13
  • 24. www.snap-interactive.comJenkins Log Parser• Log Parser plugin needs a parsing rules file:Wednesday, April 24, 13
  • 25. www.snap-interactive.comIntegration Testing• Covers critical functionality• Ensures that systems still serve theirfunction after new code has been applied• Could be as simple as:•  curl  http://testweb1/• But probably isn’t.Wednesday, April 24, 13
  • 26. www.snap-interactive.comNightly Rebuilds• What happens when the repos / packagesare updated, or a kickstart file is changed?• Or a manifest has something deleted• Rebuild your servers from scratch. Often.• Keeps you confident that you can rebuildyour DC & systems• Only you can prevent bitrot!Wednesday, April 24, 13
  • 27. www.snap-interactive.comNightly rebuilds • Create a newVM & get its MAC• Provision it in cobbler & IPA• Turn it onWednesday, April 24, 13
  • 28. www.snap-interactive.comNightly Rebuild• Clean up your toys!• Destroy theVM• Remove from Cobbler• Remove from IPA• Remove from PuppetWednesday, April 24, 13
  • 29. www.snap-interactive.comCanariesWednesday, April 24, 13
  • 30. www.snap-interactive.comGoing Live!• On canary servers:• puppet  apply  -­‐t  -­‐-­‐environment  TEST  -­‐-­‐noop• puppet  apply  -­‐t  -­‐-­‐environment  TEST• Merge the pull request :)• You do have functional checks for the app?• Your monitoring tools here :)Wednesday, April 24, 13
  • 31. www.snap-interactive.comHowYou Can Help • Join the conversation:• puppet-­‐ci-­‐• Watch for a module to be released to helpautomate this.•­‐ciWednesday, April 24, 13
  • 32. www.snap-interactive.comContact Info• SNAP Interactive,• Matthew• Twitter & Github:@MatthewBarrWednesday, April 24, 13