Your SlideShare is downloading. ×
  • Like
Continuously Integrating Puppet
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Continuously Integrating Puppet


"Continuously Integrating Puppet" by Matthew Bar at Puppet Camp New York 2013.

"Continuously Integrating Puppet" by Matthew Bar at Puppet Camp New York 2013.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Continuously Integrating PuppetSNAP Interactive, Inc.Presented by:Matthew BarrTechnical ArchitectWednesday, April 24, 13
  • 2. www.snap-interactive.comCompany Overview• SNAP Interactive, Inc.• Makes AYI: on Facebook, iPhone,Android,• One of the largest social dating platforms on the web(based on monthly active users)• More than 5 million monthly active users• Over 1 billion total pieces of structured data from its users• Synced to 20+ million Facebook profiles• Receives over 35,000 real-time updates per minute from Facebook• Subscription-based business model• SNAP is publicly traded - Ticker: STVIWednesday, April 24, 13
  • 3. www.snap-interactive.comAbout me• Team lead for the operations group• Sysadmin for over 15 years• Dev/ops,WebOps, etc.• LOPSA & USENIX member• @MatthewBarr - twitter, github• mbarr@mbarr.netWednesday, April 24, 13
  • 4. www.snap-interactive.comContinuous Integration• The practice of frequently integrating onesnew or changed code with the existing coderepository• Maintain acoderepository• Automatethe build• Make thebuild self-testing• Keep thebuild fast• Everyonecan see theresults of thelatest buildWednesday, April 24, 13
  • 5. www.snap-interactive.comPuppet• Puppet Labs•• You’re at PuppetCamp NYC :)Wednesday, April 24, 13
  • 6. www.snap-interactive.comJenkins• An extendable open source continuous integrationserver•, April 24, 13
  • 7. www.snap-interactive.comSo, let’s do this!• Setup - Dynamic Env & Jenkins• Static Analysis: Puppet Parsing & Linting• Module Testing• Catalog Testing• Dynamic Analysis:Test deployment &Integrations test• Nightly RebuildsWednesday, April 24, 13
  • 8. www.snap-interactive.comActual Deployment!• Canary servers• Production rollout!Wednesday, April 24, 13
  • 9. www.snap-interactive.comSetup• Puppet - Dynamic Environments• Jenkins• Github / Github Pull Request BuilderWednesday, April 24, 13
  • 10. www.snap-interactive.comPuppet Dynamic Env’s• google: puppet dynamic environments• Git based• Each branch is it’s own environment• Commit hook / web•, April 24, 13
  • 11. www.snap-interactive.comJenkins install• puppet module install rtyler-jenkins• Puppet forge module: rtyler/puppet-jenkins• But: module is out of date on forge :(• It’s in our github repos:••, April 24, 13
  • 12. www.snap-interactive.comJenkinsWednesday, April 24, 13
  • 13. www.snap-interactive.comGithub• Private repos for organizations• Pull Requests!Wednesday, April 24, 13
  • 14. www.snap-interactive.comGH PR Builder• Monitors github for a pull request• Triggers a Jenkins build• Updates merge statusWednesday, April 24, 13
  • 15. www.snap-interactive.comGithub Pull RequestWednesday, April 24, 13
  • 16. www.snap-interactive.comStatic Analysis• Puppet Parser / Syntax check• puppet  parser  validate  $file• ERB Syntax check• erb  -­‐P  -­‐x  -­‐T  -­‐  mytempl.erb  |  ruby  -­‐c• Puppet Lint•, April 24, 13
  • 17. www.snap-interactive.comStatic Analysis• Catches errors in syntax & stylingWednesday, April 24, 13
  • 18. www.snap-interactive.comModule Testing• Rspec-Puppet•• Works well for advanced testing of a singlemodule.• Especially useful for verifying complexinternal logic.Wednesday, April 24, 13
  • 19. www.snap-interactive.comCatalog Testing• Execute a catalog compile (on the puppetmaster) with a specific set of facts and anenvironment:• puppet  master  -­‐-­‐compile  HOST    -­‐-­‐environment  ENV  >  /dev/null• Another option is Ohad Levy’s manitest•, April 24, 13
  • 20. www.snap-interactive.comCatalog TestingWednesday, April 24, 13
  • 21. www.snap-interactive.comCatalog Testing• Catches dependency loops, unknownmodules, variables or facts.Wednesday, April 24, 13
  • 22. www.snap-interactive.comDynamic Analysis• Test deployments to all server types• See the changes to each type of server• catches: RPM isn’t in a repo, etc• Integration tests• catches: Config files that break a serviceWednesday, April 24, 13
  • 23. www.snap-interactive.comTest Deployments• ssh  root@testweb1  puppet  agent  -­‐t  -­‐-­‐environment  pullreq_env  -­‐-­‐color=false• ssh  root@testmysql1  puppet  agent  -­‐t  -­‐-­‐environment  pullreq_env  -­‐-­‐color=falseWednesday, April 24, 13
  • 24. www.snap-interactive.comJenkins Log Parser• Log Parser plugin needs a parsing rules file:Wednesday, April 24, 13
  • 25. www.snap-interactive.comIntegration Testing• Covers critical functionality• Ensures that systems still serve theirfunction after new code has been applied• Could be as simple as:•  curl  http://testweb1/• But probably isn’t.Wednesday, April 24, 13
  • 26. www.snap-interactive.comNightly Rebuilds• What happens when the repos / packagesare updated, or a kickstart file is changed?• Or a manifest has something deleted• Rebuild your servers from scratch. Often.• Keeps you confident that you can rebuildyour DC & systems• Only you can prevent bitrot!Wednesday, April 24, 13
  • 27. www.snap-interactive.comNightly rebuilds • Create a newVM & get its MAC• Provision it in cobbler & IPA• Turn it onWednesday, April 24, 13
  • 28. www.snap-interactive.comNightly Rebuild• Clean up your toys!• Destroy theVM• Remove from Cobbler• Remove from IPA• Remove from PuppetWednesday, April 24, 13
  • 29. www.snap-interactive.comCanariesWednesday, April 24, 13
  • 30. www.snap-interactive.comGoing Live!• On canary servers:• puppet  apply  -­‐t  -­‐-­‐environment  TEST  -­‐-­‐noop• puppet  apply  -­‐t  -­‐-­‐environment  TEST• Merge the pull request :)• You do have functional checks for the app?• Your monitoring tools here :)Wednesday, April 24, 13
  • 31. www.snap-interactive.comHowYou Can Help • Join the conversation:• puppet-­‐ci-­‐• Watch for a module to be released to helpautomate this.•­‐ciWednesday, April 24, 13
  • 32. www.snap-interactive.comContact Info• SNAP Interactive,• Matthew• Twitter & Github:@MatthewBarrWednesday, April 24, 13