Continuously Integrating Puppet

  • 2,772 views
Uploaded on

"Continuously Integrating Puppet" by Matthew Bar at Puppet Camp New York 2013.

"Continuously Integrating Puppet" by Matthew Bar at Puppet Camp New York 2013.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,772
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
54
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Continuously Integrating PuppetSNAP Interactive, Inc.Presented by:Matthew BarrTechnical ArchitectWednesday, April 24, 13
  • 2. www.snap-interactive.comCompany Overview• SNAP Interactive, Inc.• Makes AYI: on Facebook, iPhone,Android,AYI.com• One of the largest social dating platforms on the web(based on monthly active users)• More than 5 million monthly active users• Over 1 billion total pieces of structured data from its users• Synced to 20+ million Facebook profiles• Receives over 35,000 real-time updates per minute from Facebook• Subscription-based business model• SNAP is publicly traded - Ticker: STVIWednesday, April 24, 13
  • 3. www.snap-interactive.comAbout me• Team lead for the operations group• Sysadmin for over 15 years• Dev/ops,WebOps, etc.• LOPSA & USENIX member• @MatthewBarr - twitter, github• mbarr@mbarr.netWednesday, April 24, 13
  • 4. www.snap-interactive.comContinuous Integration• The practice of frequently integrating onesnew or changed code with the existing coderepository• Maintain acoderepository• Automatethe build• Make thebuild self-testing• Keep thebuild fast• Everyonecan see theresults of thelatest buildWednesday, April 24, 13
  • 5. www.snap-interactive.comPuppet• Puppet Labs• http://puppetlabs.com/• You’re at PuppetCamp NYC :)Wednesday, April 24, 13
  • 6. www.snap-interactive.comJenkins• An extendable open source continuous integrationserver• http://jenkins-ci.org/Wednesday, April 24, 13
  • 7. www.snap-interactive.comSo, let’s do this!• Setup - Dynamic Env & Jenkins• Static Analysis: Puppet Parsing & Linting• Module Testing• Catalog Testing• Dynamic Analysis:Test deployment &Integrations test• Nightly RebuildsWednesday, April 24, 13
  • 8. www.snap-interactive.comActual Deployment!• Canary servers• Production rollout!Wednesday, April 24, 13
  • 9. www.snap-interactive.comSetup• Puppet - Dynamic Environments• Jenkins• Github / Github Pull Request BuilderWednesday, April 24, 13
  • 10. www.snap-interactive.comPuppet Dynamic Env’s• google: puppet dynamic environments• Git based• Each branch is it’s own environment• Commit hook / web• https://puppetlabs.com/blog/git-workflow-and-puppet-environments/Wednesday, April 24, 13
  • 11. www.snap-interactive.comJenkins install• puppet module install rtyler-jenkins• Puppet forge module: rtyler/puppet-jenkins• But: module is out of date on forge :(• It’s in our github repos:• https://github.com/matthewbarr/puppet-jenkins• https://github.com/rtyler/puppet-jenkinsWednesday, April 24, 13
  • 12. www.snap-interactive.comJenkinsWednesday, April 24, 13
  • 13. www.snap-interactive.comGithub• Private repos for organizations• Pull Requests!Wednesday, April 24, 13
  • 14. www.snap-interactive.comGH PR Builder• Monitors github for a pull request• Triggers a Jenkins build• Updates merge statusWednesday, April 24, 13
  • 15. www.snap-interactive.comGithub Pull RequestWednesday, April 24, 13
  • 16. www.snap-interactive.comStatic Analysis• Puppet Parser / Syntax check• puppet  parser  validate  $file• ERB Syntax check• erb  -­‐P  -­‐x  -­‐T  -­‐  mytempl.erb  |  ruby  -­‐c• Puppet Lint• http://puppet-lint.com/Wednesday, April 24, 13
  • 17. www.snap-interactive.comStatic Analysis• Catches errors in syntax & stylingWednesday, April 24, 13
  • 18. www.snap-interactive.comModule Testing• Rspec-Puppet• http://rspec-puppet.com/• Works well for advanced testing of a singlemodule.• Especially useful for verifying complexinternal logic.Wednesday, April 24, 13
  • 19. www.snap-interactive.comCatalog Testing• Execute a catalog compile (on the puppetmaster) with a specific set of facts and anenvironment:• puppet  master  -­‐-­‐compile  HOST    -­‐-­‐environment  ENV  >  /dev/null• Another option is Ohad Levy’s manitest• https://github.com/ohadlevy/manitestWednesday, April 24, 13
  • 20. www.snap-interactive.comCatalog TestingWednesday, April 24, 13
  • 21. www.snap-interactive.comCatalog Testing• Catches dependency loops, unknownmodules, variables or facts.Wednesday, April 24, 13
  • 22. www.snap-interactive.comDynamic Analysis• Test deployments to all server types• See the changes to each type of server• catches: RPM isn’t in a repo, etc• Integration tests• catches: Config files that break a serviceWednesday, April 24, 13
  • 23. www.snap-interactive.comTest Deployments• ssh  root@testweb1  puppet  agent  -­‐t  -­‐-­‐environment  pullreq_env  -­‐-­‐color=false• ssh  root@testmysql1  puppet  agent  -­‐t  -­‐-­‐environment  pullreq_env  -­‐-­‐color=falseWednesday, April 24, 13
  • 24. www.snap-interactive.comJenkins Log Parser• Log Parser plugin needs a parsing rules file:Wednesday, April 24, 13
  • 25. www.snap-interactive.comIntegration Testing• Covers critical functionality• Ensures that systems still serve theirfunction after new code has been applied• Could be as simple as:•  curl  http://testweb1/• But probably isn’t.Wednesday, April 24, 13
  • 26. www.snap-interactive.comNightly Rebuilds• What happens when the repos / packagesare updated, or a kickstart file is changed?• Or a manifest has something deleted• Rebuild your servers from scratch. Often.• Keeps you confident that you can rebuildyour DC & systems• Only you can prevent bitrot!Wednesday, April 24, 13
  • 27. www.snap-interactive.comNightly rebuilds • Create a newVM & get its MAC• Provision it in cobbler & IPA• Turn it onWednesday, April 24, 13
  • 28. www.snap-interactive.comNightly Rebuild• Clean up your toys!• Destroy theVM• Remove from Cobbler• Remove from IPA• Remove from PuppetWednesday, April 24, 13
  • 29. www.snap-interactive.comCanariesWednesday, April 24, 13
  • 30. www.snap-interactive.comGoing Live!• On canary servers:• puppet  apply  -­‐t  -­‐-­‐environment  TEST  -­‐-­‐noop• puppet  apply  -­‐t  -­‐-­‐environment  TEST• Merge the pull request :)• You do have functional checks for the app?• Your monitoring tools here :)Wednesday, April 24, 13
  • 31. www.snap-interactive.comHowYou Can Help • Join the conversation:• puppet-­‐ci-­‐discuss@googlegroups.com• Watch for a module to be released to helpautomate this.• https://github.com/matthewbarr/puppet-­‐ciWednesday, April 24, 13
  • 32. www.snap-interactive.comContact Info• SNAP Interactive, Inc.www.snap-interactive.com• Matthew Barrmbarr@mbarr.net• Twitter & Github:@MatthewBarrWednesday, April 24, 13