Continuous Delivery of Puppet Manifests

3,794 views
3,497 views

Published on

PuppetCamp Amsterdam 2014 Talk

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,794
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
243
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Continuous Delivery of Puppet Manifests

  1. 1. Continuous Delivery of Puppet Code, Lessons Learned Kris Buytaert @krisbuytaert
  2. 2. Kris Buytaert ● ● ● ● ● ● ● I used to be a Dev, Then Became an Op Chief Trolling Officer and Open Source Consultant @inuits.eu Everything is an effing DNS Problem Building Clouds since before the bookstore Some books, some papers, some blogs Evangelizing devops
  3. 3. Todays Goals ● A reproducable way to deploy and upgrade /etc/puppet ● Automatically ● Fast ● Consistent ● Continuously
  4. 4. What's this devops thing anyhow ?
  5. 5. devops ● Culture ● (Lean) ● Automation ● Measurement ● Sharing Damon Edwards and John Willis Gene Kim
  6. 6. devops (<)> continuous delilvery
  7. 7. Nirvana An “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management to business. Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its code base. Continuous Delivery , Jez Humble
  8. 8. How many times a day ? ● 10 @ Flickr ● Deployments used to be pain ● Nobody dared to deploy a site ● Practice makes perfect ● Knowing you can vs constantly doing it
  9. 9. " Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to enable the business goals. We strongly feel that in order to do that you must have the ability to deploy code quickly and safely. Even if the business goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push all the time), having a reliable and easy deployment should be non-negotiable." Etsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
  10. 10. For years we've tolerated humans to make structural manual changes to the infrastructure our critical applications are running on. Whilst at the same time demanding those critical applications to go through rigid test scenarios. Who let this happen ?
  11. 11. I hate maturity model
  12. 12. So, what did we try already ?
  13. 13. Level -1:Hacking in production • Cd /etc/puppet/manifests • Vi site.pp
  14. 14. Level -1: more production hacking • cd /etc/puppet/manifests • vi site.pp • • • rsync -av * /etc/puppet • WFM Syndrome • Does not work in teams
  15. 15. Level 0: I use git • ssh puppetmaster • cd /etc/puppet • git init • git add *.pp • git commit -m “Initial commit of puppet tree”
  16. 16. Level 0.1: I almost understand git • • Ssh puppetmaster • Cd /etc/puppet • Git init • Git add *.pp • Git commit “Initial commit of puppet tree” • Git remote add • Git push
  17. 17. Level 0.2: But • Ssh puppetmaster • Cd /etc/puppet • Git init • Git add *.pp • Git commit “Initial commit of puppet tree” • Git remote add • Git push • Vi nodes/default.pp
  18. 18. Level 0.2:But • I don't always commit or push my changes • When I do commit I do it as root
  19. 19. Level 0.3: Development happens locally Puppetmaster runs git pull in a cronjob => code modified on puppet master, not pushed => changes never make it to the platform
  20. 20. Level 0.3: Euh modules vi modules/apache/manifest/init.pp Wait, I need to track upstream , How do I isolate my code ?
  21. 21. Git subtrees ● Looks nice ● Till you want to track upstream
  22. 22. Librarian Puppet ● Hides complexity of submodules ● Easy if you use Forge Modules • • ● ● Does anyone ? Do you trust the internet to be around Librarian = Old English for “can't use submodules” And hmm... which customer uses which patched version again ?
  23. 23. Librarian Puppet ● Insert ugly shell script ● ● Even with this in place .. people can still hack on the PuppetMaster
  24. 24. We all love branches ● When they are short lived feature branches ● Environment per branch ? • • How many hosts do you connect per branch ? Limited number of branches ?
  25. 25. Is R10K faster ? ● R10K ● But what about Testing ? ● You can't do CD without CI
  26. 26. Git Submodules ● Basic git, ● No extra tools required Integrates with other (non puppet) projects too
  27. 27. Package all the things
  28. 28. Release artifacts: ● A module ● A set of manifests ● A set of manifests that work with a strict set of modules
  29. 29. Software Release management is not a solved problem
  30. 30. Git Submodules ● Release management = main git project cd modules/blah vi manifest/init.pp git add manifests/init.pp git commit -m “Fixed bug #313” cd ../../ git add modules/blah git commit -m “Fixed bug #313” git push
  31. 31. Infrastructure as Code ● Treat configuration automation as code ● Development best practices • Model your infrastructure • Version your cookbooks / manifests • Test your cookbooks/ manifests • Dev/ test /uat / prod for your infra ● Model your infrastructure ● A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )
  32. 32. Continuous Integration ● Builds ● Nightly Builds ● Builds with tests ● Nightly Builds with tests ● Frequent integration ● Continuous Integration
  33. 33. Jenkins ● Open Source Continuous Integration Server ● A zillion plugins (400) ● Have developers build stable and deployable code ● Test Infra code
  34. 34. Jenkins Pipeline
  35. 35. What's in your Pipeline ?
  36. 36. A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package
  37. 37. Syntax and Style ● Initially , all code, all the time ● Now, only the changed code ● Why not in post Commit Hooks ?
  38. 38. Why ops like to package ● Packages give you features •Consistency, security, dependencies ● Uniquely identify where files come from •Package or cfg-mgmt ● Source repo not always available •Firewall / Cloud etc .. ● Weird deployment locations , no easy access ● Little overhead when you automate
  39. 39. Jordan Sissel is a Hero !
  40. 40. #packaginlove
  41. 41. It's not really packaging • It's an immutable branch • It's a tracable release artefact
  42. 42. https://github.com/vStone/jenkinspuppet-scripts ● Tests ● Packages full tree in / etc/puppet/environm ents/$environment/
  43. 43. A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package ● Upload to Repo
  44. 44. Repository Management ● Pulp • Pro : MirroringLove • Con : Mongo, Stability, .deb ● PRM ? ● https://github.com/ImmobilienScout24/yum-repo-se ?
  45. 45. Repository Management
  46. 46. A pipeline ● Checkout code ● Upload to Repo ● Syntax ● Deploy on Test ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package
  47. 47. mc-package
  48. 48. Repos are SLOW ● Createrepo is slow. ● Pulp is slow ● Bypass repos , upload straight to appropriate PuppetMaster ● Upload to repo for rebootstrapping
  49. 49. A pipeline ● Checkout code ● Upload to Repo ● Syntax ● Deploy on Test ● Style ● Check Puppetruns ● Code Coverage ● Check Icinga ● Tests ● Promote to UAT ● Build ● More Tests ● Package
  50. 50. Jenkins Promotion
  51. 51. Done ? ● Close the feedback loop, ● Send metric on deployment echo "deployed.$package_name 1 `date +%s`" > /dev/tcp/<%= graphite_host %>/2003
  52. 52. Contact Kris Buytaert Kris.Buytaert@inuits.be Further Reading @krisbuytaert http://www.krisbuytaert.be/blog/ http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221

×