Continuous Delivery of (y)our infrastructure.

2,472 views

Published on

FlossUK 2014 talk by Kris Buytaert about Continuous Delivery off your Puppet Manifests and Infrastructure.

Published in: Technology
  • Be the first to comment

Continuous Delivery of (y)our infrastructure.

  1. 1. Continuous Delivery of yourContinuous Delivery of your infrastructureinfrastructure Kris Buytaert @krisbuytaert
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Some books, some papers, some blogsSome books, some papers, some blogs ● Evangelizing devopsEvangelizing devops
  3. 3. Todays GoalsTodays Goals ● A reproducable way to deploy and upgradeA reproducable way to deploy and upgrade (y)our infrastructure(y)our infrastructure ● AutomaticallyAutomatically ● FastFast ● ConsistentConsistent ● ContinuouslyContinuously
  4. 4. What's this devops thingWhat's this devops thing anyhow ?anyhow ?
  5. 5. C(L)AMSC(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● MeasurementMeasurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  6. 6. devops (<)> continuous delilverydevops (<)> continuous delilvery
  7. 7. NirvanaNirvana An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management toinfrastructure, data and configuration management to business.business. Through automation of the build, deployment, and testingThrough automation of the build, deployment, and testing process, and improved collaboration between developers,process, and improved collaboration between developers, testers, and operations, delivery teams can get changestesters, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–noreleased in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its codematter what the size of a project or the complexity of its code base.base. Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble
  8. 8. How many times a day ?How many times a day ? ● 10 @ Flickr10 @ Flickr ● Deployments used to be painDeployments used to be pain ● Nobody dared to deploy a siteNobody dared to deploy a site ● Practice makes perfectPractice makes perfect ● Knowing you can vs constantly doing itKnowing you can vs constantly doing it
  9. 9. " Our job as engineers (and ops, dev-ops, QA," Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is tosupport, everyone in the company actually) is to enable the business goals. We strongly feel thatenable the business goals. We strongly feel that in order to do that you must havein order to do that you must have the ability tothe ability to deploy code quickly and safelydeploy code quickly and safely. Even if the. Even if the business goals are to deploy strongly QA’d codebusiness goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push allonce a month at 3am (it’s not for us, we push all the time), having a reliable and easy deploymentthe time), having a reliable and easy deployment should beshould be non-negotiablenon-negotiable."." Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
  10. 10. For years we've tolerated humans to makeFor years we've tolerated humans to make structural manual changes to the infrastructurestructural manual changes to the infrastructure our critical applications are running on.our critical applications are running on. Whilst at the same time demanding those criticalWhilst at the same time demanding those critical applications to go through rigid test scenarios.applications to go through rigid test scenarios. Who let this happen ?Who let this happen ?
  11. 11. Infrastructure as CodeInfrastructure as Code ● Treat configuration automation as codeTreat configuration automation as code ● Development best practicesDevelopment best practices • Model your infrastructureModel your infrastructure • Version your cookbooks / manifestsVersion your cookbooks / manifests • Test your cookbooks/ manifestsTest your cookbooks/ manifests • Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra ● Model your infrastructureModel your infrastructure ● A working service = automated ( Application Code + InfrastructureA working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Code + Security + Monitoring )
  12. 12. Version all the thingsVersion all the things No more excuses !No more excuses !• Source code ApplicationSource code Application • Source code InfrastructureSource code Infrastructure • BuildsBuilds • TestsTests • PipelinesPipelines • ScriptsScripts • DocumentationDocumentation • Monitoring scriptsMonitoring scripts
  13. 13. A random projectA random project [sdog@mine vagrant-graphite]$ ls manifests modules README TODO Vagrantfile [sdog@mine vagrant-graphite]$ tree -dL 2 . ├── manifests │ └── hosts └── modules ├── apache ├── collectd ├── graphite ├── jmxtrans ├── logster ├── statsd └── tattle 10 directories
  14. 14. Software ReleaseSoftware Release management is not amanagement is not a solved problemsolved problem
  15. 15. Use Git SubmodulesUse Git Submodules ● Basic git,Basic git, ● No extra tools requiredNo extra tools required Integrates with other projects too.Integrates with other projects too. (No need for *-librarian etc ..)(No need for *-librarian etc ..)
  16. 16. Continuous IntegrationContinuous Integration Continuous integration (CI) is the practice, in software engineering, of mergingContinuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. It wasall developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Its main aim isfirst named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as "integration hell"to prevent integration problems, referred to as "integration hell" (WikiPedia)(WikiPedia) Does the app you are deploying still work ?Does the app you are deploying still work ? Did you break your puppet / chef code ?Did you break your puppet / chef code ?
  17. 17. JenkinsJenkins ● Open Source Continuous Integration ServerOpen Source Continuous Integration Server ● A zillion plugins (400)A zillion plugins (400) ● Have developers build stable and deployableHave developers build stable and deployable codecode ● Test Infra codeTest Infra code
  18. 18. Jenkins PipelineJenkins Pipeline
  19. 19. What's in your Pipeline ?What's in your Pipeline ?
  20. 20. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage
  21. 21. Syntax and StyleSyntax and Style ● Initially ,Initially , all code, all the timeall code, all the time ● Now,Now, only the changed codeonly the changed code ● Why not in post Commit Hooks ?Why not in post Commit Hooks ?
  22. 22. Package all the thingsPackage all the things
  23. 23. Artifacts:Artifacts: ● Tested artifcts that go trough a pipelineTested artifcts that go trough a pipeline application code,application code, Infra codeInfra code metadatametadata teststests
  24. 24. Why ops like to packageWhy ops like to package ● Packages give you featuresPackages give you features •Consistency, security, dependenciesConsistency, security, dependencies ● Uniquely identify where files come fromUniquely identify where files come from •Package or cfg-mgmtPackage or cfg-mgmt ● Source repo not always availableSource repo not always available •Firewall / Cloud etc ..Firewall / Cloud etc .. ● Weird deployment locations , no easy accessWeird deployment locations , no easy access ● Little overhead when you automateLittle overhead when you automate
  25. 25. Jordan Sissel is a Hero !Jordan Sissel is a Hero !
  26. 26. #packaginlove#packaginlove
  27. 27. It's not really packagingIt's not really packaging • It's an immutable branchIt's an immutable branch • It's a tracable release artefactIt's a tracable release artefact
  28. 28. https://github.com/vStone/jenkins-https://github.com/vStone/jenkins- puppet-scriptspuppet-scripts ● TestsTests ● Packages full tree inPackages full tree in // etc/puppet/environmetc/puppet/environm ents/$environment/ents/$environment/
  29. 29. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo
  30. 30. Repository ManagementRepository Management ● PulpPulp • Pro : MirroringLovePro : MirroringLove • Con : Mongo, Stability, .debCon : Mongo, Stability, .deb ● PRM ?PRM ? ● https://github.com/ImmobilienScout24/yum-repo-sehttps://github.com/ImmobilienScout24/yum-repo-se ??
  31. 31. Repository ManagementRepository Management
  32. 32. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test
  33. 33. mc-packagemc-package
  34. 34. Repos are SLOWRepos are SLOW ● Createrepo is slow.Createrepo is slow. ● Pulp is slowPulp is slow ● Bypass repos , upload straight to appropriateBypass repos , upload straight to appropriate PuppetMasterPuppetMaster ● Upload to repo for rebootstrappingUpload to repo for rebootstrapping
  35. 35. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test ● Check PuppetrunsCheck Puppetruns ● Check MonitoringCheck Monitoring ●
  36. 36. Testing = MonitoringTesting = Monitoring ● Deploy a host,Deploy a host, ● Add it to the monitoring frameworkAdd it to the monitoring framework ● Add collection toolsAdd collection tools ● Add check definitionsAdd check definitions ● Update the monitoring tool configUpdate the monitoring tool config FULLY AUTOMATEDFULLY AUTOMATED
  37. 37. e.g. Stored Configse.g. Stored Configs
  38. 38. Collection and ExportCollection and Export Export :Export : @@resource {@@resource { ... }... } Collect:Collect: Resource <<| query |Resource <<| query | >>>> Clean out nodes that dissapearClean out nodes that dissapear puppet node cleanpuppet node clean
  39. 39. Exporting and CollectingExporting and Collecting
  40. 40. Monitoring a VhostMonitoring a Vhost
  41. 41. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test ● Check PuppetrunsCheck Puppetruns ● Check MonitoringCheck Monitoring ● Promote to UATPromote to UAT
  42. 42. Jenkins PromotionJenkins Promotion
  43. 43. Done ?Done ? ● Close the feedback loop,Close the feedback loop, ● Send metric on deploymentSend metric on deployment echo "deployed.$package_name 1 `date +%s`"echo "deployed.$package_name 1 `date +%s`" > /dev/tcp/<%= graphite_host %>/2003> /dev/tcp/<%= graphite_host %>/2003
  44. 44. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ InuitsInuits Duboistraat 50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×