Continous Delivery of your Infrastructure

265 views

Published on

devops, ci, puppet, jenkins,...
My 2016 NLUUG Talk

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
265
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
4
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Continous Delivery of your Infrastructure

  1. 1. Continuous Delivery ofContinuous Delivery of your Infrastructureyour Infrastructure Kris Buytaert @krisbuytaert
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @Consultant @inuits.euinuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Some books, some papers, some blogsSome books, some papers, some blogs ● Evangelizing devopsEvangelizing devops
  3. 3. Todays GoalsTodays Goals ● A reproducable way to deploy and upgradeA reproducable way to deploy and upgrade your infrastructureyour infrastructure ● AutomaticallyAutomatically ● FastFast ● ConsistentConsistent ● ContinuouslyContinuously
  4. 4. What's this devops thingWhat's this devops thing anyhow ?anyhow ?
  5. 5. C(L)AMSC(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● MeasurementMeasurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  6. 6. Whats in it for you ?Whats in it for you ? ● Faster time to marketFaster time to market • Features go live in hours vs yearsFeatures go live in hours vs years ● In a more safe (Secure)In a more safe (Secure) ● Reliable fashionReliable fashion • Fully automatedFully automated ● More happyMore happy {customers,developers,ops,managers,investors}{customers,developers,ops,managers,investors}
  7. 7. Less RiskLess Risk
  8. 8. devops (<)> continuous delilverydevops (<)> continuous delilvery
  9. 9. NirvanaNirvana An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management toinfrastructure, data and configuration management to business.business. Through automation of the build, deployment, and testingThrough automation of the build, deployment, and testing process, and improved collaboration between developers,process, and improved collaboration between developers, testers, and operations, delivery teams can get changestesters, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–noreleased in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its codematter what the size of a project or the complexity of its code base.base. Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble
  10. 10. How many times a day ?How many times a day ? ● 10 @ Flickr10 @ Flickr ● Deployments used to be painDeployments used to be pain ● Nobody dared to deploy a siteNobody dared to deploy a site ● Practice makes perfectPractice makes perfect ● Knowing you can vs constantly doing itKnowing you can vs constantly doing it
  11. 11. " Our job as engineers (and ops, dev-ops, QA," Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is tosupport, everyone in the company actually) is to enable the business goals. We strongly feel thatenable the business goals. We strongly feel that in order to do that you must havein order to do that you must have the ability tothe ability to deploy code quickly and safelydeploy code quickly and safely. Even if the. Even if the business goals are to deploy strongly QA’d codebusiness goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push allonce a month at 3am (it’s not for us, we push all the time), having a reliable and easy deploymentthe time), having a reliable and easy deployment should beshould be non-negotiablenon-negotiable."." Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
  12. 12. For years we've tolerated humans to makeFor years we've tolerated humans to make structural manual changes to the infrastructurestructural manual changes to the infrastructure our critical applications are running on.our critical applications are running on. Whilst at the same time demanding those criticalWhilst at the same time demanding those critical applications to go through rigid test scenarios.applications to go through rigid test scenarios. Who let this happen ?Who let this happen ?
  13. 13. Why start with infra ?Why start with infra ? ● LearnLearn ● Be able to support your peersBe able to support your peers ● It's a test requirement anyhowIt's a test requirement anyhow
  14. 14. Automate all the thingsAutomate all the things ● BuildBuild • reproducable builds are undiscussablereproducable builds are undiscussable ● TestTest • testing reduces risktesting reduces risk • automate deployments of your test infraautomate deployments of your test infra ● DeployDeploy • Infrastructure as CodeInfrastructure as Code • 100% automation100% automation • Can you rebuild your infrastructure ?Can you rebuild your infrastructure ?
  15. 15. Use VersionUse Version Control,Control, No ExcusesNo Excuses Also forAlso for scripts/config/cookbooksscripts/config/cookbooks ,manifests,etc,manifests,etc
  16. 16. Infrastructure as CodeInfrastructure as Code ● Treat configuration automation as codeTreat configuration automation as code ● Development best practicesDevelopment best practices • Model your infrastructureModel your infrastructure • Version your cookbooks / manifestsVersion your cookbooks / manifests • Test your cookbooks/ manifestsTest your cookbooks/ manifests • Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra ● Model your infrastructureModel your infrastructure ● A working service = automated ( Application Code + InfrastructureA working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Code + Security + Monitoring ) ● IAC -ne scriptingIAC -ne scripting
  17. 17. Release ManagementRelease Management ● What parts go in the build ?What parts go in the build ? ● What are the dependencies ?What are the dependencies ? ● What versions work together ?What versions work together ? ● Every frameworks “invents” it's own.Every frameworks “invents” it's own. • None workNone work
  18. 18. Release ManagementRelease Management ● Git Submodules !Git Submodules ! ● Submodules and you will never need a ReleaseSubmodules and you will never need a Release Management tool again , everManagement tool again , ever ● Tool independent pattern,Tool independent pattern, • PuppetPuppet • DrupalDrupal • SymfonySymfony • ……..
  19. 19. CI ToolsCI Tools ● HudsonHudson ● JenkinsJenkins A zillion pluginsA zillion plugins ● Go (Cruise Control)?Go (Cruise Control)? ● Travis ?Travis ? ● Also test your (Puppet/Chef/CFengine)Also test your (Puppet/Chef/CFengine)
  20. 20. Build PipelinesBuild Pipelines
  21. 21. JenkinsJenkins ● Open Source Continuous Integration ServerOpen Source Continuous Integration Server ● A zillion plugins (400)A zillion plugins (400) ● Have developers build stable and deployableHave developers build stable and deployable codecode ● Test Infra codeTest Infra code
  22. 22. Jenkins PipelineJenkins Pipeline
  23. 23. It gets harderIt gets harder ● Try to fully automate JenkinsTry to fully automate Jenkins • Including plugin configIncluding plugin config • Including all the jobsIncluding all the jobs • Try templating the jobsTry templating the jobs • JJBJJB • Job DSLJob DSL • ……
  24. 24. What's in your Pipeline ?What's in your Pipeline ?
  25. 25. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage
  26. 26. Syntax and StyleSyntax and Style ● Initially ,Initially , all code, all the timeall code, all the time ● Now,Now, only the changed codeonly the changed code ● Why not in post Commit Hooks ?Why not in post Commit Hooks ?
  27. 27. Package all the thingsPackage all the things
  28. 28. Why ops like to packageWhy ops like to package ● Packages give you featuresPackages give you features •Consistency, security, dependenciesConsistency, security, dependencies ● Uniquely identify where files come fromUniquely identify where files come from •Package or cfg-mgmtPackage or cfg-mgmt ● Source repo not always availableSource repo not always available •Firewall / Cloud etc ..Firewall / Cloud etc .. ● Weird deployment locations , no easy accessWeird deployment locations , no easy access ● Little overhead when you automateLittle overhead when you automate
  29. 29. #packaginlove#packaginlove
  30. 30. SaySay thanks tothanks to JordanJordan SisselSissel
  31. 31. It's not really packagingIt's not really packaging • It's an immutable branchIt's an immutable branch • It's a tracable release artefactIt's a tracable release artefact
  32. 32. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo
  33. 33. Repository ManagementRepository Management ● PulpPulp • Pro : MirroringLovePro : MirroringLove • Con : Mongo, Stability, .debCon : Mongo, Stability, .deb ● PRM ?PRM ? ● https://github.com/ImmobilienScout24/yum-repo-sehttps://github.com/ImmobilienScout24/yum-repo-se ??
  34. 34. Repository ManagementRepository Management
  35. 35. Deploy SoftwareDeploy Software ● Repo per environmentRepo per environment ● Package {'mysoftware': ensure => latest}Package {'mysoftware': ensure => latest} • You control the repos !You control the repos ! ● Strict versioning in config ?Strict versioning in config ? Ensure => '0.98.4'Ensure => '0.98.4'
  36. 36. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test
  37. 37. Repos are SLOWRepos are SLOW ● Createrepo is slow.Createrepo is slow. ● Pulp is slowPulp is slow ● Bypass repos , upload straight to appropriateBypass repos , upload straight to appropriate PuppetMasterPuppetMaster ● Upload to repo for rebootstrappingUpload to repo for rebootstrapping
  38. 38. OrchestrationOrchestration● Manage 1000 nodes,Manage 1000 nodes, ● TriggerTrigger • UpgradesUpgrades • Config RunsConfig Runs • Service ChangesService Changes ● Think :Think : • McollectiveMcollective • ConsulConsul • AnsibleAnsible • RundeckRundeck
  39. 39. Orchestration 2Orchestration 2ndnd gengen AkaAka ChoreographyChoreography ● While ....While .... ● First install XFirst install X ● When it is ready configure YWhen it is ready configure Y ● Then notify ZThen notify Z ● Think : Zookeeper, Serf , JujuThink : Zookeeper, Serf , Juju R EA R C H ITE T !
  40. 40. Testing your infraTesting your infra ● One Icinga check at the timeOne Icinga check at the time ● Roll out new (puppet) codeRoll out new (puppet) code ● Trigger puppet runs on environmentTrigger puppet runs on environment • Load on Puppetmaster , but more on AnsibleLoad on Puppetmaster , but more on Ansible ● Trigger check icingaTrigger check icinga
  41. 41. Reuse tests for monitoringReuse tests for monitoring ● Unit tests vs Acceptance TestsUnit tests vs Acceptance Tests ● Your monitoring platformYour monitoring platform • also has a test / uat / prod flowalso has a test / uat / prod flow • Is fully automatedIs fully automated ● Keeping UAT green is a mustKeeping UAT green is a must • This is where you learn PROD behaviourThis is where you learn PROD behaviour
  42. 42. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test ● Check PuppetrunsCheck Puppetruns ● Check IcingaCheck Icinga ● Promote to UATPromote to UAT
  43. 43. Jenkins PromotionJenkins Promotion
  44. 44. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test ● Check PuppetrunsCheck Puppetruns ● Check IcingaCheck Icinga ● Promote to UATPromote to UAT ● Test even more ,Test even more , security etc..security etc.. ● Promote to PRODPromote to PROD
  45. 45. Attack yourselve onAttack yourselve on every buildevery build ● Gauntlt , write security testsGauntlt , write security tests ● Vulnerability scans (Arachni)Vulnerability scans (Arachni) ● Content Scanner (DIRB)Content Scanner (DIRB) ● …… ● https://github.com/garethr/pentesting-playgroundhttps://github.com/garethr/pentesting-playground
  46. 46. Done ?Done ? ● Close the feedback loop,Close the feedback loop, ● Send metric on deploymentSend metric on deployment echo "deployed.$package_name 1 `date +%s`"echo "deployed.$package_name 1 `date +%s`" > /dev/tcp/<%= graphite_host %>/2003> /dev/tcp/<%= graphite_host %>/2003
  47. 47. No moreNo more ● Hacking in productionHacking in production ● Can't change thisCan't change this ● I don't have monitoring for thisI don't have monitoring for this
  48. 48. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ InuitsInuits Essensteenweg 31Essensteenweg 31 BrasschaatBrasschaat BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×