Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Pipeline as CodePipeline as Code
for yourfor your
IACIAC
Kris Buytaert
@krisbuytaert
Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling ...
Todays GoalsTodays Goals
● Infrastructure as CodeInfrastructure as Code
● A reproducable way to deploy and upgradeA reprod...
What's this devops thingWhat's this devops thing
anyhow ?anyhow ?
C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwa...
devops (<)> continuous deliverydevops (<)> continuous delivery
NirvanaNirvana
An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, fro...
How many times a day ?How many times a day ?
● 10 @ Flickr10 @ Flickr
● Deployments used to be painDeployments used to be ...
" Our job as engineers (and ops, dev-ops, QA," Our job as engineers (and ops, dev-ops, QA,
support, everyone in the compan...
For years we've tolerated humans to makeFor years we've tolerated humans to make
structural manual changes to the infrastr...
Infrastructure as CodeInfrastructure as Code
● Treat configuration automation as codeTreat configuration automation as cod...
Version all the thingsVersion all the things
No more excuses !No more excuses !• Source code ApplicationSource code Applic...
PuppetPuppet
● Open Source Config ManagementOpen Source Config Management ToolTool EcosystemEcosystem
● 10y old ; very mat...
Puppet in shortPuppet in short
● You define the desired state of your infraYou define the desired state of your infra
● Pu...
How Puppet WorksHow Puppet Works
● Puppet works by default on a server/agentPuppet works by default on a server/agent
mode...
A random projectA random project
[sdog@mine vagrant-graphite]$ ls
manifests modules README TODO Vagrantfile
[sdog@mine vag...
Manage the modulesManage the modules
● Insert ugly shell scriptInsert ugly shell script
● Librarian PuppetLibrarian Puppet...
Librarian Puppet / R10kLibrarian Puppet / R10k
● Hides complexity of submodulesHides complexity of submodules
● Easy if yo...
Software ReleaseSoftware Release
management is not amanagement is not a
solved problemsolved problem
Unless you understandUnless you understand
Git SubmodulesGit Submodules
● Basic git,Basic git,
● No extra tools requiredNo...
Continuous IntegrationContinuous Integration
Continuous integration (CI) is the practice, in software engineering, ofConti...
JenkinsJenkins
● Open Source Continuous Integration ServerOpen Source Continuous Integration Server
● A zillion plugins (4...
Jenkins PipelineJenkins Pipeline
What's in your Pipeline ?What's in your Pipeline ?
A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● ...
Syntax and StyleSyntax and Style
● Initially ,Initially ,
all code, all the timeall code, all the time
● Now,Now,
only the...
Package all the thingsPackage all the things
Artifacts:Artifacts:
● Tested artifacts that go through a pipelineTested artifacts that go through a pipeline
application ...
Why ops like to packageWhy ops like to package
● Packages give you featuresPackages give you features
• Consistency, secur...
Jordan Sissel is a Hero !Jordan Sissel is a Hero !
#packaginlove#packaginlove
It's not really packagingIt's not really packaging
•
It's an immutable branchIt's an immutable branch
•
It's a tracable re...
A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● ...
Repository ManagementRepository Management
● PulpPulp
•
Pro : MirroringLovePro : MirroringLove
•
Con : Mongo, Stability, ....
Repository ManagementRepository Management
A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● ...
Repos are SLOWRepos are SLOW
● Createrepo is slow.Createrepo is slow.
● Pulp is slowPulp is slow
● Bypass repos , upload s...
A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● ...
Testing = MonitoringTesting = Monitoring
● Deploy a host,Deploy a host,
● Add it to the monitoring frameworkAdd it to the ...
e.g. Stored Configse.g. Stored Configs
A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● ...
Jenkins PromotionJenkins Promotion
PipelinesPipelines
● Lots of themLots of them
● Similar ones, but not identical onesSimilar ones, but not identical ones
●...
Dirty ClickersDirty Clickers
Complex PipelinesComplex Pipelines
Scaling PipelinesScaling Pipelines
● Create a Pipeline,Create a Pipeline,
● For job in PipelineFor job in Pipeline
•
Creat...
Generating Jenkins PipelinesGenerating Jenkins Pipelines
● Template the XMLTemplate the XML
● Put it in PuppetPut it in Pu...
JenkinsJobBuilderJenkinsJobBuilder
● First Usable AttemptFirst Usable Attempt
● Python BasedPython Based
● Openstack Commu...
PipelineDSLPipelineDSL
● JenkinsfileJenkinsfile
● Inside a repoInside a repo
● GroovyGroovy
● Limited functionalityLimited...
Jenkins Job DSLJenkins Job DSL
● GroovyGroovy
● FlexibleFlexible
● Well SupportedWell Supported
● Suitable for more comple...
SeedjobsSeedjobs
● GroovyGroovy
● GitGit
● Rebuild jobs on commitRebuild jobs on commit
● Projects in foldersProjects in f...
Larger CI StacksLarger CI Stacks
● Generate Pipelines / Jobs based on config files ,Generate Pipelines / Jobs based on con...
Testing Multiple VersionsTesting Multiple Versions
● Initial stage tests code on multipleInitial stage tests code on multi...
Problems solvedProblems solved
● One job per task, no reuse of jobs with differentOne job per task, no reuse of jobs with ...
Stop Clicking , Write CodeStop Clicking , Write Code
ContactContact
Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu
Further ReadingFurther Reading
@...
Pipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as Code
Upcoming SlideShare
Loading in …5
×

Pipeline as code for your infrastructure as Code

642 views

Published on

My Open Source Summit Europe talk about Pipeline as Code for Your Infrastructure as code

Published in: Technology
  • Be the first to comment

Pipeline as code for your infrastructure as Code

  1. 1. Pipeline as CodePipeline as Code for yourfor your IACIAC Kris Buytaert @krisbuytaert
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Some books, some papers, some blogsSome books, some papers, some blogs ● Evangelizing devopsEvangelizing devops
  3. 3. Todays GoalsTodays Goals ● Infrastructure as CodeInfrastructure as Code ● A reproducable way to deploy and upgradeA reproducable way to deploy and upgrade /etc/puppet/etc/puppet ● With 10+ environments inWith 10+ environments in /etc/puppet/environments/etc/puppet/environments ● AutomaticallyAutomatically ● FastFast ● ConsistentConsistent ● ContinuouslyContinuously
  4. 4. What's this devops thingWhat's this devops thing anyhow ?anyhow ?
  5. 5. C(L)AMSC(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● MeasurementMeasurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  6. 6. devops (<)> continuous deliverydevops (<)> continuous delivery
  7. 7. NirvanaNirvana An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management toinfrastructure, data and configuration management to business.business. Through automation of the build, deployment, and testingThrough automation of the build, deployment, and testing process, and improved collaboration between developers,process, and improved collaboration between developers, testers, and operations, delivery teams can get changestesters, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–noreleased in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its codematter what the size of a project or the complexity of its code base.base. Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble
  8. 8. How many times a day ?How many times a day ? ● 10 @ Flickr10 @ Flickr ● Deployments used to be painDeployments used to be pain ● Nobody dared to deploy a siteNobody dared to deploy a site ● Practice makes perfectPractice makes perfect ● Knowing you can vs constantly doing itKnowing you can vs constantly doing it
  9. 9. " Our job as engineers (and ops, dev-ops, QA," Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is tosupport, everyone in the company actually) is to enable the business goals. We strongly feel thatenable the business goals. We strongly feel that in order to do that you must havein order to do that you must have the ability tothe ability to deploy code quickly and safelydeploy code quickly and safely. Even if the. Even if the business goals are to deploy strongly QA’d codebusiness goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push allonce a month at 3am (it’s not for us, we push all the time), having a reliable and easythe time), having a reliable and easy deployment should bedeployment should be non-negotiablenon-negotiable."." Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
  10. 10. For years we've tolerated humans to makeFor years we've tolerated humans to make structural manual changes to the infrastructurestructural manual changes to the infrastructure our critical applications are running on.our critical applications are running on. Whilst at the same time demanding those criticalWhilst at the same time demanding those critical applications to go through rigid test scenarios.applications to go through rigid test scenarios. Who let this happen ?Who let this happen ?
  11. 11. Infrastructure as CodeInfrastructure as Code ● Treat configuration automation as codeTreat configuration automation as code ● Development best practicesDevelopment best practices • Model your infrastructureModel your infrastructure • Version your cookbooks / manifestsVersion your cookbooks / manifests • Test your cookbooks/ manifestsTest your cookbooks/ manifests • Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra ● Model your infrastructureModel your infrastructure ● A working service = automated ( Application Code +A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Infrastructure Code + Security + Monitoring ) ● PS. Converting Bash to Yaml != IACPS. Converting Bash to Yaml != IAC
  12. 12. Version all the thingsVersion all the things No more excuses !No more excuses !• Source code ApplicationSource code Application • Source code InfrastructureSource code Infrastructure • BuildsBuilds • TestsTests • PipelinesPipelines • ScriptsScripts • DocumentationDocumentation • Monitoring scriptsMonitoring scripts
  13. 13. PuppetPuppet ● Open Source Config ManagementOpen Source Config Management ToolTool EcosystemEcosystem ● 10y old ; very mature10y old ; very mature ● Ruby, Clojure, Jruby, C++, ...Ruby, Clojure, Jruby, C++, ... ● Large communityLarge community ● Lots of open source modules (code shared byLots of open source modules (code shared by other people to automate things)other people to automate things)
  14. 14. Puppet in shortPuppet in short ● You define the desired state of your infraYou define the desired state of your infra ● Puppet looks at the current state and takesPuppet looks at the current state and takes actions to get to the final stateactions to get to the final state ● You have the report of what has beenYou have the report of what has been changed/errored/…changed/errored/… ● It is declarativeIt is declarative
  15. 15. How Puppet WorksHow Puppet Works ● Puppet works by default on a server/agentPuppet works by default on a server/agent modelmodel ● Agents fetch their “catalog” (= final desiredAgents fetch their “catalog” (= final desired state) from the master and apply that locallystate) from the master and apply that locally ● Then they report back to the serverThen they report back to the server ● The catalog is computed depending on “facts”The catalog is computed depending on “facts” the machine sends to the server, user inputthe machine sends to the server, user input (parameters) and modules (source code)(parameters) and modules (source code)
  16. 16. A random projectA random project [sdog@mine vagrant-graphite]$ ls manifests modules README TODO Vagrantfile [sdog@mine vagrant-graphite]$ tree -dL 2 . ├── manifests │   └── hosts └── modules ├── apache ├── collectd ├── graphite ├── jmxtrans ├── logster ├── statsd └── tattle 10 directories
  17. 17. Manage the modulesManage the modules ● Insert ugly shell scriptInsert ugly shell script ● Librarian PuppetLibrarian Puppet ● R10kR10k ● ........
  18. 18. Librarian Puppet / R10kLibrarian Puppet / R10k ● Hides complexity of submodulesHides complexity of submodules ● Easy if you use Forge ModulesEasy if you use Forge Modules • Does anyone ?Does anyone ? • Do you trust the internet to be around ?Do you trust the internet to be around ? ● Branching is considered the antipattern in aBranching is considered the antipattern in a CI/CD EnvironmentCI/CD Environment ● Librarian = Old English for “can't useLibrarian = Old English for “can't use submodules”submodules” ● And hmm... which customer uses which patched version again ?And hmm... which customer uses which patched version again ?
  19. 19. Software ReleaseSoftware Release management is not amanagement is not a solved problemsolved problem
  20. 20. Unless you understandUnless you understand Git SubmodulesGit Submodules ● Basic git,Basic git, ● No extra tools requiredNo extra tools required Integrates with other projects too.Integrates with other projects too. (No need for *-librarian etc ..)(No need for *-librarian etc ..)
  21. 21. Continuous IntegrationContinuous Integration Continuous integration (CI) is the practice, in software engineering, ofContinuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times amerging all developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Itsday. It was first named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as "integration hell"main aim is to prevent integration problems, referred to as "integration hell" (WikiPedia)(WikiPedia) Does the app you are deploying still work ?Does the app you are deploying still work ? Did you break your puppet / chef code ?Did you break your puppet / chef code ?
  22. 22. JenkinsJenkins ● Open Source Continuous Integration ServerOpen Source Continuous Integration Server ● A zillion plugins (400)A zillion plugins (400) ● Have developers build stable and deployableHave developers build stable and deployable codecode ● Test Infra codeTest Infra code
  23. 23. Jenkins PipelineJenkins Pipeline
  24. 24. What's in your Pipeline ?What's in your Pipeline ?
  25. 25. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage
  26. 26. Syntax and StyleSyntax and Style ● Initially ,Initially , all code, all the timeall code, all the time ● Now,Now, only the changed codeonly the changed code ● Why not in post Commit Hooks ?Why not in post Commit Hooks ?
  27. 27. Package all the thingsPackage all the things
  28. 28. Artifacts:Artifacts: ● Tested artifacts that go through a pipelineTested artifacts that go through a pipeline application code,application code, Infra codeInfra code metadatametadata teststests
  29. 29. Why ops like to packageWhy ops like to package ● Packages give you featuresPackages give you features • Consistency, security, dependenciesConsistency, security, dependencies ● Uniquely identify where files come fromUniquely identify where files come from •Package or cfg-mgmtPackage or cfg-mgmt ● Source repo not always availableSource repo not always available •Firewall / Cloud etc ..Firewall / Cloud etc .. ● Weird deployment locations , no easy accessWeird deployment locations , no easy access ● Little overhead when you automateLittle overhead when you automate
  30. 30. Jordan Sissel is a Hero !Jordan Sissel is a Hero !
  31. 31. #packaginlove#packaginlove
  32. 32. It's not really packagingIt's not really packaging • It's an immutable branchIt's an immutable branch • It's a tracable release artefactIt's a tracable release artefact
  33. 33. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo
  34. 34. Repository ManagementRepository Management ● PulpPulp • Pro : MirroringLovePro : MirroringLove • Con : Mongo, Stability, .debCon : Mongo, Stability, .deb ● Aptly (deb only)Aptly (deb only) ● Prm (missing snapshot features for .rpm)Prm (missing snapshot features for .rpm)
  35. 35. Repository ManagementRepository Management
  36. 36. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test
  37. 37. Repos are SLOWRepos are SLOW ● Createrepo is slow.Createrepo is slow. ● Pulp is slowPulp is slow ● Bypass repos , upload straight to appropriateBypass repos , upload straight to appropriate PuppetMasterPuppetMaster ● Upload to repo for rebootstrappingUpload to repo for rebootstrapping
  38. 38. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test ● Check PuppetrunsCheck Puppetruns ● Check MonitoringCheck Monitoring
  39. 39. Testing = MonitoringTesting = Monitoring ● Deploy a host,Deploy a host, ● Add it to the monitoring frameworkAdd it to the monitoring framework ● Add collection toolsAdd collection tools ● Add check definitionsAdd check definitions ● Update the monitoring tool configUpdate the monitoring tool config FULLY AUTOMATEDFULLY AUTOMATED
  40. 40. e.g. Stored Configse.g. Stored Configs
  41. 41. A pipelineA pipeline ● Checkout codeCheckout code ● SyntaxSyntax ● StyleStyle ● Code CoverageCode Coverage ● TestsTests ● BuildBuild ● More TestsMore Tests ● PackagePackage ● Upload to RepoUpload to Repo ● Deploy on TestDeploy on Test ● Check PuppetrunsCheck Puppetruns ● Check MonitoringCheck Monitoring ● Promote to UATPromote to UAT
  42. 42. Jenkins PromotionJenkins Promotion
  43. 43. PipelinesPipelines ● Lots of themLots of them ● Similar ones, but not identical onesSimilar ones, but not identical ones ● One project = different deploymentOne project = different deployment targetstargets ● People move teams expect samePeople move teams expect same patternspatterns ● Mostly unmaintanedMostly unmaintaned
  44. 44. Dirty ClickersDirty Clickers
  45. 45. Complex PipelinesComplex Pipelines
  46. 46. Scaling PipelinesScaling Pipelines ● Create a Pipeline,Create a Pipeline, ● For job in PipelineFor job in Pipeline • Create new Job Based on OldJobCreate new Job Based on OldJob ● Update One JobUpdate One Job ● Never refactor the restNever refactor the rest
  47. 47. Generating Jenkins PipelinesGenerating Jenkins Pipelines ● Template the XMLTemplate the XML ● Put it in PuppetPut it in Puppet • Worked for stable pipelinesWorked for stable pipelines • KindaKinda ● XML gets rewritten on the flyXML gets rewritten on the fly
  48. 48. JenkinsJobBuilderJenkinsJobBuilder ● First Usable AttemptFirst Usable Attempt ● Python BasedPython Based ● Openstack CommunityOpenstack Community ● Limited FunctionalityLimited Functionality ● Little Adoption / OpenstackLittle Adoption / Openstack Abandonned itAbandonned it
  49. 49. PipelineDSLPipelineDSL ● JenkinsfileJenkinsfile ● Inside a repoInside a repo ● GroovyGroovy ● Limited functionalityLimited functionality ● Initially buggyInitially buggy ● Popular for Easy tasksPopular for Easy tasks • (straight, no splits etc)(straight, no splits etc)
  50. 50. Jenkins Job DSLJenkins Job DSL ● GroovyGroovy ● FlexibleFlexible ● Well SupportedWell Supported ● Suitable for more complex PipelinesSuitable for more complex Pipelines https://jenkinsci.github.io/job-dsl-https://jenkinsci.github.io/job-dsl- plugin/plugin/
  51. 51. SeedjobsSeedjobs ● GroovyGroovy ● GitGit ● Rebuild jobs on commitRebuild jobs on commit ● Projects in foldersProjects in folders
  52. 52. Larger CI StacksLarger CI Stacks ● Generate Pipelines / Jobs based on config files ,Generate Pipelines / Jobs based on config files , ● Build librariesBuild libraries • CheckoutJobCheckoutJob • DeployJobDeployJob • PackageJobPackageJob ● Use Groovy / JobDSL to generate PipelineDSLUse Groovy / JobDSL to generate PipelineDSL
  53. 53. Testing Multiple VersionsTesting Multiple Versions ● Initial stage tests code on multipleInitial stage tests code on multiple versionsversions • e.g current puppet versione.g current puppet version • Next puppet versionNext puppet version • Bleeding Edge versionBleeding Edge version ● Only current version breaks buildOnly current version breaks build ● Goal = get all versions greenGoal = get all versions green
  54. 54. Problems solvedProblems solved ● One job per task, no reuse of jobs with differentOne job per task, no reuse of jobs with different parametersparameters ● All Jobs are in sync, update one = update allAll Jobs are in sync, update one = update all ● Centrally managed jobs (git)Centrally managed jobs (git)
  55. 55. Stop Clicking , Write CodeStop Clicking , Write Code
  56. 56. ContactContact Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ https://inuits.eu/https://inuits.eu/ InuitsInuits Essensteenweg 31Essensteenweg 31 BrasschaatBrasschaat BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×