FlossUK 2014 talk by Kris Buytaert about Continuous Delivery off your Puppet Manifests and Infrastructure.

1. Continuous Delivery of yourContinuous Delivery of your
infrastructureinfrastructure
Kris Buytaert
@krisbuytaert

2. Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Some books, some papers, some blogsSome books, some papers, some blogs
● Evangelizing devopsEvangelizing devops

3. Todays GoalsTodays Goals
● A reproducable way to deploy and upgradeA reproducable way to deploy and upgrade
(y)our infrastructure(y)our infrastructure
● AutomaticallyAutomatically
● FastFast
● ConsistentConsistent
● ContinuouslyContinuously

4. What's this devops thingWhat's this devops thing
anyhow ?anyhow ?

5. C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

6. devops (<)> continuous delilverydevops (<)> continuous delilvery

7. NirvanaNirvana
An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, from
infrastructure, data and configuration management toinfrastructure, data and configuration management to
business.business.
Through automation of the build, deployment, and testingThrough automation of the build, deployment, and testing
process, and improved collaboration between developers,process, and improved collaboration between developers,
testers, and operations, delivery teams can get changestesters, and operations, delivery teams can get changes
released in a matter of hours — sometimes even minutes–noreleased in a matter of hours — sometimes even minutes–no
matter what the size of a project or the complexity of its codematter what the size of a project or the complexity of its code
base.base.
Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble

9. How many times a day ?How many times a day ?
● 10 @ Flickr10 @ Flickr
● Deployments used to be painDeployments used to be pain
● Nobody dared to deploy a siteNobody dared to deploy a site
● Practice makes perfectPractice makes perfect
● Knowing you can vs constantly doing itKnowing you can vs constantly doing it

10. " Our job as engineers (and ops, dev-ops, QA," Our job as engineers (and ops, dev-ops, QA,
support, everyone in the company actually) is tosupport, everyone in the company actually) is to
enable the business goals. We strongly feel thatenable the business goals. We strongly feel that
in order to do that you must havein order to do that you must have the ability tothe ability to
deploy code quickly and safelydeploy code quickly and safely. Even if the. Even if the
business goals are to deploy strongly QA’d codebusiness goals are to deploy strongly QA’d code
once a month at 3am (it’s not for us, we push allonce a month at 3am (it’s not for us, we push all
the time), having a reliable and easy deploymentthe time), having a reliable and easy deployment
should beshould be non-negotiablenon-negotiable."."
Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

11. For years we've tolerated humans to makeFor years we've tolerated humans to make
structural manual changes to the infrastructurestructural manual changes to the infrastructure
our critical applications are running on.our critical applications are running on.
Whilst at the same time demanding those criticalWhilst at the same time demanding those critical
applications to go through rigid test scenarios.applications to go through rigid test scenarios.
Who let this happen ?Who let this happen ?

12. Infrastructure as CodeInfrastructure as Code
● Treat configuration automation as codeTreat configuration automation as code
● Development best practicesDevelopment best practices
•
Model your infrastructureModel your infrastructure
•
Version your cookbooks / manifestsVersion your cookbooks / manifests
•
Test your cookbooks/ manifestsTest your cookbooks/ manifests
•
Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra
● Model your infrastructureModel your infrastructure
● A working service = automated ( Application Code + InfrastructureA working service = automated ( Application Code + Infrastructure
Code + Security + Monitoring )Code + Security + Monitoring )

13. Version all the thingsVersion all the things
No more excuses !No more excuses !• Source code ApplicationSource code Application
• Source code InfrastructureSource code Infrastructure
• BuildsBuilds
• TestsTests
• PipelinesPipelines
• ScriptsScripts
• DocumentationDocumentation
• Monitoring scriptsMonitoring scripts

14. A random projectA random project
[sdog@mine vagrant-graphite]$ ls
manifests modules README TODO Vagrantfile
[sdog@mine vagrant-graphite]$ tree -dL 2
.
├── manifests
│ └── hosts
└── modules
├── apache
├── collectd
├── graphite
├── jmxtrans
├── logster
├── statsd
└── tattle
10 directories

15. Software ReleaseSoftware Release
management is not amanagement is not a
solved problemsolved problem

16. Use Git SubmodulesUse Git Submodules
● Basic git,Basic git,
● No extra tools requiredNo extra tools required
Integrates with other projects too.Integrates with other projects too.
(No need for *-librarian etc ..)(No need for *-librarian etc ..)

17. Continuous IntegrationContinuous Integration
Continuous integration (CI) is the practice, in software engineering, of mergingContinuous integration (CI) is the practice, in software engineering, of merging
all developer working copies with a shared mainline several times a day. It wasall developer working copies with a shared mainline several times a day. It was
first named and proposed as part of extreme programming (XP). Its main aim isfirst named and proposed as part of extreme programming (XP). Its main aim is
to prevent integration problems, referred to as "integration hell"to prevent integration problems, referred to as "integration hell"
(WikiPedia)(WikiPedia)
Does the app you are deploying still work ?Does the app you are deploying still work ?
Did you break your puppet / chef code ?Did you break your puppet / chef code ?

18. JenkinsJenkins
● Open Source Continuous Integration ServerOpen Source Continuous Integration Server
● A zillion plugins (400)A zillion plugins (400)
● Have developers build stable and deployableHave developers build stable and deployable
codecode
● Test Infra codeTest Infra code

19. Jenkins PipelineJenkins Pipeline

20. What's in your Pipeline ?What's in your Pipeline ?

21. A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage

22. Syntax and StyleSyntax and Style
● Initially ,Initially ,
all code, all the timeall code, all the time
● Now,Now,
only the changed codeonly the changed code
● Why not in post Commit Hooks ?Why not in post Commit Hooks ?

23. Package all the thingsPackage all the things

24. Artifacts:Artifacts:
● Tested artifcts that go trough a pipelineTested artifcts that go trough a pipeline
application code,application code,
Infra codeInfra code
metadatametadata
teststests

25. Why ops like to packageWhy ops like to package
● Packages give you featuresPackages give you features
•Consistency, security, dependenciesConsistency, security, dependencies
● Uniquely identify where files come fromUniquely identify where files come from
•Package or cfg-mgmtPackage or cfg-mgmt
● Source repo not always availableSource repo not always available
•Firewall / Cloud etc ..Firewall / Cloud etc ..
● Weird deployment locations , no easy accessWeird deployment locations , no easy access
● Little overhead when you automateLittle overhead when you automate

26. Jordan Sissel is a Hero !Jordan Sissel is a Hero !

27. #packaginlove#packaginlove

28. It's not really packagingIt's not really packaging
•
It's an immutable branchIt's an immutable branch
•
It's a tracable release artefactIt's a tracable release artefact

29. https://github.com/vStone/jenkins-https://github.com/vStone/jenkins-
puppet-scriptspuppet-scripts
● TestsTests
● Packages full tree inPackages full tree in
//
etc/puppet/environmetc/puppet/environm
ents/$environment/ents/$environment/

30. A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo

31. Repository ManagementRepository Management
● PulpPulp
•
Pro : MirroringLovePro : MirroringLove
•
Con : Mongo, Stability, .debCon : Mongo, Stability, .deb
● PRM ?PRM ?
● https://github.com/ImmobilienScout24/yum-repo-sehttps://github.com/ImmobilienScout24/yum-repo-se
??

32. Repository ManagementRepository Management

33. A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test

34. mc-packagemc-package

35. Repos are SLOWRepos are SLOW
● Createrepo is slow.Createrepo is slow.
● Pulp is slowPulp is slow
● Bypass repos , upload straight to appropriateBypass repos , upload straight to appropriate
PuppetMasterPuppetMaster
● Upload to repo for rebootstrappingUpload to repo for rebootstrapping

36. A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
● Check PuppetrunsCheck Puppetruns
● Check MonitoringCheck Monitoring
●

37. Testing = MonitoringTesting = Monitoring
● Deploy a host,Deploy a host,
● Add it to the monitoring frameworkAdd it to the monitoring framework
● Add collection toolsAdd collection tools
● Add check definitionsAdd check definitions
● Update the monitoring tool configUpdate the monitoring tool config
FULLY AUTOMATEDFULLY AUTOMATED

38. e.g. Stored Configse.g. Stored Configs

39. Collection and ExportCollection and Export
Export :Export :
@@resource {@@resource {
... }... }
Collect:Collect:
Resource <<| query |Resource <<| query |
>>>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node cleanpuppet node clean

40. Exporting and CollectingExporting and Collecting

41. Monitoring a VhostMonitoring a Vhost

42. A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
● Check PuppetrunsCheck Puppetruns
● Check MonitoringCheck Monitoring
● Promote to UATPromote to UAT

43. Jenkins PromotionJenkins Promotion

44. Done ?Done ?
● Close the feedback loop,Close the feedback loop,
● Send metric on deploymentSend metric on deployment
echo "deployed.$package_name 1 `date +%s`"echo "deployed.$package_name 1 `date +%s`"
> /dev/tcp/<%= graphite_host %>/2003> /dev/tcp/<%= graphite_host %>/2003

45. ContactContact
Kris BuytaertKris Buytaert
Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Duboistraat 50Duboistraat 50
2060 Antwerpen2060 Antwerpen
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221