Continuous Delivery ofContinuous Delivery of
your Infrastructureyour Infrastructure
Kris Buytaert
@krisbuytaert

Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @Consultant @inuits.euinuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Some books, some papers, some blogsSome books, some papers, some blogs
● Evangelizing devopsEvangelizing devops

Todays GoalsTodays Goals
● A reproducable way to deploy and upgradeA reproducable way to deploy and upgrade
your infrastructureyour infrastructure
● AutomaticallyAutomatically
● FastFast
● ConsistentConsistent
● ContinuouslyContinuously

What's this devops thingWhat's this devops thing
anyhow ?anyhow ?

C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

Whats in it for you ?Whats in it for you ?
● Faster time to marketFaster time to market
•
Features go live in hours vs yearsFeatures go live in hours vs years
● In a more safe (Secure)In a more safe (Secure)
● Reliable fashionReliable fashion
•
Fully automatedFully automated
● More happyMore happy {customers,developers,ops,managers,investors}{customers,developers,ops,managers,investors}

Less RiskLess Risk

devops (<)> continuous delilverydevops (<)> continuous delilvery

NirvanaNirvana
An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, from
infrastructure, data and configuration management toinfrastructure, data and configuration management to
business.business.
Through automation of the build, deployment, and testingThrough automation of the build, deployment, and testing
process, and improved collaboration between developers,process, and improved collaboration between developers,
testers, and operations, delivery teams can get changestesters, and operations, delivery teams can get changes
released in a matter of hours — sometimes even minutes–noreleased in a matter of hours — sometimes even minutes–no
matter what the size of a project or the complexity of its codematter what the size of a project or the complexity of its code
base.base.
Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble

How many times a day ?How many times a day ?
● 10 @ Flickr10 @ Flickr
● Deployments used to be painDeployments used to be pain
● Nobody dared to deploy a siteNobody dared to deploy a site
● Practice makes perfectPractice makes perfect
● Knowing you can vs constantly doing itKnowing you can vs constantly doing it

" Our job as engineers (and ops, dev-ops, QA," Our job as engineers (and ops, dev-ops, QA,
support, everyone in the company actually) is tosupport, everyone in the company actually) is to
enable the business goals. We strongly feel thatenable the business goals. We strongly feel that
in order to do that you must havein order to do that you must have the ability tothe ability to
deploy code quickly and safelydeploy code quickly and safely. Even if the. Even if the
business goals are to deploy strongly QA’d codebusiness goals are to deploy strongly QA’d code
once a month at 3am (it’s not for us, we push allonce a month at 3am (it’s not for us, we push all
the time), having a reliable and easy deploymentthe time), having a reliable and easy deployment
should beshould be non-negotiablenon-negotiable."."
Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

For years we've tolerated humans to makeFor years we've tolerated humans to make
structural manual changes to the infrastructurestructural manual changes to the infrastructure
our critical applications are running on.our critical applications are running on.
Whilst at the same time demanding those criticalWhilst at the same time demanding those critical
applications to go through rigid test scenarios.applications to go through rigid test scenarios.
Who let this happen ?Who let this happen ?

Why start with infra ?Why start with infra ?
● LearnLearn
● Be able to support your peersBe able to support your peers
● It's a test requirement anyhowIt's a test requirement anyhow

Automate all the thingsAutomate all the things
● BuildBuild
•
reproducable builds are undiscussablereproducable builds are undiscussable
● TestTest
•
testing reduces risktesting reduces risk
•
automate deployments of your test infraautomate deployments of your test infra
● DeployDeploy
•
Infrastructure as CodeInfrastructure as Code
•
100% automation100% automation
•
Can you rebuild your infrastructure ?Can you rebuild your infrastructure ?

Use VersionUse Version
Control,Control,
No ExcusesNo Excuses
Also forAlso for
scripts/config/cookbooksscripts/config/cookbooks
,manifests,etc,manifests,etc

Infrastructure as CodeInfrastructure as Code
● Treat configuration automation as codeTreat configuration automation as code
● Development best practicesDevelopment best practices
•
Model your infrastructureModel your infrastructure
•
Version your cookbooks / manifestsVersion your cookbooks / manifests
•
Test your cookbooks/ manifestsTest your cookbooks/ manifests
•
Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra
● Model your infrastructureModel your infrastructure
● A working service = automated ( Application Code + InfrastructureA working service = automated ( Application Code + Infrastructure
Code + Security + Monitoring )Code + Security + Monitoring )
● IAC -ne scriptingIAC -ne scripting

Release ManagementRelease Management
● What parts go in the build ?What parts go in the build ?
● What are the dependencies ?What are the dependencies ?
● What versions work together ?What versions work together ?
● Every frameworks “invents” it's own.Every frameworks “invents” it's own.
•
None workNone work

Release ManagementRelease Management
● Git Submodules !Git Submodules !
● Submodules and you will never need a ReleaseSubmodules and you will never need a Release
Management tool again , everManagement tool again , ever
● Tool independent pattern,Tool independent pattern,
•
PuppetPuppet
•
DrupalDrupal
•
SymfonySymfony
•
……..

CI ToolsCI Tools
● HudsonHudson
● JenkinsJenkins
A zillion pluginsA zillion plugins
● Go (Cruise Control)?Go (Cruise Control)?
● Travis ?Travis ?
● Also test your (Puppet/Chef/CFengine)Also test your (Puppet/Chef/CFengine)

Build PipelinesBuild Pipelines

JenkinsJenkins
● Open Source Continuous Integration ServerOpen Source Continuous Integration Server
● A zillion plugins (400)A zillion plugins (400)
● Have developers build stable and deployableHave developers build stable and deployable
codecode
● Test Infra codeTest Infra code

Jenkins PipelineJenkins Pipeline

It gets harderIt gets harder
● Try to fully automate JenkinsTry to fully automate Jenkins
•
Including plugin configIncluding plugin config
•
Including all the jobsIncluding all the jobs
•
Try templating the jobsTry templating the jobs
•
JJBJJB
•
Job DSLJob DSL
•
……

What's in your Pipeline ?What's in your Pipeline ?

A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage

Syntax and StyleSyntax and Style
● Initially ,Initially ,
all code, all the timeall code, all the time
● Now,Now,
only the changed codeonly the changed code
● Why not in post Commit Hooks ?Why not in post Commit Hooks ?

Package all the thingsPackage all the things

Why ops like to packageWhy ops like to package
● Packages give you featuresPackages give you features
•Consistency, security, dependenciesConsistency, security, dependencies
● Uniquely identify where files come fromUniquely identify where files come from
•Package or cfg-mgmtPackage or cfg-mgmt
● Source repo not always availableSource repo not always available
•Firewall / Cloud etc ..Firewall / Cloud etc ..
● Weird deployment locations , no easy accessWeird deployment locations , no easy access
● Little overhead when you automateLittle overhead when you automate

#packaginlove#packaginlove

SaySay
thanks tothanks to
JordanJordan
SisselSissel

It's not really packagingIt's not really packaging
•
It's an immutable branchIt's an immutable branch
•
It's a tracable release artefactIt's a tracable release artefact

A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo

Repository ManagementRepository Management
● PulpPulp
•
Pro : MirroringLovePro : MirroringLove
•
Con : Mongo, Stability, .debCon : Mongo, Stability, .deb
● PRM ?PRM ?
● https://github.com/ImmobilienScout24/yum-repo-sehttps://github.com/ImmobilienScout24/yum-repo-se
??

Repository ManagementRepository Management

Deploy SoftwareDeploy Software
● Repo per environmentRepo per environment
● Package {'mysoftware': ensure => latest}Package {'mysoftware': ensure => latest}
•
You control the repos !You control the repos !
● Strict versioning in config ?Strict versioning in config ?
Ensure => '0.98.4'Ensure => '0.98.4'

A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test

Repos are SLOWRepos are SLOW
● Createrepo is slow.Createrepo is slow.
● Pulp is slowPulp is slow
● Bypass repos , upload straight to appropriateBypass repos , upload straight to appropriate
PuppetMasterPuppetMaster
● Upload to repo for rebootstrappingUpload to repo for rebootstrapping

OrchestrationOrchestration● Manage 1000 nodes,Manage 1000 nodes,
● TriggerTrigger
•
UpgradesUpgrades
•
Config RunsConfig Runs
•
Service ChangesService Changes
● Think :Think :
•
McollectiveMcollective
•
ConsulConsul
•
AnsibleAnsible
•
RundeckRundeck

Orchestration 2Orchestration 2ndnd
gengen
AkaAka ChoreographyChoreography
● While ....While ....
● First install XFirst install X
● When it is ready configure YWhen it is ready configure Y
● Then notify ZThen notify Z
● Think : Zookeeper, Serf , JujuThink : Zookeeper, Serf , Juju
R
EA
R
C
H
ITE
T
!

Testing your infraTesting your infra
● One Icinga check at the timeOne Icinga check at the time
● Roll out new (puppet) codeRoll out new (puppet) code
● Trigger puppet runs on environmentTrigger puppet runs on environment
•
Load on Puppetmaster , but more on AnsibleLoad on Puppetmaster , but more on Ansible
● Trigger check icingaTrigger check icinga

Reuse tests for monitoringReuse tests for monitoring
● Unit tests vs Acceptance TestsUnit tests vs Acceptance Tests
● Your monitoring platformYour monitoring platform
•
also has a test / uat / prod flowalso has a test / uat / prod flow
•
Is fully automatedIs fully automated
● Keeping UAT green is a mustKeeping UAT green is a must
•
This is where you learn PROD behaviourThis is where you learn PROD behaviour

A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
● Check PuppetrunsCheck Puppetruns
● Check IcingaCheck Icinga
● Promote to UATPromote to UAT

Jenkins PromotionJenkins Promotion

A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
● Check PuppetrunsCheck Puppetruns
● Check IcingaCheck Icinga
● Promote to UATPromote to UAT
● Test even more ,Test even more ,
security etc..security etc..
● Promote to PRODPromote to PROD

Attack yourselve onAttack yourselve on
every buildevery build
● Gauntlt , write security testsGauntlt , write security tests
● Vulnerability scans (Arachni)Vulnerability scans (Arachni)
● Content Scanner (DIRB)Content Scanner (DIRB)
● ……
● https://github.com/garethr/pentesting-playgroundhttps://github.com/garethr/pentesting-playground

Done ?Done ?
● Close the feedback loop,Close the feedback loop,
● Send metric on deploymentSend metric on deployment
echo "deployed.$package_name 1 `date +%s`"echo "deployed.$package_name 1 `date +%s`"
> /dev/tcp/<%= graphite_host %>/2003> /dev/tcp/<%= graphite_host %>/2003

No moreNo more
● Hacking in productionHacking in production
● Can't change thisCan't change this
● I don't have monitoring for thisI don't have monitoring for this

ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221