No, we can not do
Continuous Deployment
Kris Buytaert
@krisbuytaert

Kris BuytaertKris Buytaert
•I used to be a Dev,I used to be a Dev,
•Then Became an OpThen Became an Op
•Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @Consultant @inuits.euinuits.eu
•Everything is an effing DNS ProblemEverything is an effing DNS Problem
•Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
•Some books, some papers, some blogsSome books, some papers, some blogs
•Evangelizing devopsEvangelizing devops
•Organiser of #devopsdays, #cfgmgmtcamp,Organiser of #devopsdays, #cfgmgmtcamp,
#loadays, ….#loadays, ….

What's this devopsWhat's this devops
thing anyhow ?thing anyhow ?

World , 200X-2009World , 200X-2009
Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, JezzPatrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz
Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, andHumble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and
lots of others ..lots of others ..
Gent , October 2009Gent , October 2009
Mountain View , June 2010Mountain View , June 2010
5th aniversary 2 years ago in Gent5th aniversary 2 years ago in Gent
........

C(L)AMSC(L)AMS
•CultureCulture
•(Lean)(Lean)
•AutomationAutomation
•MeasurementMeasurement
•SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

NirvanaNirvana
An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, from
infrastructure, data and configuration management toinfrastructure, data and configuration management to
business.business.
Through automation of the build, deployment, and testingThrough automation of the build, deployment, and testing
process, and improved collaboration between developers,process, and improved collaboration between developers,
testers, and operations, delivery teams can get changestesters, and operations, delivery teams can get changes
released in a matter of hours — sometimes even minutes–noreleased in a matter of hours — sometimes even minutes–no
matter what the size of a project or the complexity of its codematter what the size of a project or the complexity of its code
base.base.
Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble

Continuous IntegrationContinuous Integration
Continuous integration (CI) is the practice, in software engineering, of mergingContinuous integration (CI) is the practice, in software engineering, of merging
all developer working copies with a shared mainline several times a day. It wasall developer working copies with a shared mainline several times a day. It was
first named and proposed as part of extreme programming (XP). Its main aim isfirst named and proposed as part of extreme programming (XP). Its main aim is
to prevent integration problems, referred to as "integration hell"to prevent integration problems, referred to as "integration hell"
(WikiPedia)(WikiPedia)
Does the app you are deploying still work ?Does the app you are deploying still work ?
Did you break your infrastructure ?Did you break your infrastructure ?

CD vs CDCD vs CD
•ContinuousContinuous
DeliveryDelivery
•Knowing you canKnowing you can
•One buttonOne button
•Human decisionHuman decision
•ContinuousContinuous
DeploymentDeployment
•Constantly doing itConstantly doing it
•No More buttonsNo More buttons
•Machine decisionMachine decision

devops (<)> continuous deliverydevops (<)> continuous delivery

Culture,Culture,
automation,automation,
Measturement,Measturement,
sharingsharing

How many times a day ?How many times a day ?
•10 @ Flickr10 @ Flickr
•Deployments used to be painDeployments used to be pain
•Nobody dared to deploy a siteNobody dared to deploy a site
•Practice makes perfectPractice makes perfect
•Knowing you can vs constantly doing itKnowing you can vs constantly doing it

" Our job as engineers (and ops, dev-ops," Our job as engineers (and ops, dev-ops,
QA, support, everyone in the companyQA, support, everyone in the company
actually) is to enable the business goals.actually) is to enable the business goals.
We strongly feel that in order to do thatWe strongly feel that in order to do that
you must have theyou must have the ability to deploy codeability to deploy code
quickly and safely.quickly and safely. Even if the businessEven if the business
goals are to deploy strongly QA’d codegoals are to deploy strongly QA’d code
once a month at 3am (it’s not for us, weonce a month at 3am (it’s not for us, we
push all the time), having apush all the time), having a reliable andreliable and
easy deploymenteasy deployment should be non-should be non-
negotiable."negotiable."
Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

Whats in it for you ?Whats in it for you ?
•Faster time to marketFaster time to market
•Features go live in hours vs yearsFeatures go live in hours vs years
•In a more safe (Secure)In a more safe (Secure)
•Reliable fashionReliable fashion
•Fully automatedFully automated
•More happyMore happy {customers,developers,managers,investors}{customers,developers,managers,investors}

Pushing to production 5 times a day ..Pushing to production 5 times a day ..
Are you out of your mind ?Are you out of your mind ?
vsvs
Sure we do thatSure we do that

This is not Continuous DeploymentThis is not Continuous Deployment
•@stahnma@stahnma
@#devopsdays Ohio@#devopsdays Ohio

It is Pink SombreroIt is Pink Sombrero

But people sayBut people say
•It's too riskyIt's too risky
•We can't , ourWe can't , our
users don't wantusers don't want
to ...to ...
•But <insertBut <insert
favourite proceduralfavourite procedural
framework >framework >
•But auditors/But auditors/
ComplianceCompliance
•It's too expensiveIt's too expensive

It's too riskyIt's too risky
•We deployed 6 months ago, it wasWe deployed 6 months ago, it was
painfull, we needed 3 weeks aftercarepainfull, we needed 3 weeks aftercare
•There's 3576 changes in the newThere's 3576 changes in the new
deploy, we have no clue what causeddeploy, we have no clue what caused
this problemthis problem
•We need 20 people in a room for 8-12We need 20 people in a room for 8-12
hourshours
•I have no clue why I wrote that line ofI have no clue why I wrote that line of
code 3 months agocode 3 months ago
•The person who wrote this left 2The person who wrote this left 2
weeks agoweeks ago
•Ooops we forgot to delete that featureOoops we forgot to delete that feature
they don't want anymore.they don't want anymore.
•We deploy automaticaly,We deploy automaticaly,
•I clearly remember what we fixedI clearly remember what we fixed
yesterdayyesterday
•And that's the only thing that hasAnd that's the only thing that has
changed in the last commitchanged in the last commit
•The person who wrote the code is stillThe person who wrote the code is still
in the buildingin the building
•We really need this feature now, weWe really need this feature now, we
can remove it latercan remove it later

Every commitEvery commit withwith successful testssuccessful tests
willwill automaticallyautomatically bebe deployeddeployed
productionproduction

Every commitEvery commit with successful testwith successful test
will automatically be deployed towill automatically be deployed to
productionproduction
Version controlVersion control
Who, changed what, why and whenWho, changed what, why and when

Every commit withEvery commit with successful testssuccessful tests
will automatically be deployed towill automatically be deployed to
productionproduction
Automated testing strategy, is keyAutomated testing strategy, is key
Successful tests, no bypassing of theSuccessful tests, no bypassing of the
teststests

Test all the thingsTest all the things
•Unit testsUnit tests
•Integration TestsIntegration Tests
•System TestsSystem Tests
•Acceptance TestsAcceptance Tests
•Security TestsSecurity Tests
•Performance TestsPerformance Tests
•Regression TestsRegression Tests
•Functional TestsFunctional Tests

24
-
While Culture = OKWhile Culture = OK
- Write code- Write code
- Keep pipeline running- Keep pipeline running
- Stop the pipeline on failure- Stop the pipeline on failure
- Fix the pipeline- Fix the pipeline
- Don't go home on a broken pipeline- Don't go home on a broken pipeline

PromotionsPromotions
When you don't trust your test yet, you'llWhen you don't trust your test yet, you'll
put in manual checkpoints where humansput in manual checkpoints where humans
verify and approve, while adding moreverify and approve, while adding more
tests.tests.

Beware of the unpromoted builds :Beware of the unpromoted builds :
They risk quickly ending up in big,They risk quickly ending up in big,
delayed, problematic releases again.delayed, problematic releases again.

Testing = MonitoringTesting = Monitoring
•Add it to the monitoring frameworkAdd it to the monitoring framework
•Add collection toolsAdd collection tools
•Add check definitionsAdd check definitions
•Update the monitoring tool configUpdate the monitoring tool config
•Deploy a host,Deploy a host,
FULLY AUTOMATEDFULLY AUTOMATED

Every commit with successful testsEvery commit with successful tests
willwill automaticallyautomatically be deployed tobe deployed to
productionproduction
Automate all the things !Automate all the things !
No humans involved,No humans involved,
Less error proneLess error prone
Less boringLess boring

Every commit with successful testsEvery commit with successful tests
will automatically bewill automatically be deployed todeployed to
productionproduction
Deployed code does not meanDeployed code does not mean
enabled feature.enabled feature.

We can't , our ...We can't , our ...
•Marketing campaign only launches onMarketing campaign only launches on
1/121/12
•Users won't get trained before 15/1Users won't get trained before 15/1
•Legislation requires us to enable thatLegislation requires us to enable that
feature on 1/9feature on 1/9

We can't , our ...We can't , our ...
•XYZ requires us toXYZ requires us to enableenable that featurethat feature
on 1/9on 1/9
Deployment: does not mean EnablingDeployment: does not mean Enabling
by default !by default !

Canary , Feature , Dark launchesCanary , Feature , Dark launches
•Feature Flags :Feature Flags :
●
Only executed if this is on.Only executed if this is on.
•Canary Releases:Canary Releases:
●
10% of the audience gets a feature10% of the audience gets a feature
•Dark LaunchesDark Launches
●
Log, do, but don't show the endLog, do, but don't show the end
useruser
●
Test load of a feature , in real lifeTest load of a feature , in real life

AB testing, Blue GreenAB testing, Blue Green
•AB testingAB testing
●
Deploy both alternatives ,Deploy both alternatives ,
●
Show to subsets of usersShow to subsets of users
●
Compare resultsCompare results
•Blue Green deployments :Blue Green deployments :
●
2 identical production platforms2 identical production platforms
●
Only one is activeOnly one is active
●
Ideally on the same databasseIdeally on the same databasse
backendbackend

Auditors / ComplianceAuditors / Compliance
•We do the same, just automatedWe do the same, just automated
•Separation of DutiesSeparation of Duties
• Man vs MachineMan vs Machine
•Authentication and Audit TrailAuthentication and Audit Trail
•Full automation, Git logs, Deploy logs,Full automation, Git logs, Deploy logs,
no more manual actionsno more manual actions
•Have you tried talking to them ?Have you tried talking to them ?

We are already agile !We are already agile !
•We've implemented Scaled AgileWe've implemented Scaled Agile
FrameworkFramework
•We've commited to 4 quarterly releases /We've commited to 4 quarterly releases /
yearyear
•We can't change thisWe can't change this
•Our users can't followOur users can't follow

To ExpensiveTo Expensive
•Setting up the stack costs timeSetting up the stack costs time
•We don't have the budget to write testsWe don't have the budget to write tests
•You also don't have the budget to failYou also don't have the budget to fail
•Thats why you are still runningThats why you are still running
vulnerable security publically !vulnerable security publically !
•Operations and development areOperations and development are
different budgetsdifferent budgets
•One shot projects , fire and forgetOne shot projects , fire and forget

Culture Hack:Culture Hack:
Set up CI / CD for your infrastructure first,Set up CI / CD for your infrastructure first,
If the people running your infra don't knowIf the people running your infra don't know
how CI/CD works , how do you expecthow CI/CD works , how do you expect
them to support / teach your applicationthem to support / teach your application
teams ?teams ?
You also get them to learn about theYou also get them to learn about the
tooling they will need to support and theytooling they will need to support and they
will share the pain and the joy of thewill share the pain and the joy of the
application developersapplication developers

Culture,Culture,
Automation,Automation,
Measurement,Measurement,
SharingSharing

Broken ArchitectureBroken Architecture
•Legacy LanguagesLegacy Languages
•Big monolithBig monolith
•Stored proceduresStored procedures
•Our team doesn't understand the impactOur team doesn't understand the impact
of our changesof our changes

Our application isOur application is
•Un buildableUn buildable
•Un packageableUn packageable
•Un deployableUn deployable
•Un configurableUn configurable
•Un runnableUn runnable
•Un clusterableUn clusterable
•Un scalableUn scalable
•Un monitorableUn monitorable
•Un measurableUn measurable
•Un securedUn secured

““If myIf my
computer can'tcomputer can't
install it, theinstall it, the
installer isinstaller is
broken”broken”
Luke Kanies atLuke Kanies at
Fosdem (2007)Fosdem (2007)

As an Ops personAs an Ops person
““As a system administrator, I can tell whenAs a system administrator, I can tell when
software vendors hate me. It shows in theirsoftware vendors hate me. It shows in their
products.”products.”
““DON'T make the administrative interface aDON'T make the administrative interface a
GUI. System administrators need aGUI. System administrators need a
command-line tool for constructingcommand-line tool for constructing
repeatable processes. Procedures are bestrepeatable processes. Procedures are best
documented by providing commands thatdocumented by providing commands that
we can copy and paste from the procedurewe can copy and paste from the procedure
document to the command line. We cannotdocument to the command line. We cannot
achieve the same repeatability when theachieve the same repeatability when the
instructions are: "Checkmark the 3rd andinstructions are: "Checkmark the 3rd and
5th options, but not the 2nd option, then5th options, but not the 2nd option, then
click OK." Sysadmins do not want a GUI thatclick OK." Sysadmins do not want a GUI that
requires 25 clicks for each new user.”requires 25 clicks for each new user.”
Thomas A. Limoncelli in ACM Queue December 2010Thomas A. Limoncelli in ACM Queue December 2010
http://queue.acm.org/detail.cfm?id=1921361http://queue.acm.org/detail.cfm?id=1921361

Our process is really complexOur process is really complex
•Different people decideDifferent people decide
•Different needsDifferent needs
•Merges are complexMerges are complex
•Release management takes agesRelease management takes ages
•Testing takes agesTesting takes ages

We don't understand gitWe don't understand git
•We've copied ourWe've copied our
svn modelsvn model
•We're still in mergeWe're still in merge
hellhell

We still don't understandWe still don't understand
•Stop Branching !Stop Branching !
•Master onlyMaster only
developmentdevelopment
•Short lived featureShort lived feature
branchesbranches
• Short is hours , notShort is hours , not
days, certainly notdays, certainly not
weeksweeks

But our data migrations !But our data migrations !
•FlywayFlyway
•DB-MigrateDB-Migrate
•Liquibase,Liquibase,

Data flows backwardsData flows backwards

Every small step you take thinking you goEvery small step you take thinking you go
closer to continuous deliverycloser to continuous delivery
From 3 months to 1 monthFrom 3 months to 1 month
From 1 month to bi weeklyFrom 1 month to bi weekly
Makes your delivery process moreMakes your delivery process more
complexcomplex
Going all the way is much less painfullGoing all the way is much less painfull
than step by stepthan step by step

ConclusionsConclusions
•Most reasons why you can't areMost reasons why you can't are
•WrongWrong
•MisconceptionsMisconceptions
•On your Backlog alreadyOn your Backlog already
•It's a requirement for securityIt's a requirement for security
•hapiness(users,developers,ops,managemhapiness(users,developers,ops,managem
ent,customers,shareholders,*)++ent,customers,shareholders,*)++

No,No,
you can notyou can not
not donot do
Continuous DeliveryContinuous Delivery

A software project is not done untilA software project is not done until
your last enduser is in his grave !your last enduser is in his grave !
Kris Buytaert, DOD Amsterdam 2013Kris Buytaert, DOD Amsterdam 2013

ContactContact
Kris BuytaertKris Buytaert kris.buytaert@inuits.eukris.buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
Inuits HQInuits HQ
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221