Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

No, we can't do continuous delivery

1,036 views

Published on

My Keynote for Code-STHLM , about the various bogus reasons why people can't do Continuous Delivery /
Deployment

Published in: Technology
  • Be the first to comment

No, we can't do continuous delivery

  1. 1. No, we can not do Continuous Deployment Kris Buytaert @krisbuytaert
  2. 2. Kris BuytaertKris Buytaert •I used to be a Dev,I used to be a Dev, •Then Became an OpThen Became an Op •Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @Consultant @inuits.euinuits.eu •Everything is an effing DNS ProblemEverything is an effing DNS Problem •Building Clouds since before the bookstoreBuilding Clouds since before the bookstore •Some books, some papers, some blogsSome books, some papers, some blogs •Evangelizing devopsEvangelizing devops •Organiser of #devopsdays, #cfgmgmtcamp,Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ….#loadays, ….
  3. 3. What's this devopsWhat's this devops thing anyhow ?thing anyhow ?
  4. 4. World , 200X-2009World , 200X-2009 Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, JezzPatrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, andHumble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and lots of others ..lots of others .. Gent , October 2009Gent , October 2009 Mountain View , June 2010Mountain View , June 2010 5th aniversary 2 years ago in Gent5th aniversary 2 years ago in Gent ........
  5. 5. C(L)AMSC(L)AMS •CultureCulture •(Lean)(Lean) •AutomationAutomation •MeasurementMeasurement •SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  6. 6. NirvanaNirvana An “ecosystem” that supports continuous delivery, fromAn “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management toinfrastructure, data and configuration management to business.business. Through automation of the build, deployment, and testingThrough automation of the build, deployment, and testing process, and improved collaboration between developers,process, and improved collaboration between developers, testers, and operations, delivery teams can get changestesters, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–noreleased in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its codematter what the size of a project or the complexity of its code base.base. Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble
  7. 7. Continuous IntegrationContinuous Integration Continuous integration (CI) is the practice, in software engineering, of mergingContinuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. It wasall developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Its main aim isfirst named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as "integration hell"to prevent integration problems, referred to as "integration hell" (WikiPedia)(WikiPedia) Does the app you are deploying still work ?Does the app you are deploying still work ? Did you break your infrastructure ?Did you break your infrastructure ?
  8. 8. CD vs CDCD vs CD •ContinuousContinuous DeliveryDelivery •Knowing you canKnowing you can •One buttonOne button •Human decisionHuman decision •ContinuousContinuous DeploymentDeployment •Constantly doing itConstantly doing it •No More buttonsNo More buttons •Machine decisionMachine decision
  9. 9. devops (<)> continuous deliverydevops (<)> continuous delivery
  10. 10. Culture,Culture, automation,automation, Measturement,Measturement, sharingsharing
  11. 11. How many times a day ?How many times a day ? •10 @ Flickr10 @ Flickr •Deployments used to be painDeployments used to be pain •Nobody dared to deploy a siteNobody dared to deploy a site •Practice makes perfectPractice makes perfect •Knowing you can vs constantly doing itKnowing you can vs constantly doing it
  12. 12. " Our job as engineers (and ops, dev-ops," Our job as engineers (and ops, dev-ops, QA, support, everyone in the companyQA, support, everyone in the company actually) is to enable the business goals.actually) is to enable the business goals. We strongly feel that in order to do thatWe strongly feel that in order to do that you must have theyou must have the ability to deploy codeability to deploy code quickly and safely.quickly and safely. Even if the businessEven if the business goals are to deploy strongly QA’d codegoals are to deploy strongly QA’d code once a month at 3am (it’s not for us, weonce a month at 3am (it’s not for us, we push all the time), having apush all the time), having a reliable andreliable and easy deploymenteasy deployment should be non-should be non- negotiable."negotiable." Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
  13. 13. Whats in it for you ?Whats in it for you ? •Faster time to marketFaster time to market •Features go live in hours vs yearsFeatures go live in hours vs years •In a more safe (Secure)In a more safe (Secure) •Reliable fashionReliable fashion •Fully automatedFully automated •More happyMore happy {customers,developers,managers,investors}{customers,developers,managers,investors}
  14. 14. Pushing to production 5 times a day ..Pushing to production 5 times a day .. Are you out of your mind ?Are you out of your mind ? vsvs Sure we do thatSure we do that
  15. 15. This is not Continuous DeploymentThis is not Continuous Deployment •@stahnma@stahnma @#devopsdays Ohio@#devopsdays Ohio
  16. 16. It is Pink SombreroIt is Pink Sombrero
  17. 17. But people sayBut people say •It's too riskyIt's too risky •We can't , ourWe can't , our users don't wantusers don't want to ...to ... •But <insertBut <insert favourite proceduralfavourite procedural framework >framework > •But auditors/But auditors/ ComplianceCompliance •It's too expensiveIt's too expensive
  18. 18. It's too riskyIt's too risky •We deployed 6 months ago, it wasWe deployed 6 months ago, it was painfull, we needed 3 weeks aftercarepainfull, we needed 3 weeks aftercare •There's 3576 changes in the newThere's 3576 changes in the new deploy, we have no clue what causeddeploy, we have no clue what caused this problemthis problem •We need 20 people in a room for 8-12We need 20 people in a room for 8-12 hourshours •I have no clue why I wrote that line ofI have no clue why I wrote that line of code 3 months agocode 3 months ago •The person who wrote this left 2The person who wrote this left 2 weeks agoweeks ago •Ooops we forgot to delete that featureOoops we forgot to delete that feature they don't want anymore.they don't want anymore. •We deploy automaticaly,We deploy automaticaly, •I clearly remember what we fixedI clearly remember what we fixed yesterdayyesterday •And that's the only thing that hasAnd that's the only thing that has changed in the last commitchanged in the last commit •The person who wrote the code is stillThe person who wrote the code is still in the buildingin the building •We really need this feature now, weWe really need this feature now, we can remove it latercan remove it later
  19. 19. Every commitEvery commit withwith successful testssuccessful tests willwill automaticallyautomatically bebe deployeddeployed productionproduction
  20. 20. Every commitEvery commit with successful testwith successful test will automatically be deployed towill automatically be deployed to productionproduction Version controlVersion control Who, changed what, why and whenWho, changed what, why and when
  21. 21. Every commit withEvery commit with successful testssuccessful tests will automatically be deployed towill automatically be deployed to productionproduction Automated testing strategy, is keyAutomated testing strategy, is key Successful tests, no bypassing of theSuccessful tests, no bypassing of the teststests
  22. 22. Test all the thingsTest all the things •Unit testsUnit tests •Integration TestsIntegration Tests •System TestsSystem Tests •Acceptance TestsAcceptance Tests •Security TestsSecurity Tests •Performance TestsPerformance Tests •Regression TestsRegression Tests •Functional TestsFunctional Tests
  23. 23. 24 - While Culture = OKWhile Culture = OK - Write code- Write code - Keep pipeline running- Keep pipeline running - Stop the pipeline on failure- Stop the pipeline on failure - Fix the pipeline- Fix the pipeline - Don't go home on a broken pipeline- Don't go home on a broken pipeline
  24. 24. PromotionsPromotions When you don't trust your test yet, you'llWhen you don't trust your test yet, you'll put in manual checkpoints where humansput in manual checkpoints where humans verify and approve, while adding moreverify and approve, while adding more tests.tests.
  25. 25. Beware of the unpromoted builds :Beware of the unpromoted builds : They risk quickly ending up in big,They risk quickly ending up in big, delayed, problematic releases again.delayed, problematic releases again.
  26. 26. Testing = MonitoringTesting = Monitoring •Add it to the monitoring frameworkAdd it to the monitoring framework •Add collection toolsAdd collection tools •Add check definitionsAdd check definitions •Update the monitoring tool configUpdate the monitoring tool config •Deploy a host,Deploy a host, FULLY AUTOMATEDFULLY AUTOMATED
  27. 27. Every commit with successful testsEvery commit with successful tests willwill automaticallyautomatically be deployed tobe deployed to productionproduction Automate all the things !Automate all the things ! No humans involved,No humans involved, Less error proneLess error prone Less boringLess boring
  28. 28. Every commit with successful testsEvery commit with successful tests will automatically bewill automatically be deployed todeployed to productionproduction Deployed code does not meanDeployed code does not mean enabled feature.enabled feature.
  29. 29. We can't , our ...We can't , our ... •Marketing campaign only launches onMarketing campaign only launches on 1/121/12 •Users won't get trained before 15/1Users won't get trained before 15/1 •Legislation requires us to enable thatLegislation requires us to enable that feature on 1/9feature on 1/9
  30. 30. We can't , our ...We can't , our ... •XYZ requires us toXYZ requires us to enableenable that featurethat feature on 1/9on 1/9 Deployment: does not mean EnablingDeployment: does not mean Enabling by default !by default !
  31. 31. Canary , Feature , Dark launchesCanary , Feature , Dark launches •Feature Flags :Feature Flags : ● Only executed if this is on.Only executed if this is on. •Canary Releases:Canary Releases: ● 10% of the audience gets a feature10% of the audience gets a feature •Dark LaunchesDark Launches ● Log, do, but don't show the endLog, do, but don't show the end useruser ● Test load of a feature , in real lifeTest load of a feature , in real life
  32. 32. AB testing, Blue GreenAB testing, Blue Green •AB testingAB testing ● Deploy both alternatives ,Deploy both alternatives , ● Show to subsets of usersShow to subsets of users ● Compare resultsCompare results •Blue Green deployments :Blue Green deployments : ● 2 identical production platforms2 identical production platforms ● Only one is activeOnly one is active ● Ideally on the same databasseIdeally on the same databasse backendbackend
  33. 33. Auditors / ComplianceAuditors / Compliance •We do the same, just automatedWe do the same, just automated •Separation of DutiesSeparation of Duties • Man vs MachineMan vs Machine •Authentication and Audit TrailAuthentication and Audit Trail •Full automation, Git logs, Deploy logs,Full automation, Git logs, Deploy logs, no more manual actionsno more manual actions •Have you tried talking to them ?Have you tried talking to them ?
  34. 34. We are already agile !We are already agile ! •We've implemented Scaled AgileWe've implemented Scaled Agile FrameworkFramework •We've commited to 4 quarterly releases /We've commited to 4 quarterly releases / yearyear •We can't change thisWe can't change this •Our users can't followOur users can't follow
  35. 35. To ExpensiveTo Expensive •Setting up the stack costs timeSetting up the stack costs time •We don't have the budget to write testsWe don't have the budget to write tests •You also don't have the budget to failYou also don't have the budget to fail •Thats why you are still runningThats why you are still running vulnerable security publically !vulnerable security publically ! •Operations and development areOperations and development are different budgetsdifferent budgets •One shot projects , fire and forgetOne shot projects , fire and forget
  36. 36. Culture Hack:Culture Hack: Set up CI / CD for your infrastructure first,Set up CI / CD for your infrastructure first, If the people running your infra don't knowIf the people running your infra don't know how CI/CD works , how do you expecthow CI/CD works , how do you expect them to support / teach your applicationthem to support / teach your application teams ?teams ? You also get them to learn about theYou also get them to learn about the tooling they will need to support and theytooling they will need to support and they will share the pain and the joy of thewill share the pain and the joy of the application developersapplication developers
  37. 37. Culture,Culture, Automation,Automation, Measurement,Measurement, SharingSharing
  38. 38. Broken ArchitectureBroken Architecture •Legacy LanguagesLegacy Languages •Big monolithBig monolith •Stored proceduresStored procedures •Our team doesn't understand the impactOur team doesn't understand the impact of our changesof our changes
  39. 39. Our application isOur application is •Un buildableUn buildable •Un packageableUn packageable •Un deployableUn deployable •Un configurableUn configurable •Un runnableUn runnable •Un clusterableUn clusterable •Un scalableUn scalable •Un monitorableUn monitorable •Un measurableUn measurable •Un securedUn secured
  40. 40. ““If myIf my computer can'tcomputer can't install it, theinstall it, the installer isinstaller is broken”broken” Luke Kanies atLuke Kanies at Fosdem (2007)Fosdem (2007)
  41. 41. As an Ops personAs an Ops person ““As a system administrator, I can tell whenAs a system administrator, I can tell when software vendors hate me. It shows in theirsoftware vendors hate me. It shows in their products.”products.” ““DON'T make the administrative interface aDON'T make the administrative interface a GUI. System administrators need aGUI. System administrators need a command-line tool for constructingcommand-line tool for constructing repeatable processes. Procedures are bestrepeatable processes. Procedures are best documented by providing commands thatdocumented by providing commands that we can copy and paste from the procedurewe can copy and paste from the procedure document to the command line. We cannotdocument to the command line. We cannot achieve the same repeatability when theachieve the same repeatability when the instructions are: "Checkmark the 3rd andinstructions are: "Checkmark the 3rd and 5th options, but not the 2nd option, then5th options, but not the 2nd option, then click OK." Sysadmins do not want a GUI thatclick OK." Sysadmins do not want a GUI that requires 25 clicks for each new user.”requires 25 clicks for each new user.” Thomas A. Limoncelli in ACM Queue December 2010Thomas A. Limoncelli in ACM Queue December 2010 http://queue.acm.org/detail.cfm?id=1921361http://queue.acm.org/detail.cfm?id=1921361
  42. 42. Our process is really complexOur process is really complex •Different people decideDifferent people decide •Different needsDifferent needs •Merges are complexMerges are complex •Release management takes agesRelease management takes ages •Testing takes agesTesting takes ages
  43. 43. We don't understand gitWe don't understand git •We've copied ourWe've copied our svn modelsvn model •We're still in mergeWe're still in merge hellhell
  44. 44. We still don't understandWe still don't understand •Stop Branching !Stop Branching ! •Master onlyMaster only developmentdevelopment •Short lived featureShort lived feature branchesbranches • Short is hours , notShort is hours , not days, certainly notdays, certainly not weeksweeks
  45. 45. But our data migrations !But our data migrations ! •FlywayFlyway •DB-MigrateDB-Migrate •Liquibase,Liquibase,
  46. 46. Data flows backwardsData flows backwards
  47. 47. Every small step you take thinking you goEvery small step you take thinking you go closer to continuous deliverycloser to continuous delivery From 3 months to 1 monthFrom 3 months to 1 month From 1 month to bi weeklyFrom 1 month to bi weekly Makes your delivery process moreMakes your delivery process more complexcomplex Going all the way is much less painfullGoing all the way is much less painfull than step by stepthan step by step
  48. 48. ConclusionsConclusions •Most reasons why you can't areMost reasons why you can't are •WrongWrong •MisconceptionsMisconceptions •On your Backlog alreadyOn your Backlog already •It's a requirement for securityIt's a requirement for security •hapiness(users,developers,ops,managemhapiness(users,developers,ops,managem ent,customers,shareholders,*)++ent,customers,shareholders,*)++
  49. 49. No,No, you can notyou can not not donot do Continuous DeliveryContinuous Delivery
  50. 50. A software project is not done untilA software project is not done until your last enduser is in his grave !your last enduser is in his grave ! Kris Buytaert, DOD Amsterdam 2013Kris Buytaert, DOD Amsterdam 2013
  51. 51. ContactContact Kris BuytaertKris Buytaert kris.buytaert@inuits.eukris.buytaert@inuits.eu Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ Inuits HQInuits HQ Essensteenweg 31Essensteenweg 31 BrasschaatBrasschaat BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×