Continuous Infrastructure
First
Kris Buytaert
@krisbuytaert
Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● CTO and Open Source Consultant @CTO and Open Source Consultant @inuits.euinuits.eu
● Everything is a freaking DNS ProblemEverything is a freaking DNS Problem
● Evangelizing devopsEvangelizing devops
● Organiser of #devopsdays, #cfgmgmtcamp,Organiser of #devopsdays, #cfgmgmtcamp,
#loadays, ….#loadays, ….
What hasWhat has
Ops ,Ops ,
automation,automation,
engineering ,engineering ,
got to do with it ?got to do with it ?
#devops=~C(L)AMS#devops=~C(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● Monitoring and MeasurementMonitoring and Measurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim
UmbrellaOpsUmbrellaOps
NoOps & YOLO OpsNoOps & YOLO Ops
● StartupStartup
● VCVC
● Exit StrategyExit Strategy
● 6-9 months6-9 months
● Actual BusinessActual Business
● Real CustomersReal Customers
● SurvivalSurvival
● 6-9 years6-9 years
HistoryHistory
● Europe :Europe :
•
Starting from Operations,Starting from Operations,
•
Improved Artifact Quality,Improved Artifact Quality,
•
Less pain / stabilityLess pain / stability
● US :US :
•
Push from devoplmentPush from devoplment
•
Faster PlatformsFaster Platforms
•
Faster changeFaster change
4 Diferent4 Diferent TransitionTransition Cases:Cases:
Startups & MultinationalsStartups & Multinationals
Case 1: Chaotic OpsCase 1: Chaotic Ops
● Complete ChaosComplete Chaos
● 10% reproducability10% reproducability
● CI infra hides under a dev’s deskCI infra hides under a dev’s desk
● Ops in Debug ModeOps in Debug Mode
● No standardisationNo standardisation
● Apollo MomentApollo Moment
Case 1: T0+3monthsCase 1: T0+3months
● Build a Reproducable Jenkins + SlavesBuild a Reproducable Jenkins + Slaves
● CI for Puppet by OPSCI for Puppet by OPS
● Test your codeTest your code
● Promotion Stage for Infrastructure CodePromotion Stage for Infrastructure Code
● Split confg out of codeSplit confg out of code
● Keep delivering updatesKeep delivering updates
Case 1: T0+6 monthsCase 1: T0+6 months
● Stack AlignmentStack Alignment
•
1 jdk, 1 jboss , ...1 jdk, 1 jboss , ...
● Project Dolly :Project Dolly :
•
Puppet for everythingPuppet for everything
● 90% reproducability90% reproducability
● Standardised BuildsStandardised Builds
● Increased Test CoverageIncreased Test Coverage
● Java Developers contribute to Infra TestsJava Developers contribute to Infra Tests
Case 1: ConclusionCase 1: Conclusion
● Started with preparing ops folks to automateStarted with preparing ops folks to automate
● Learned the same tools developers useLearned the same tools developers use
● Developers help the ops folks to improveDevelopers help the ops folks to improve
● Collaboration + Progress ++Collaboration + Progress ++
Case 2: CI by DevsCase 2: CI by Devs
● Some devs have testsSome devs have tests
● Some dev teams have “CI”Some dev teams have “CI”
● Deployments are ChaosDeployments are Chaos
● Ops nags about Artifact QualityOps nags about Artifact Quality
Case 2: T0+18 monthsCase 2: T0+18 months
● Found the frst ops skills in the orgFound the frst ops skills in the org
● Mostly overworked Brent’sMostly overworked Brent’s
● Move them out of their ofcesMove them out of their ofces
● Focus teamFocus team
● Teach AgileTeach Agile
● Adopt IAC (puppet)Adopt IAC (puppet)
● First SuccessesFirst Successes
● Move people back to teamsMove people back to teams
Case 2: T0+24 monthsCase 2: T0+24 months
● Grey Beard Ops person has converted to AgileGrey Beard Ops person has converted to Agile
EvangelistEvangelist
● Preaches Kanban (for ops) and ScrumPreaches Kanban (for ops) and Scrum
● Writes Test for his CodeWrites Test for his Code
● Coaches developers to achieve CI/CDCoaches developers to achieve CI/CD
Case 2: ConclusionsCase 2: Conclusions
● Starting with dev delayed the collaboration forStarting with dev delayed the collaboration for
1+ year1+ year
● Ops were fre fghting and not involvedOps were fre fghting and not involved
● Once ops resources were dedicatedOnce ops resources were dedicated
collaboration and quality improvement startedcollaboration and quality improvement started
to happento happen
Case 3: Countries FightCase 3: Countries Fight
● Country 1 has full IAC with CICountry 1 has full IAC with CI
● Country 2 has no IAC (but claims they do), lowCountry 2 has no IAC (but claims they do), low
test coverage and manual deploystest coverage and manual deploys
● Country 3 has to deploy in country 2 , extremeCountry 3 has to deploy in country 2 , extreme
pain is the result.pain is the result.
● Once it has been proven that Country 2 reallyOnce it has been proven that Country 2 really
doesn’t have IAC , Country 3 is allowed todoesn’t have IAC , Country 3 is allowed to
deploy in Country 1. Things smooth outdeploy in Country 1. Things smooth out
Case 3: ConclusionsCase 3: Conclusions
You might be able to fake automation tillYou might be able to fake automation till
someone who has automation shows upsomeone who has automation shows up
Case 4:Ops NOT involvedCase 4:Ops NOT involved
● Large TransformationLarge Transformation
● ““devops” team dictates tools they have neverdevops” team dictates tools they have never
used them selvesused them selves
● Tools they as a team don’t need themselvesTools they as a team don’t need themselves
● Developers complain about unusable toolsDevelopers complain about unusable tools
● Developers complain about broken toolsDevelopers complain about broken tools
● Tools enforce a manual processTools enforce a manual process
Case 4: 2 years laterCase 4: 2 years later
● Average “devops” role stays for 2 months ,Average “devops” role stays for 2 months ,
then leavesthen leaves
● Senior IT management has left (2x)Senior IT management has left (2x)
● Only In house analysts remainOnly In house analysts remain
● Mostly contract based developersMostly contract based developers
● Failing Cloud StrategyFailing Cloud Strategy
● Legacy Container EcosystemLegacy Container Ecosystem
Case ConclusionsCase Conclusions
Earlier involvement of the Ops Skills createsEarlier involvement of the Ops Skills creates
higher success rates , and smoother adoptionhigher success rates , and smoother adoption
Change Management &Change Management &
ResistanceResistance
● 20 – 60 – 2020 – 60 – 20
● 20% Early adopters , fnd your peers20% Early adopters , fnd your peers
● 60% Will wait , but adopt slowly60% Will wait , but adopt slowly
● 20%20%
Why ops frst ?Why ops frst ?
● You can’t support / understand what you don’tYou can’t support / understand what you don’t
do yourselve.do yourselve.
● Code = CodeCode = Code
● Unblock deliveryUnblock delivery
● Unblock provisioningUnblock provisioning
● Metrics & Monitoring Build inMetrics & Monitoring Build in
On Prem vs CloudnativeOn Prem vs Cloudnative
● Puppet, Chef, AnsiblePuppet, Chef, Ansible
● VPN, FirewallsVPN, Firewalls
● Pipelines & PipelinesPipelines & Pipelines
● StandardisationStandardisation
● SecuritySecurity
● MonitoringMonitoring
● Terraform, Cloudinit,Terraform, Cloudinit,
● VPC, SGVPC, SG
● Pipelines & PipelinesPipelines & Pipelines
● StandardisationStandardisation
● SecuritySecurity
● MonitoringMonitoring
Container NaiveContainer Naive
Dev OoopsDev Ooops #container edition#container edition
● ““Put this Code Live, here's a DockerPut this Code Live, here's a Docker
Image ”Image ”
● No machines available ?No machines available ?
● What database ? Where to store theWhat database ? Where to store the
data ?data ?
● Security ? What distro is this even ?Security ? What distro is this even ?
Bad Cows ?Bad Cows ?
● How do we monitor his ?How do we monitor his ?
● Backups ?Backups ?
● How did you build this ?How did you build this ?
● Has DockerHub been hacked again ?Has DockerHub been hacked again ?
Culture vs ToolsCulture vs Tools
Waterfall & ITIL vs AgileWaterfall & ITIL vs Agile
● Ops has been doing waterfall projects forOps has been doing waterfall projects for
decadesdecades
● They need time to adaptThey need time to adapt
● Time to UnlearnTime to Unlearn
A CI EcosystemA CI Ecosystem
● Version ControlVersion Control
● DeploymentDeployment
● Build ToolingBuild Tooling
● Artifcact RepositoryArtifcact Repository
● Code Coverage ToolingCode Coverage Tooling
● Testing ToTesting To
How many of those tools is your average opsHow many of those tools is your average ops
person used to use ?person used to use ?
Understanding CIUnderstanding CI
● Understanding Dependency Management,Understanding Dependency Management,
Testing, Packaging , Deploying =>Testing, Packaging , Deploying =>
•
Same LanguageSame Language
•
Same ToolsSame Tools
•
Same PainSame Pain
People Argument against CIPeople Argument against CI
•Setting up the stack costs timeSetting up the stack costs time
•U don’t have testsU don’t have tests
•Thats why you are still runningThats why you are still running
vulnerable security publically !vulnerable security publically !
•Operations and development areOperations and development are
diferent budgetsdiferent budgets
•One shot projects , fre and forgetOne shot projects , fre and forget
Typical Dev FirstTypical Dev First
● Dev lacks environments or uses wrong stackDev lacks environments or uses wrong stack
● Dev can’t deploy yetDev can’t deploy yet
● No PipelinesNo Pipelines
•
Technical debt is createdTechnical debt is created
● Ops is catching up with envsOps is catching up with envs
● NFR get ignoredNFR get ignored
● Brent stays a heroBrent stays a hero
This is not Continuous DeploymentThis is not Continuous Deployment
•@stahnma@stahnma
@#devopsdays Ohio@#devopsdays Ohio
Culture Hack:Culture Hack:
Set up CI/CD for your CI/CD infrastructureSet up CI/CD for your CI/CD infrastructure
frst, If the people running your infra don'tfrst, If the people running your infra don't
know how CI/CD works , how do youknow how CI/CD works , how do you
expect them to support / teach yourexpect them to support / teach your
application teams ?application teams ?
Sprint 0Sprint 0
● Onboard the teamOnboard the team
● Hello World from source to prodHello World from source to prod
● Keep the pipeline greenKeep the pipeline green
● Create basic monitoring / healthCreate basic monitoring / health
● Keep it healthyKeep it healthy
Security ?Security ?
How do we get fromHow do we get from
We don’t dare to patchWe don’t dare to patch
ToTo
All systems green , lets goAll systems green , lets go
Continuous Delivery is aContinuous Delivery is a
Security RequirementSecurity Requirement
MTTR ~> 0MTTR ~> 0
It's not about the toolsIt's not about the tools
It's about changeIt's about change
It's about the peopleIt's about the people
Devopsdays Gent 2019Devopsdays Gent 2019
29-30 october 201929-30 october 2019
Vooruit GhentVooruit Ghent
https://devopsdays.org/https://devopsdays.org/
events/2019-ghent/events/2019-ghent/
Early Bird = OPENEarly Bird = OPEN
CFP = OPENCFP = OPEN
ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221

Continuous Infrastructure First

  • 1.
  • 2.
    Kris BuytaertKris Buytaert ●I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● CTO and Open Source Consultant @CTO and Open Source Consultant @inuits.euinuits.eu ● Everything is a freaking DNS ProblemEverything is a freaking DNS Problem ● Evangelizing devopsEvangelizing devops ● Organiser of #devopsdays, #cfgmgmtcamp,Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ….#loadays, ….
  • 3.
    What hasWhat has Ops,Ops , automation,automation, engineering ,engineering , got to do with it ?got to do with it ?
  • 4.
    #devops=~C(L)AMS#devops=~C(L)AMS ● CultureCulture ● (Lean)(Lean) ●AutomationAutomation ● Monitoring and MeasurementMonitoring and Measurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  • 5.
  • 6.
    NoOps & YOLOOpsNoOps & YOLO Ops ● StartupStartup ● VCVC ● Exit StrategyExit Strategy ● 6-9 months6-9 months ● Actual BusinessActual Business ● Real CustomersReal Customers ● SurvivalSurvival ● 6-9 years6-9 years
  • 7.
    HistoryHistory ● Europe :Europe: • Starting from Operations,Starting from Operations, • Improved Artifact Quality,Improved Artifact Quality, • Less pain / stabilityLess pain / stability ● US :US : • Push from devoplmentPush from devoplment • Faster PlatformsFaster Platforms • Faster changeFaster change
  • 8.
    4 Diferent4 DiferentTransitionTransition Cases:Cases: Startups & MultinationalsStartups & Multinationals
  • 9.
    Case 1: ChaoticOpsCase 1: Chaotic Ops ● Complete ChaosComplete Chaos ● 10% reproducability10% reproducability ● CI infra hides under a dev’s deskCI infra hides under a dev’s desk ● Ops in Debug ModeOps in Debug Mode ● No standardisationNo standardisation ● Apollo MomentApollo Moment
  • 10.
    Case 1: T0+3monthsCase1: T0+3months ● Build a Reproducable Jenkins + SlavesBuild a Reproducable Jenkins + Slaves ● CI for Puppet by OPSCI for Puppet by OPS ● Test your codeTest your code ● Promotion Stage for Infrastructure CodePromotion Stage for Infrastructure Code ● Split confg out of codeSplit confg out of code ● Keep delivering updatesKeep delivering updates
  • 11.
    Case 1: T0+6monthsCase 1: T0+6 months ● Stack AlignmentStack Alignment • 1 jdk, 1 jboss , ...1 jdk, 1 jboss , ... ● Project Dolly :Project Dolly : • Puppet for everythingPuppet for everything ● 90% reproducability90% reproducability ● Standardised BuildsStandardised Builds ● Increased Test CoverageIncreased Test Coverage ● Java Developers contribute to Infra TestsJava Developers contribute to Infra Tests
  • 12.
    Case 1: ConclusionCase1: Conclusion ● Started with preparing ops folks to automateStarted with preparing ops folks to automate ● Learned the same tools developers useLearned the same tools developers use ● Developers help the ops folks to improveDevelopers help the ops folks to improve ● Collaboration + Progress ++Collaboration + Progress ++
  • 13.
    Case 2: CIby DevsCase 2: CI by Devs ● Some devs have testsSome devs have tests ● Some dev teams have “CI”Some dev teams have “CI” ● Deployments are ChaosDeployments are Chaos ● Ops nags about Artifact QualityOps nags about Artifact Quality
  • 14.
    Case 2: T0+18monthsCase 2: T0+18 months ● Found the frst ops skills in the orgFound the frst ops skills in the org ● Mostly overworked Brent’sMostly overworked Brent’s ● Move them out of their ofcesMove them out of their ofces ● Focus teamFocus team ● Teach AgileTeach Agile ● Adopt IAC (puppet)Adopt IAC (puppet) ● First SuccessesFirst Successes ● Move people back to teamsMove people back to teams
  • 15.
    Case 2: T0+24monthsCase 2: T0+24 months ● Grey Beard Ops person has converted to AgileGrey Beard Ops person has converted to Agile EvangelistEvangelist ● Preaches Kanban (for ops) and ScrumPreaches Kanban (for ops) and Scrum ● Writes Test for his CodeWrites Test for his Code ● Coaches developers to achieve CI/CDCoaches developers to achieve CI/CD
  • 16.
    Case 2: ConclusionsCase2: Conclusions ● Starting with dev delayed the collaboration forStarting with dev delayed the collaboration for 1+ year1+ year ● Ops were fre fghting and not involvedOps were fre fghting and not involved ● Once ops resources were dedicatedOnce ops resources were dedicated collaboration and quality improvement startedcollaboration and quality improvement started to happento happen
  • 17.
    Case 3: CountriesFightCase 3: Countries Fight ● Country 1 has full IAC with CICountry 1 has full IAC with CI ● Country 2 has no IAC (but claims they do), lowCountry 2 has no IAC (but claims they do), low test coverage and manual deploystest coverage and manual deploys ● Country 3 has to deploy in country 2 , extremeCountry 3 has to deploy in country 2 , extreme pain is the result.pain is the result. ● Once it has been proven that Country 2 reallyOnce it has been proven that Country 2 really doesn’t have IAC , Country 3 is allowed todoesn’t have IAC , Country 3 is allowed to deploy in Country 1. Things smooth outdeploy in Country 1. Things smooth out
  • 18.
    Case 3: ConclusionsCase3: Conclusions You might be able to fake automation tillYou might be able to fake automation till someone who has automation shows upsomeone who has automation shows up
  • 19.
    Case 4:Ops NOTinvolvedCase 4:Ops NOT involved ● Large TransformationLarge Transformation ● ““devops” team dictates tools they have neverdevops” team dictates tools they have never used them selvesused them selves ● Tools they as a team don’t need themselvesTools they as a team don’t need themselves ● Developers complain about unusable toolsDevelopers complain about unusable tools ● Developers complain about broken toolsDevelopers complain about broken tools ● Tools enforce a manual processTools enforce a manual process
  • 20.
    Case 4: 2years laterCase 4: 2 years later ● Average “devops” role stays for 2 months ,Average “devops” role stays for 2 months , then leavesthen leaves ● Senior IT management has left (2x)Senior IT management has left (2x) ● Only In house analysts remainOnly In house analysts remain ● Mostly contract based developersMostly contract based developers ● Failing Cloud StrategyFailing Cloud Strategy ● Legacy Container EcosystemLegacy Container Ecosystem
  • 21.
    Case ConclusionsCase Conclusions Earlierinvolvement of the Ops Skills createsEarlier involvement of the Ops Skills creates higher success rates , and smoother adoptionhigher success rates , and smoother adoption
  • 22.
    Change Management &ChangeManagement & ResistanceResistance ● 20 – 60 – 2020 – 60 – 20 ● 20% Early adopters , fnd your peers20% Early adopters , fnd your peers ● 60% Will wait , but adopt slowly60% Will wait , but adopt slowly ● 20%20%
  • 23.
    Why ops frst?Why ops frst ? ● You can’t support / understand what you don’tYou can’t support / understand what you don’t do yourselve.do yourselve. ● Code = CodeCode = Code ● Unblock deliveryUnblock delivery ● Unblock provisioningUnblock provisioning ● Metrics & Monitoring Build inMetrics & Monitoring Build in
  • 24.
    On Prem vsCloudnativeOn Prem vs Cloudnative ● Puppet, Chef, AnsiblePuppet, Chef, Ansible ● VPN, FirewallsVPN, Firewalls ● Pipelines & PipelinesPipelines & Pipelines ● StandardisationStandardisation ● SecuritySecurity ● MonitoringMonitoring ● Terraform, Cloudinit,Terraform, Cloudinit, ● VPC, SGVPC, SG ● Pipelines & PipelinesPipelines & Pipelines ● StandardisationStandardisation ● SecuritySecurity ● MonitoringMonitoring
  • 25.
  • 26.
    Dev OoopsDev Ooops#container edition#container edition ● ““Put this Code Live, here's a DockerPut this Code Live, here's a Docker Image ”Image ” ● No machines available ?No machines available ? ● What database ? Where to store theWhat database ? Where to store the data ?data ? ● Security ? What distro is this even ?Security ? What distro is this even ? Bad Cows ?Bad Cows ? ● How do we monitor his ?How do we monitor his ? ● Backups ?Backups ? ● How did you build this ?How did you build this ? ● Has DockerHub been hacked again ?Has DockerHub been hacked again ?
  • 27.
  • 28.
    Waterfall & ITILvs AgileWaterfall & ITIL vs Agile ● Ops has been doing waterfall projects forOps has been doing waterfall projects for decadesdecades ● They need time to adaptThey need time to adapt ● Time to UnlearnTime to Unlearn
  • 29.
    A CI EcosystemACI Ecosystem ● Version ControlVersion Control ● DeploymentDeployment ● Build ToolingBuild Tooling ● Artifcact RepositoryArtifcact Repository ● Code Coverage ToolingCode Coverage Tooling ● Testing ToTesting To How many of those tools is your average opsHow many of those tools is your average ops person used to use ?person used to use ?
  • 30.
    Understanding CIUnderstanding CI ●Understanding Dependency Management,Understanding Dependency Management, Testing, Packaging , Deploying =>Testing, Packaging , Deploying => • Same LanguageSame Language • Same ToolsSame Tools • Same PainSame Pain
  • 31.
    People Argument againstCIPeople Argument against CI •Setting up the stack costs timeSetting up the stack costs time •U don’t have testsU don’t have tests •Thats why you are still runningThats why you are still running vulnerable security publically !vulnerable security publically ! •Operations and development areOperations and development are diferent budgetsdiferent budgets •One shot projects , fre and forgetOne shot projects , fre and forget
  • 32.
    Typical Dev FirstTypicalDev First ● Dev lacks environments or uses wrong stackDev lacks environments or uses wrong stack ● Dev can’t deploy yetDev can’t deploy yet ● No PipelinesNo Pipelines • Technical debt is createdTechnical debt is created ● Ops is catching up with envsOps is catching up with envs ● NFR get ignoredNFR get ignored ● Brent stays a heroBrent stays a hero
  • 33.
    This is notContinuous DeploymentThis is not Continuous Deployment •@stahnma@stahnma @#devopsdays Ohio@#devopsdays Ohio
  • 34.
    Culture Hack:Culture Hack: Setup CI/CD for your CI/CD infrastructureSet up CI/CD for your CI/CD infrastructure frst, If the people running your infra don'tfrst, If the people running your infra don't know how CI/CD works , how do youknow how CI/CD works , how do you expect them to support / teach yourexpect them to support / teach your application teams ?application teams ?
  • 35.
    Sprint 0Sprint 0 ●Onboard the teamOnboard the team ● Hello World from source to prodHello World from source to prod ● Keep the pipeline greenKeep the pipeline green ● Create basic monitoring / healthCreate basic monitoring / health ● Keep it healthyKeep it healthy
  • 36.
  • 37.
    How do weget fromHow do we get from We don’t dare to patchWe don’t dare to patch ToTo All systems green , lets goAll systems green , lets go
  • 38.
    Continuous Delivery isaContinuous Delivery is a Security RequirementSecurity Requirement
  • 39.
  • 40.
    It's not aboutthe toolsIt's not about the tools It's about changeIt's about change It's about the peopleIt's about the people
  • 41.
    Devopsdays Gent 2019DevopsdaysGent 2019 29-30 october 201929-30 october 2019 Vooruit GhentVooruit Ghent https://devopsdays.org/https://devopsdays.org/ events/2019-ghent/events/2019-ghent/ Early Bird = OPENEarly Bird = OPEN CFP = OPENCFP = OPEN
  • 42.
    ContactContact Kris BuytaertKris BuytaertKris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ InuitsInuits Essensteenweg 31Essensteenweg 31 BrasschaatBrasschaat BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221