DEVOPS WITH SMELL
ANTONS KRANGA
~ WHOAMI
▸ Full stack developer ~ 15years
▸ Cloud Architect
▸ DevOps evangelist
▸ Innovation Center of Accenture Cloud
Platform
▸ Speaker
▸ Marathon runner
ANTONS KRANGA
WHY DEVOPS
DEVELOPERS VERSION
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
▸ specialization silos
▸ ops comes first
▸ cross functional
▸ software defined

data centers
▸ complex communication ▸ complexity theory
DEVOPS IS AN ENVIRONMENT
WHERE PEOPLE TALK
me :)
DEFINITION OF DEVOPS
SMELL SYMPTOMA
DEVOPS ANTIPATTERNS
ANTIPATTERN # 0
UNICORNS VS HORSES
You cannot be a
unicorn when all
others are horses
ANTIPATTERN # 0.1
DEVOPS IN A BOX
You cannot buy culture!
▸ Buy a Golden DevOps LVL5
▸ Become certified DevOps master
▸ Give me DevOps compliance checklist
▸ Give me 5 key DevOps control metrics
ANTIPATTERN # 0.2
REBRANDING
DevOps != Configuration Management

DevOps != Release Management|

DevOps != Product Management

…

DevOps != (*) Management
DEVOPS IS THE CULTURE YOU
CANNOT FIND IN IN ITIL CATALOG
ANTIPATTERN # 1
DEVOPS AS A BLACK BOX
What is the value in DevOps?
! Developer
! Sys-Op
ANTIPATTERN # 2
SORRY, NOT MY DEPARTMENT
Bread ownership and
specialization with
autonomous teams
vs
http://martinfowler.com/bliki/DevOpsCulture.html
ANTIPATTERN # 2.1
DEVOPS AS A SILO
DevOps teams build their own Silo
▸ You cannot talk to DevOps
▸ Use Jira instead !!!!
▸ RTFM Architecture
ANTIPATTERN # 3
DEFINITION OF DONE
▸ DoD fails with “ilities”
▸ Services can be easily “undone”
ANTIPATTERN # 4
FEAR OF RELEASE
RELEASE === RISK
NO RELEAE NO REVENUE
vs
ANTIPATTERN # 5
SNOWFLAKE SERVER
Applying changes to Server Instance
manually leads to unique and distinct
server configuration footprint
(TECHNICAL DEBT)
http://martinfowler.com/bliki/SnowflakeServer.html
VITAMINS
DEVOPS ANTIPATTERNS
VITAMINS
INFRASTRUCTURE AS CODE
ENV
ENV
ENV
DEV …CODE
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is a

Contract
OPS
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is a

Contract
OPS
Put infrastructure into SCM
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is a

Contract
OPS`
Make infrastructure part of app baseline
VITAMINS
WE LIKE CI/CD
▸ We need feedback not to be afraid
▸ Feedback != SPAM
▸ It’s about size of release not frequency
▸ Don’t judge for broken builds
▸ Go home when build is green
PAINKILLERS
DEVOPS ANTIPATTERNS
ANTIPATTERN # 6
“JENKINS” DRIVEN DEVELOPMENT
`
`
`
Pipelines are easily becomes your
single point of failure
CI
ANTIPATTERN # 6: PROPOSED SOLUTION
SHIFTING RESPONSIBILITY
CI users (DEVs or OPs) are best for managing their jobs
▸ Use DSL to build CI/CD pipelines
▸ Store CI/CD pipelines in git 

if possible make it part of app baseline
▸ Generate and bootsrap pipelines with API
ANTIPATTERN # 7
SLOW PIPELINES
` ` `
Over-engineered Pipelines
Pipeline execution takes too long
Leads to delayed feedback
ANTIPATTERN # 6: PROPOSED SOLUTION
SLOW PIPELINES
`
`
`
Parallelize where you can!
ANTIPATTERN # 7
MANUAL PROMOTION
Engineer PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
ANTIPATTERN # 7: PROPOSED SOLUTION
KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
ANTIPATTERN # 7: PROPOSED SOLUTION
AB TESTING
PROD A
Integr TestsDEV System Tests
ENV
ENVENV
TEST A
PROD B
ENV
TEST B
feedback
feedback
measure
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr Tests System Tests
ENVENVENV ENV
DEV B
feature 

branch DEV C
code 

review
ANTIPATTERN # 7.1: PROPOSED SOLUTION
REACTIVE CODEREVIEW
ReleaseIntegr Tests System Tests
ENVENV ENV
Full 

Regression
Calc technical

debt
ENV
Nightly
DEV A
Standup
code 

review
…
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr TestsDEV A System Tests
ENVENVENV ENV
master
DEV B
feature 

branch DEV C
code 

review
ANTIPATTERN # 8
DEV DEPLOYMENT VIA CI
DEV

ENV
DEV

ENV
DEV

ENV
…DEV CI
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
DEV …NO CI
DEV

ENV
DEV

ENV
DEV

ENV
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
ENV
ENV
DEV

ENV
…DEV NO CI
WANT TO HACK? SURE!
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
ENV
ENV
DEV

ENV
…
YOU BROKE IT? YOU FIX IT!
DEV NO CI
ANTIBIOTICS
DEVOPS ANTIPATTERNS
ANTIPATTERN # 9
GOLDEN IMAGE
VM
OS
Problems
▸ Maintained manually
▸ No collaboration
▸ Hard to distribute
▸ Non versioning
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
cmdb
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
complicated?
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
static dynamic
OS
ConfigureHarden Download Install
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
code
packer
PROVISIONSNAPSHOT
docker
Chnorr Service
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructure

code
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
Chnorr Service
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructure

code
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
▸ Version of your infra code
▸ Maintain dependencies
Chnorr Service
ANTIPATTERN # B: PROPOSED SOLUTION
CANARY BUILDS
Accept Tests
CI
Unit Test Provision
…
PRECISE version libs
Accept Tests
CI
Unit Test Provision
…
LATEST version libs
Chnorr Service
Chnorr Service
ANTIPATTERN # C
INFRASTRUCTURE PET
Attributes of Pet
▸ Have meaningful names
▸ Long living instance
▸ Often needs manual nursing
▸ Requires scary patching
▸ Leads to snowflakes
▸ PaaS is modern pet
ANTIPATTERN # C: PROPOSED SOLUTION
INFRASTRUCTURE CATTLE
Attributes of Pet
▸ Have numbers in its name
▸ Short living instance
▸ Immutable configuration
▸ Recreate instead of patching
▸ Requires careful planning
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
User Data
ANTIPATTERN # D
SECRETS LEAK
OS
Configure
Infrastructure

code
Install
cmdb
wrong place 

for your secrets
wrong place 

for your secrets
Chnorr Service
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Don’t store secrets with code
▸ Don’t store secrets with configuration
▸ Don’t leave secrets in service
Secrets DON’Ts
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Secret can be leased and rotated
▸ tmpfs is your fiend
Secrets DOs
ANTIPATTERN # D: PROPOSED SOLUTION
SECURITY LEASING EXAMPLE
Chnorr Service vault
consul
IAM
Database
api
x hours leasing
AWS
TAKEAWAYS
GOOD INFRA CODE
TAKEAWAYS
LEARN PATTERNS BEFORE TOOLS
Patterns Tools
vs
TAKEAWAYS
EVERYTHING MUST HAVE AN API
REST
DSL
CLI
TAKEAWAYS
SELF TESTABLE CODE
▸ Use assertions for infrastructure code
▸ Use acceptance test frameworks
Tests improves your confidence
TAKEAWAYS
READING
▸ Book: A Human Error Approach to
Aviation Accident Analysis
▸ Author: Douglas A. Wiegmann

Scott A. Shappell
▸ ISBN: 978-0754618737
TAKEAWAYS
READING
▸ Book: Clean Code
▸ Author: Robert C Martin
▸ ISBN: 978-0132350884
THANK YOU

DevTernity - DevOps with smell

  • 1.
  • 2.
    ~ WHOAMI ▸ Fullstack developer ~ 15years ▸ Cloud Architect ▸ DevOps evangelist ▸ Innovation Center of Accenture Cloud Platform ▸ Speaker ▸ Marathon runner ANTONS KRANGA
  • 3.
  • 4.
    DRIVERS FOR SOFTWAREDEVELOPMENT ▸ development price BEFORE AFTER ▸ innovations speed
  • 5.
    DRIVERS FOR SOFTWAREDEVELOPMENT ▸ development price BEFORE AFTER ▸ innovations speed ▸ specialization silos ▸ ops comes first ▸ cross functional ▸ software defined
 data centers ▸ complex communication ▸ complexity theory
  • 6.
    DEVOPS IS ANENVIRONMENT WHERE PEOPLE TALK me :) DEFINITION OF DEVOPS
  • 7.
  • 8.
    ANTIPATTERN # 0 UNICORNSVS HORSES You cannot be a unicorn when all others are horses
  • 9.
    ANTIPATTERN # 0.1 DEVOPSIN A BOX You cannot buy culture! ▸ Buy a Golden DevOps LVL5 ▸ Become certified DevOps master ▸ Give me DevOps compliance checklist ▸ Give me 5 key DevOps control metrics
  • 10.
    ANTIPATTERN # 0.2 REBRANDING DevOps!= Configuration Management
 DevOps != Release Management|
 DevOps != Product Management
 …
 DevOps != (*) Management DEVOPS IS THE CULTURE YOU CANNOT FIND IN IN ITIL CATALOG
  • 11.
    ANTIPATTERN # 1 DEVOPSAS A BLACK BOX What is the value in DevOps? ! Developer ! Sys-Op
  • 12.
    ANTIPATTERN # 2 SORRY,NOT MY DEPARTMENT Bread ownership and specialization with autonomous teams vs http://martinfowler.com/bliki/DevOpsCulture.html
  • 13.
    ANTIPATTERN # 2.1 DEVOPSAS A SILO DevOps teams build their own Silo ▸ You cannot talk to DevOps ▸ Use Jira instead !!!! ▸ RTFM Architecture
  • 14.
    ANTIPATTERN # 3 DEFINITIONOF DONE ▸ DoD fails with “ilities” ▸ Services can be easily “undone”
  • 15.
    ANTIPATTERN # 4 FEAROF RELEASE RELEASE === RISK NO RELEAE NO REVENUE vs
  • 16.
    ANTIPATTERN # 5 SNOWFLAKESERVER Applying changes to Server Instance manually leads to unique and distinct server configuration footprint (TECHNICAL DEBT) http://martinfowler.com/bliki/SnowflakeServer.html
  • 17.
  • 18.
  • 19.
  • 20.
    VITAMINS INFRASTRUCTURE AS CODE DEV Codeis a
 Contract OPS Put infrastructure into SCM
  • 21.
    VITAMINS INFRASTRUCTURE AS CODE DEV Codeis a
 Contract OPS` Make infrastructure part of app baseline
  • 22.
    VITAMINS WE LIKE CI/CD ▸We need feedback not to be afraid ▸ Feedback != SPAM ▸ It’s about size of release not frequency ▸ Don’t judge for broken builds ▸ Go home when build is green
  • 23.
  • 24.
    ANTIPATTERN # 6 “JENKINS”DRIVEN DEVELOPMENT ` ` ` Pipelines are easily becomes your single point of failure CI
  • 25.
    ANTIPATTERN # 6:PROPOSED SOLUTION SHIFTING RESPONSIBILITY CI users (DEVs or OPs) are best for managing their jobs ▸ Use DSL to build CI/CD pipelines ▸ Store CI/CD pipelines in git 
 if possible make it part of app baseline ▸ Generate and bootsrap pipelines with API
  • 26.
    ANTIPATTERN # 7 SLOWPIPELINES ` ` ` Over-engineered Pipelines Pipeline execution takes too long Leads to delayed feedback
  • 27.
    ANTIPATTERN # 6:PROPOSED SOLUTION SLOW PIPELINES ` ` ` Parallelize where you can!
  • 28.
    ANTIPATTERN # 7 MANUALPROMOTION Engineer PRODUATIntegr TestsDEV System Tests ENVENVENV ENV
  • 29.
    ANTIPATTERN # 7:PROPOSED SOLUTION KILL SWITCH FOR MANUAL TESTING PRODUATIntegr TestsDEV System Tests ENVENVENV ENV TEST
  • 30.
    ANTIPATTERN # 7:PROPOSED SOLUTION AB TESTING PROD A Integr TestsDEV System Tests ENV ENVENV TEST A PROD B ENV TEST B feedback feedback measure
  • 31.
    ANTIPATTERN # 7.1 CODEREVIEW PRODUATIntegrTests System Tests ENVENVENV ENV DEV B feature 
 branch DEV C code 
 review
  • 32.
    ANTIPATTERN # 7.1:PROPOSED SOLUTION REACTIVE CODEREVIEW ReleaseIntegr Tests System Tests ENVENV ENV Full 
 Regression Calc technical
 debt ENV Nightly DEV A Standup code 
 review …
  • 33.
    ANTIPATTERN # 7.1 CODEREVIEW PRODUATIntegrTestsDEV A System Tests ENVENVENV ENV master DEV B feature 
 branch DEV C code 
 review
  • 34.
    ANTIPATTERN # 8 DEVDEPLOYMENT VIA CI DEV
 ENV DEV
 ENV DEV
 ENV …DEV CI
  • 35.
    ANTIPATTERN # 8:PROPOSED SOLUTION DEV DEPLOYMENT VIA CI DEV …NO CI DEV
 ENV DEV
 ENV DEV
 ENV
  • 36.
    ANTIPATTERN # 8:PROPOSED SOLUTION DEV DEPLOYMENT VIA CI ENV ENV DEV
 ENV …DEV NO CI WANT TO HACK? SURE!
  • 37.
    ANTIPATTERN # 8:PROPOSED SOLUTION DEV DEPLOYMENT VIA CI ENV ENV DEV
 ENV … YOU BROKE IT? YOU FIX IT! DEV NO CI
  • 38.
  • 39.
    ANTIPATTERN # 9 GOLDENIMAGE VM OS Problems ▸ Maintained manually ▸ No collaboration ▸ Hard to distribute ▸ Non versioning Chnorr Service
  • 40.
    ANTIPATTERN # 9:PROPOSED SOLUTION PROVISIONING OS Chnorr Service
  • 41.
    ANTIPATTERN # 9:PROPOSED SOLUTION PROVISIONING OS Configure Infrastructure
 code Harden Download Install Chnorr Service
  • 42.
    ANTIPATTERN # 9:PROPOSED SOLUTION PROVISIONING OS Configure Infrastructure
 code Harden Download Install cmdb Chnorr Service
  • 43.
    ANTIPATTERN # A FRAGILEPROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  • 44.
    ANTIPATTERN # A FRAGILEPROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  • 45.
    ANTIPATTERN # A FRAGILEPROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  • 46.
    ANTIPATTERN # A FRAGILEPROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  • 47.
    ANTIPATTERN # A.1 PRIVATEDATA CENTER PROVISIONING PRIVATE DATA CENTER: NO INTERNET!!! OS ConfigureHarden Download Install Chnorr Service
  • 48.
    ANTIPATTERN # A.1 PRIVATEDATA CENTER PROVISIONING PRIVATE DATA CENTER: NO INTERNET!!! OS ConfigureHarden Download Install … apt yum gem Chnorr Service
  • 49.
    ANTIPATTERN # A.1 PRIVATEDATA CENTER PROVISIONING PRIVATE DATA CENTER: NO INTERNET!!! OS ConfigureHarden Download Install … apt yum gem complicated? Chnorr Service
  • 50.
    ANTIPATTERN # A.1:PROPOSED SOLUTION STEM CELLS static dynamic OS ConfigureHarden Download Install Chnorr Service
  • 51.
    ANTIPATTERN # A.1:PROPOSED SOLUTION STEM CELLS OS ConfigureHarden Download Install LAB PRIVATE DATA CENTER Chnorr Service
  • 52.
    ANTIPATTERN # A.1:PROPOSED SOLUTION STEM CELLS OS ConfigureHarden Download Install LAB PRIVATE DATA CENTER code packer PROVISIONSNAPSHOT docker Chnorr Service
  • 53.
    ANTIPATTERN # B DEPENDENCYHELL OS Configure Infrastructure
 code Harden Download Install ▸ Version of libraries ▸ Version of packages ▸ Maintain dependencies Chnorr Service
  • 54.
    ANTIPATTERN # B DEPENDENCYHELL OS Configure Infrastructure
 code Harden Download Install ▸ Version of libraries ▸ Version of packages ▸ Maintain dependencies ▸ Version of your infra code ▸ Maintain dependencies Chnorr Service
  • 55.
    ANTIPATTERN # B:PROPOSED SOLUTION CANARY BUILDS Accept Tests CI Unit Test Provision … PRECISE version libs Accept Tests CI Unit Test Provision … LATEST version libs Chnorr Service Chnorr Service
  • 56.
    ANTIPATTERN # C INFRASTRUCTUREPET Attributes of Pet ▸ Have meaningful names ▸ Long living instance ▸ Often needs manual nursing ▸ Requires scary patching ▸ Leads to snowflakes ▸ PaaS is modern pet
  • 57.
    ANTIPATTERN # C:PROPOSED SOLUTION INFRASTRUCTURE CATTLE Attributes of Pet ▸ Have numbers in its name ▸ Short living instance ▸ Immutable configuration ▸ Recreate instead of patching ▸ Requires careful planning
  • 58.
    ANTIPATTERN # C:PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database
  • 59.
    ANTIPATTERN # C:PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database
  • 60.
    ANTIPATTERN # C:PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database Cattle Pet
  • 61.
    ANTIPATTERN # C:PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database Cattle Pet User Data
  • 62.
    ANTIPATTERN # D SECRETSLEAK OS Configure Infrastructure
 code Install cmdb wrong place 
 for your secrets wrong place 
 for your secrets Chnorr Service
  • 63.
    ANTIPATTERN # D:PROPOSED SOLUTION SECRETS LEAK ▸ Don’t store secrets with code ▸ Don’t store secrets with configuration ▸ Don’t leave secrets in service Secrets DON’Ts
  • 64.
    ANTIPATTERN # D:PROPOSED SOLUTION SECRETS LEAK ▸ Secret can be leased and rotated ▸ tmpfs is your fiend Secrets DOs
  • 65.
    ANTIPATTERN # D:PROPOSED SOLUTION SECURITY LEASING EXAMPLE Chnorr Service vault consul IAM Database api x hours leasing AWS
  • 66.
  • 67.
    TAKEAWAYS LEARN PATTERNS BEFORETOOLS Patterns Tools vs
  • 68.
    TAKEAWAYS EVERYTHING MUST HAVEAN API REST DSL CLI
  • 69.
    TAKEAWAYS SELF TESTABLE CODE ▸Use assertions for infrastructure code ▸ Use acceptance test frameworks Tests improves your confidence
  • 70.
    TAKEAWAYS READING ▸ Book: AHuman Error Approach to Aviation Accident Analysis ▸ Author: Douglas A. Wiegmann
 Scott A. Shappell ▸ ISBN: 978-0754618737
  • 71.
    TAKEAWAYS READING ▸ Book: CleanCode ▸ Author: Robert C Martin ▸ ISBN: 978-0132350884
  • 72.