Dev ops with smell v1.2

~WHOAMI
ANTONS KRANGA
▸ Full stack developer ~ 15years
▸ Cloud Architect
▸ DevOps Advocate
▸ Full-Stack Developer
▸ Speaker
▸ Marathon runner
@acankr

TALK STRUCTURE
PRESO PLAN
▸ Smell Symptoms: organizational anti patterns for devops
▸ Vitamins: proposed solutions from devops library
▸ Painkillers: deployment anti patterns
▸ Antibiotics: environment anti patterns brought by devops
pic from wonderful “matrix” movie

DEFINITION OF DEVOPS
What is DevOps?
pic from “hitchhiker's guide to the galaxy” movie

What is DevOps?
BFG900 from classical game DOOM (c) id software

What is DevOps?
▸ Development teams ▸ Operations
▸ Security and compliance ▸ Management
▸ Strategy, IT Strategy
BFG900 from classical game DOOM (c) id software
▸ QA▸ HR

DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed

DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
▸ specialization silos
▸ ops comes ﬁrst
▸ cross functional
▸ software deﬁned 
data centers
▸ complex communication ▸ complexity theory

SMELL SYMPTOM
DEVOPS ANTIPATTERNS

ANTIPATTERN # 0
BI MODAL IT
Sparky from Frankenweenie
▸ Slow IT
▸ Safe and Predictable
▸ IT Centric
▸ Secure and Regulated
▸ Fast IT
▸ Fast and Agile
▸ Business Centric
▸ Innovative

ANTIPATTERN # 1
DEVOPS IN A BOX
You cannot buy culture!
▸ Buy a Golden DevOps LVL5
▸ Become certiﬁed DevOps master
▸ Give me DevOps compliance checklist
▸ Give me 5 key DevOps control metrics

ANTIPATTERN # 1.1
REBRANDED IT
DevOps != Conﬁguration Management 
DevOps != Release Management| 
DevOps != Product Management 
… 
DevOps != (*) Management
DEVOPS IS THE CULTURE YOU
CANNOT FIND IN IN ITIL CATALOG

ANTIPATTERN # 2
DEVOPS AS SEPARATE TEAM
You don’t need to change
anything in your org
You extend organization
with new capabilities

ANTIPATTERN # 2
DEVOPS AS SEPARATE TEAM
Creates Unicorn vs Horses dilemma
DevOps builds their own Silo
Often loses focus on value
and speed

ANTIPATTERN # 2.1
DEVOPS AS A SILO
DevOps teams build their own Silo
▸ You cannot talk to DevOps
▸ Use Jira instead !!!!
▸ RTFM Architecture

ANTIPATTERN # 2: PROPOSED SOLUTION
SORRY, NOT MY DEPARTMENT
Bread ownership and
specialization with
autonomous teams
vs
http://martinfowler.com/bliki/DevOpsCulture.html

ANTIPATTERN # 3
DEVOPS IN AGILE
▸ DoD fails with “ilities”
▸ Services can be easily “undone”

ANTIPATTERN # 4
FEAR OF RELEASE
RELEASE === RISK
NO RELEAE NO REVENUE
vs

ANTIPATTERN # 5
SNOWFLAKE SERVER
Applying changes to Server Instance
manually leads to unique and distinct
server conﬁguration footprint
(TECHNICAL DEBT)
http://martinfowler.com/bliki/SnowﬂakeServer.html

VITAMINS
INFRASTRUCTURE AS CODE
ENV
ENV
ENV
DEV …CODE

VITAMINS
DEV
Code is a 
Contract
OPS
Essence to collectively create and share knowledge

VITAMINS
DEV
Code is a 
Contract
OPS
Enables: built in quality

VITAMINS
DEV
Code is a 
Contract
OPS
Put infrastructure into SCM

VITAMINS
DEV
Code is a 
Contract
OPS`
Make infrastructure part of app

VITAMINS
USE CI ENVIRONMENT
CI
test infra depl AT
..
.
release

VITAMINS
USE CI ENVIRONMENT
▸ We need feedback not to be afraid
▸ Feedback != SPAM
▸ It’s about size of release not frequency
▸ Don’t judge for broken builds
▸ Go home when build is green

PAINKILLERS
DEVOPS ANTIPATTERNS

CONTINUOUS DELIVERY
CI/CD PROCESS
CI
test infra depl AT
..
.
release

ANTIPATTERN # 6
SLOW PIPELINES
CI
..
.
release
test infra depl DT
test DT depl infra
infra depl DT AT

SLOW PIPELINES
`
`
`
Design for parallelization

ANTIPATTERN # 6
“JENKINS” DRIVEN DEVELOPMENT
With single master CI you easily get a
single point of failure
CI master

MULTI MASTER “JENKINS”
Multi-master CI can dedicate CI Master
per group of related components
CI master CI master CI master

ANTIPATTERN # 6: PROPOSED SOLUTION # 2
SHARE RESPONSIBILITY
CI users (DEVs or OPs) are best for managing their jobs
▸ Implement Pipeline as Code
▸ Store CI/CD pipelines in git
▸ Let pipeline evolve together with your app

ANTIPATTERN # 7
MANUAL PROMOTION
Tester PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV

KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST

KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
Time adjustable 
if TREND is good

AB TESTING
PROD A
Integr TestsDEV System Tests
ENV
ENVENV
TEST A
PROD B
ENV
TEST B
feedback
feedback
measure

ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr Tests System Tests
ENVENVENV ENV
DEV B
feature  
branch DEV C
code  
review

ANTIPATTERN # 7.1: PROPOSED SOLUTION
REACTIVE CODEREVIEW
ReleaseIntegr Tests System Tests
ENVENV ENV
Full  
Regression
Calc technical 
debt
ENV
Nightly
DEV A
Standup
code  
review
…

ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr TestsDEV A System Tests
ENVENVENV ENV
master
DEV B
feature  
branch DEV C
code  
review

ANTIPATTERN # 7.2
GIT FLOWS
▸ Git ﬂow helps to work feature dev isolated
▸ Indirectly breaks Continuous Delivery Principle
▸ Complex merge routine distracts teams
http://classicprogrammerpaintings.com/post/142586036029/junior-programmer-learns-git-rebase

GIT FLOWS
develop
master
test
merge
CI
DEV
pull
push

GIT FLOWS
develop
master
test
merge
CI
DEV
DEV
pull
push

ANTIPATTERN # 8
DEV DOESN’T CONTROL DEV ENV
DEV 
ENV
DEV 
ENV
DEV 
ENV
…DEV CI

DEV …NO CI
DEV 
ENV
DEV 
ENV
DEV 
ENV

ENV
ENV
DEV 
ENV
…DEV NO CI
WANT TO HACK? SURE!

ENV
ENV
DEV 
ENV
…
YOU BROKE IT? YOU FIX IT!
DEV NO CI

ANTIBIOTICS
DEVOPS ANTIPATTERNS

ANTIPATTERN # 9
GOLDEN IMAGE
VM
OS
Problems
▸ Maintained manually
▸ No collaboration
▸ Hard to distribute
▸ Non versioning
Chnorr Service

PROVISIONING
OS Chnorr Service

PROVISIONING
OS
Conﬁgure
Infrastructure 
code
Harden Download Install
Chnorr Service

PROVISIONING
OS
Conﬁgure
Infrastructure 
code
cmdb
Chnorr Service

ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Conﬁgure
Infrastructure 
code
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service

ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConﬁgureHarden Download Install
Chnorr Service

ANTIPATTERN # A.1
OS
…
apt yum gem
Chnorr Service

ANTIPATTERN # A.1
OS
…
apt yum gem
complicated?
Chnorr Service

ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
static dynamic
OS
Chnorr Service

STEM CELLS
OS
LAB PRIVATE DATA CENTER
Chnorr Service

STEM CELLS
OS
LAB PRIVATE DATA CENTER
code
packer
PROVISION
MAKE 
SNAPSHOT
docker
Chnorr Service
RECONFIGURE

ANTIPATTERN # B
DEPENDENCY HELL
OS
Conﬁgure
Infrastructure 
code
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
Chnorr Service

ANTIPATTERN # B
DEPENDENCY HELL
OS
Conﬁgure
Infrastructure 
code
▸ Version of libraries
▸ Version of packages
▸ Version of your infra code
Chnorr Service

ANTIPATTERN # B: PROPOSED SOLUTION
CANARY BUILDS
Accept Tests
CI
Unit Test Provision
…
PRECISE version libs
Accept Tests
CI
Unit Test Provision
…
LATEST version libs
Chnorr Service
Chnorr Service

ANTIPATTERN # C
INFRASTRUCTURE PET
Attributes of Pet
▸ Have meaningful names
▸ Long living instance
▸ Often needs manual nursing
▸ Requires scary patching
▸ Leads to snowﬂakes
▸ PaaS is modern pet

ANTIPATTERN # C: PROPOSED SOLUTION
INFRASTRUCTURE CATTLE
Attributes of Pet
▸ Have numbers in its name
▸ Short living instance
▸ Immutable conﬁguration
▸ Recreate instead of patching
▸ Requires careful planning

PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database

Chnorr Service
Database
Cattle
Pet

Chnorr Service
Database
Cattle
Pet
User Data

ANTIPATTERN # D
SECRETS LEAK
OS
Conﬁgure
Infrastructure 
code
Install
cmdb
wrong place  
for your secrets
wrong place  
for your secrets
Chnorr Service

ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Don’t store secrets with code
▸ Don’t store secrets with conﬁguration
▸ Don’t leave secrets in service
Secrets DON’Ts

SECRETS LEAK
▸ Secret can be leased and rotated
▸ tmpfs is your ﬁend
Secrets DOs

SECURITY LEASING EXAMPLE
Chnorr Service vault
consul
IAM
Database
api
x hours leasing
AWS

TAKEAWAYS
PATTERNS GOES BEFORE TOOLS
Patterns Tools
vs

TAKEAWAYS
EVERYTHING MUST HAVE AN API
REST
DSL
CLI

TAKEAWAYS
SPLIT YOUR SECRETS FROM CODE OR CONFIGURATION
(c) @eduardsi

TAKEAWAYS
SELF TESTABLE CODE
▸ Use assertions for infrastructure code
▸ Use acceptance test frameworks
Tests improves your conﬁdence
CHEF MINITEST
package "mysql"
assert_file "/etc/my.cnf", "mysql", "mysql", "644"
SHELL
apt-get install -yqq mysql
test ! -f /etc/my.cnf || error_exit “boo”

TAKEAWAYS
HOW TO START DEVOPS
▸ Fastest way to start with Vitamins and Coaching
▸ Apply measures early
▸ Apply right measures
▸ Tools comes later

TAKEAWAYS
READING
▸ Book: The Phoenix Project
▸ Author: Gene Kim 
Kevin Behr 
George Stafford
▸ ISBN: 978-0988262508

TAKEAWAYS
READING
▸ Book: The Lean Startup
▸ Author: Eric Ries
▸ ISBN: 978-0307887894

TAKEAWAYS
READING
▸ Book: A Human Error Approach to
Aviation Accident Analysis
▸ Author: Douglas A. Wiegmann 
Scott A. Shappell
▸ ISBN: 978-0754618737

THANK YOU
Download me here: http://www.slideshare.net/akranga/dev-ops-with-smell-v12
Twitter: @acankr

Dev ops with smell v1.2

More Related Content

What's hot

Viewers also liked

Similar to Dev ops with smell v1.2

More from Antons Kranga

Recently uploaded

Dev ops with smell v1.2