Austin
Copenhagen
Dubai
London
Madrid
Melbourne
Munich
New York
Nice
Raleigh
San Francisco
São Paulo
Singapore
Tokyo
Toronto
Tales from the DevOps
Transformation Trenches
Holly Cummins
IBM Garage
@holly_cummins
yes, you (still) need to start
with culture, not containers
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
hi.
i’m a consultant.
@holly_cummins#IBMGarage
these are my
scary stories
@holly_cummins#IBMGarage
how to fail at
devops
@holly_cummins#IBMGarage
“this is our devops team”
@holly_cummins#IBMGarage
“this is our devops team”
“… last year we called them the build team.”
containers will not fix
your broken devops culture
8
even kubernetes will not fix
your broken devops culture 9
@holly_cummins#IBMGarage
“we’re going too slowly.
we need to get rid of COBOL
and make microservices!”
@holly_cummins#IBMGarage
“we’re going too slowly.
we need to get rid of COBOL
and make microservices!”
“… but our release board
only meets twice a year.”
@holly_cummins#IBMGarage
https://hackernoon.com/8-devops-trends-to-be-aware-of-in-2019-b4232ac8f351
@holly_cummins#IBMGarage
https://hackernoon.com/8-devops-trends-to-be-aware-of-in-2019-b4232ac8f351
@holly_cummins#IBMGarage
“every time we change
code, something breaks”
@holly_cummins#IBMGarage
distributed monolith
@holly_cummins#IBMGarage
distributed monolith
but without compile-time checking
@holly_cummins#IBMGarage
just because a system runs
across 6 containers doesn’t
mean it’s decoupled
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
mars climate
explorer
@holly_cummins#IBMGarage
other people’s
trenches
for clarity: this wasn’t a
client of mine.
@holly_cummins#IBMGarage
Courtesy NASA/
JPL-Caltech
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
metric units
metric units
imperial
units
distributing
did not help
metric units
imperial
units
@holly_cummins#IBMGarage
microservices need
consumer-driven contract tests
@holly_cummins#IBMGarage
$370 million loss
https://en.wikipedia.org/wiki/Cluster_(spacecraft)
Cluster + Ariane 5
@holly_cummins#IBMGarage
$370 million loss
https://en.wikipedia.org/wiki/Cluster_(spacecraft)
Cluster + Ariane 5
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
they tested it …
@holly_cummins#IBMGarage
they tested it …
but stubbed out one component.
@holly_cummins#IBMGarage
they tested it …
but stubbed out one component.
that component was the one that broke.
@holly_cummins#IBMGarage
“Had we done end-to-end testing,
we believe this error would have
been caught.”
Arthur Stephenson
Chief Investigator
@holly_cummins#IBMGarage
microservices need
automated integration tests
@holly_cummins#IBMGarage
“we have a CI/CD”
@holly_cummins#IBMGarage
CI/CD is something
you do, it’s not a
tool you buy
@holly_cummins#IBMGarage
“i’ll merge my branch
into our CI next week”
@holly_cummins#IBMGarage
“CI/CD … CI/CD … CI/CD …
we release every six months …
CI/CD …. ”
@holly_cummins#IBMGarage
continuous.
I don’t think that word means
what you think it means.
@holly_cummins#IBMGarage
how often should you push to master?
@holly_cummins#IBMGarage
how often should you push to master?integrate?
@holly_cummins#IBMGarage
how often should you push to master?
every character
integrate?
@holly_cummins#IBMGarage
how often should you push to master?
every character
integrate?
actually continuous
… but stupid
@holly_cummins#IBMGarage
how often should you push to master?
every character every commit
(several times an hour)
integrate?
actually continuous
… but stupid
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character every commit
(several times an hour)
integrate?
actually continuous
… but stupid
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
integrate?
actually continuous
… but stupid
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
integrate?
actually continuous
… but stupid
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once a
month
integrate?
actually continuous
… but stupid
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
ok
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
ok
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
bad
ok
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
bad
seriously?
ok
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
bad
seriously?
my favourite
ok
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh
ok
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh
ok
hardcore
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh
my favourite
ok
hardcore
@holly_cummins#IBMGarage
how often should you test in staging?
@holly_cummins#IBMGarage
how often should you test in staging?deliver?
@holly_cummins#IBMGarage
how often should you test in staging?
every push
deliver?
my favourite
@holly_cummins#IBMGarage
“we can’t actually release this.”
@holly_cummins#IBMGarage
why?
@holly_cummins#IBMGarage
what’s stopping more
frequent deploys?
@holly_cummins#IBMGarage
“we can’t release this microservice…
we deploy all our microservices at
the same time.”
@holly_cummins#IBMGarage
“it looks like it’s complete … but
nothing works if you click on it.”
@holly_cummins#IBMGarage
front-end
integration layer
back-end
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
front-
end
int.
layer
back-
end
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
front-
end
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
back-
end
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
int.
layer
back-
end
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
front-
end
int.
layer
back-
end
✓ it works by the time anyone sees it
stakeholders need to be careful what they incentivise
@holly_cummins#IBMGarage
vertical slices
@holly_cummins#IBMGarage
back-out
development
@holly_cummins#IBMGarage
back-out
development
back-first development
@holly_cummins#IBMGarage
deferred wiring
@holly_cummins#IBMGarage
feature flags
@holly_cummins#IBMGarage
“we can’t ship until every
feature is complete”
@holly_cummins#IBMGarage
but why?
@holly_cummins#IBMGarage
“users won’t find it compelling
enough if we release now”
@holly_cummins#IBMGarage
if you're not embarrassed by
your first release it was too late
- Reid Hoffman
@holly_cummins#IBMGarage
lean
@holly_cummins#IBMGarage
“we only get one chance
to get it right”
@holly_cummins#IBMGarage
you can’t a/b test a
$370 million rocket
the ariadne failed in 36 seconds
@holly_cummins#IBMGarage
one chance
we think
we’re here
@holly_cummins#IBMGarage
one chance
brand damage
we think
we’re here
@holly_cummins#IBMGarage
one chance
brand damage
we think
we’re here
market failure
(indifference)
@holly_cummins#IBMGarage
one chance
brand damage
continuous
improvement delights
growing user base
we think
we’re here
market failure
(indifference)
@holly_cummins#IBMGarage
a/b testing
one chance
brand damage
continuous
improvement delights
growing user base
we think
we’re here
market failure
(indifference)
@holly_cummins#IBMGarage
a/b testing
one chance
brand damage
continuous
improvement delights
growing user base
we think
we’re here
could we be
here?
market failure
(indifference)
@holly_cummins#IBMGarage
feedback is good
engineering
@holly_cummins#IBMGarage
they often couldn’t
see the explorer
@holly_cummins#IBMGarage
“but our change
control process …”
@holly_cummins#IBMGarage
“this provisioning
software is broken”
@holly_cummins#IBMGarage
what we sold
“this provisioning
software is broken”
10 minute
provision-time
@holly_cummins#IBMGarage
what we sold
“this provisioning
software is broken”
10 minute
provision-time
3 month
provision-
time
what the
client
thought
they’d got
@holly_cummins#IBMGarage
what we sold
“this provisioning
software is broken”
10 minute
provision-time
3 month
provision-
time
what the
client
thought
they’d got
the reason
84-step
pre-approval process
@holly_cummins#IBMGarage
“we’ve scheduled
the architecture
board review for a
month after the
project ships”
@holly_cummins#IBMGarage
does the process
add value?
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
navigators warned
something was wrong
@holly_cummins#IBMGarage
navigators warned
something was wrong
they didn’t fill in the
right form
@holly_cummins#IBMGarage
navigators warned
something was wrong
they didn’t fill in the
right form
so nothing was done
@holly_cummins#IBMGarage
“we can’t ship until we have
more confidence in the
quality”
@holly_cummins#IBMGarage
“we can’t ship until we have
more confidence in the
quality”
you
can fix
that
@holly_cummins#IBMGarage
“this is the test team … who
don’t have the skills to
automate their tests.”
@holly_cummins#IBMGarage
“our tests aren’t
automated”
@holly_cummins#IBMGarage
“our tests aren’t
automated”
@holly_cummins#IBMGarage
“we don’t know if our
code currently works”
@holly_cummins#IBMGarage
“we don’t know if our
code currently works”
@holly_cummins#IBMGarage
“it costs too much
to release”
@holly_cummins#IBMGarage
“it costs too much
to release”you
can fix
that
@holly_cummins#IBMGarage
“we don’t know when
the build is broken”
not a good CI/CD indicator a good CI/CD indicator
@holly_cummins#IBMGarage
get the pipeline status
into the physical
spaces
@holly_cummins#IBMGarage
“only Bob can
change Jenkins”
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
“oh yes, that build
has been broken
for a few weeks…”
@holly_cummins#IBMGarage
judge judge
judge
@holly_cummins#IBMGarage
hybrid and multi-
cloud toolchains and
deployments
toolchains support
lean delivery
processes and
business agility
many, single-tenant
toolchains
modern devops
toolchains and processes reflect

cloud native apps and cultural transformation

@holly_cummins#IBMGarage
on-premise
automation tools
release management
and dependency
coordination are hard
shared, multi-tenant
toolchain “backbone”
heritage devops
toolchains and processes reflect

heritage apps and cultural inertia
@holly_cummins#IBMGarage
“you’ll be coding
on the mainframe”
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
@holly_cummins#IBMGarage
this can
get tiring
@holly_cummins#IBMGarage
transformation
endurance
@holly_cummins#IBMGarage
remember
the why
®
@holly_cummins
IBM Cloud Garage

Tales from the devops transformation trenches