1.
Austin
Copenhagen
Dubai
London
Madrid
Melbourne
Munich
New York
Nice
Raleigh
San Francisco
São Paulo
Singapore
Tokyo
Toronto
Tales from the DevOps
Transformation Trenches
Holly Cummins
IBM Garage
@holly_cummins
yes, you (still) need to start
with culture, not containers

2.
@holly_cummins#IBMGarage

3.
@holly_cummins#IBMGarage

4.
@holly_cummins#IBMGarage

5.
@holly_cummins#IBMGarage

6.
@holly_cummins#IBMGarage

7.
@holly_cummins#IBMGarage

8.
@holly_cummins#IBMGarage
hi.
i’m a consultant.

9.
@holly_cummins#IBMGarage
these are my
scary stories

10.
@holly_cummins#IBMGarage
how to fail at
devops

11.
@holly_cummins#IBMGarage
“this is our devops team”

12.
@holly_cummins#IBMGarage
“this is our devops team”
“… last year we called them the build team.”

13.
containers will not fix
your broken devops culture
8

14.
even kubernetes will not fix
your broken devops culture 9

15.
@holly_cummins#IBMGarage
“we’re going too slowly.
we need to get rid of COBOL
and make microservices!”

16.
@holly_cummins#IBMGarage
“we’re going too slowly.
we need to get rid of COBOL
and make microservices!”
“… but our release board
only meets twice a year.”

17.
@holly_cummins#IBMGarage
https://hackernoon.com/8-devops-trends-to-be-aware-of-in-2019-b4232ac8f351

18.
@holly_cummins#IBMGarage
https://hackernoon.com/8-devops-trends-to-be-aware-of-in-2019-b4232ac8f351

19.
@holly_cummins#IBMGarage
“every time we change
code, something breaks”

20.
@holly_cummins#IBMGarage
distributed monolith

21.
@holly_cummins#IBMGarage
distributed monolith
but without compile-time checking

22.
@holly_cummins#IBMGarage
just because a system runs
across 6 containers doesn’t
mean it’s decoupled

23.
@holly_cummins#IBMGarage

24.
@holly_cummins#IBMGarage
mars climate
explorer

25.
@holly_cummins#IBMGarage
other people’s
trenches
for clarity: this wasn’t a
client of mine.

26.
@holly_cummins#IBMGarage
Courtesy NASA/
JPL-Caltech

27.
@holly_cummins#IBMGarage

28.
@holly_cummins#IBMGarage

29.
metric units

30.
metric units
imperial
units

31.
distributing
did not help
metric units
imperial
units

32.
@holly_cummins#IBMGarage
microservices need
consumer-driven contract tests

33.
@holly_cummins#IBMGarage
$370 million loss
https://en.wikipedia.org/wiki/Cluster_(spacecraft)
Cluster + Ariane 5

34.
@holly_cummins#IBMGarage
$370 million loss
https://en.wikipedia.org/wiki/Cluster_(spacecraft)
Cluster + Ariane 5

35.
@holly_cummins#IBMGarage

36.
@holly_cummins#IBMGarage
they tested it …

37.
@holly_cummins#IBMGarage
they tested it …
but stubbed out one component.

38.
@holly_cummins#IBMGarage
they tested it …
but stubbed out one component.
that component was the one that broke.

39.
@holly_cummins#IBMGarage
“Had we done end-to-end testing,
we believe this error would have
been caught.”
Arthur Stephenson
Chief Investigator

40.
@holly_cummins#IBMGarage
microservices need
automated integration tests

41.
@holly_cummins#IBMGarage
“we have a CI/CD”

42.
@holly_cummins#IBMGarage
CI/CD is something
you do, it’s not a
tool you buy

43.
@holly_cummins#IBMGarage
“i’ll merge my branch
into our CI next week”

44.
@holly_cummins#IBMGarage
“CI/CD … CI/CD … CI/CD …
we release every six months …
CI/CD …. ”

45.
@holly_cummins#IBMGarage
continuous.
I don’t think that word means
what you think it means.

46.
@holly_cummins#IBMGarage
how often should you push to master?

47.
@holly_cummins#IBMGarage
how often should you push to master?integrate?

48.
@holly_cummins#IBMGarage
how often should you push to master?
every character
integrate?

49.
@holly_cummins#IBMGarage
how often should you push to master?
every character
integrate?
actually continuous
… but stupid

50.
@holly_cummins#IBMGarage
how often should you push to master?
every character every commit
(several times an hour)
integrate?
actually continuous
… but stupid

51.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character every commit
(several times an hour)
integrate?
actually continuous
… but stupid

52.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
integrate?
actually continuous
… but stupid

53.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
integrate?
actually continuous
… but stupid

54.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once a
month
integrate?
actually continuous
… but stupid

55.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid

56.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development

57.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
ok

58.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
ok

59.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
bad
ok

60.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
bad
seriously?
ok

61.
@holly_cummins#IBMGarage
how often should you push to master?
every few commits
(several times a day)
every character
once a day
every commit
(several times an hour)
once a
week
once every
six months
once a
month
integrate?
actually continuous
… but stupid
trunk-based
development
bad
bad
seriously?
my favourite
ok

62.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter

63.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?

64.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)

65.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok

66.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school

67.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh

68.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh
ok

69.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh
ok
hardcore

70.
@holly_cummins#IBMGarage
how often should you release?
every user story
every push
(many times a day)
every epic once a sprint
once
every two
years
once a
quarter
deploy?
(need a good handle on
feature flags)
ok
old-
school
sigh
my favourite
ok
hardcore

71.
@holly_cummins#IBMGarage
how often should you test in staging?

72.
@holly_cummins#IBMGarage
how often should you test in staging?deliver?

73.
@holly_cummins#IBMGarage
how often should you test in staging?
every push
deliver?
my favourite

74.
@holly_cummins#IBMGarage
“we can’t actually release this.”

75.
@holly_cummins#IBMGarage
why?

76.
@holly_cummins#IBMGarage
what’s stopping more
frequent deploys?

77.
@holly_cummins#IBMGarage
“we can’t release this microservice…
we deploy all our microservices at
the same time.”

78.
@holly_cummins#IBMGarage
“it looks like it’s complete … but
nothing works if you click on it.”

79.
@holly_cummins#IBMGarage
front-end
integration layer
back-end

80.
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story

81.
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
front-
end
int.
layer
back-
end

82.
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
front-
end

83.
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story

84.
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
back-
end

85.
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
int.
layer
back-
end

86.
@holly_cummins#IBMGarage
front-end
integration layer
back-end
user
story
front-
end
int.
layer
back-
end
✓ it works by the time anyone sees it
stakeholders need to be careful what they incentivise

87.
@holly_cummins#IBMGarage
vertical slices

88.
@holly_cummins#IBMGarage
back-out
development

89.
@holly_cummins#IBMGarage
back-out
development
back-first development

90.
@holly_cummins#IBMGarage
deferred wiring

91.
@holly_cummins#IBMGarage
feature flags

92.
@holly_cummins#IBMGarage
“we can’t ship until every
feature is complete”

93.
@holly_cummins#IBMGarage
but why?

94.
@holly_cummins#IBMGarage
“users won’t find it compelling
enough if we release now”

95.
@holly_cummins#IBMGarage
if you're not embarrassed by
your first release it was too late
- Reid Hoffman

96.
@holly_cummins#IBMGarage
lean

97.
@holly_cummins#IBMGarage
“we only get one chance
to get it right”

98.
@holly_cummins#IBMGarage
you can’t a/b test a
$370 million rocket
the ariadne failed in 36 seconds

99.
@holly_cummins#IBMGarage
one chance
we think
we’re here

100.
@holly_cummins#IBMGarage
one chance
brand damage
we think
we’re here

101.
@holly_cummins#IBMGarage
one chance
brand damage
we think
we’re here
market failure
(indifference)

102.
@holly_cummins#IBMGarage
one chance
brand damage
continuous
improvement delights
growing user base
we think
we’re here
market failure
(indifference)

103.
@holly_cummins#IBMGarage
a/b testing
one chance
brand damage
continuous
improvement delights
growing user base
we think
we’re here
market failure
(indifference)

104.
@holly_cummins#IBMGarage
a/b testing
one chance
brand damage
continuous
improvement delights
growing user base
we think
we’re here
could we be
here?
market failure
(indifference)

105.
@holly_cummins#IBMGarage
feedback is good
engineering

106.
@holly_cummins#IBMGarage
they often couldn’t
see the explorer

107.
@holly_cummins#IBMGarage
“but our change
control process …”

108.
@holly_cummins#IBMGarage
“this provisioning
software is broken”

109.
@holly_cummins#IBMGarage
what we sold
“this provisioning
software is broken”
10 minute
provision-time

110.
@holly_cummins#IBMGarage
what we sold
“this provisioning
software is broken”
10 minute
provision-time
3 month
provision-
time
what the
client
thought
they’d got

111.
@holly_cummins#IBMGarage
what we sold
“this provisioning
software is broken”
10 minute
provision-time
3 month
provision-
time
what the
client
thought
they’d got
the reason
84-step
pre-approval process

112.
@holly_cummins#IBMGarage
“we’ve scheduled
the architecture
board review for a
month after the
project ships”

113.
@holly_cummins#IBMGarage
does the process
add value?

114.
@holly_cummins#IBMGarage

115.
@holly_cummins#IBMGarage
navigators warned
something was wrong

116.
@holly_cummins#IBMGarage
navigators warned
something was wrong
they didn’t fill in the
right form

117.
@holly_cummins#IBMGarage
navigators warned
something was wrong
they didn’t fill in the
right form
so nothing was done

118.
@holly_cummins#IBMGarage
“we can’t ship until we have
more confidence in the
quality”

119.
@holly_cummins#IBMGarage
“we can’t ship until we have
more confidence in the
quality”
you
can fix
that

120.
@holly_cummins#IBMGarage
“this is the test team … who
don’t have the skills to
automate their tests.”

121.
@holly_cummins#IBMGarage
“our tests aren’t
automated”

122.
@holly_cummins#IBMGarage
“our tests aren’t
automated”

123.
@holly_cummins#IBMGarage
“we don’t know if our
code currently works”

124.
@holly_cummins#IBMGarage
“we don’t know if our
code currently works”

125.
@holly_cummins#IBMGarage
“it costs too much
to release”

126.
@holly_cummins#IBMGarage
“it costs too much
to release”you
can fix
that

127.
@holly_cummins#IBMGarage
“we don’t know when
the build is broken”
not a good CI/CD indicator a good CI/CD indicator

128.
@holly_cummins#IBMGarage
get the pipeline status
into the physical
spaces

129.
@holly_cummins#IBMGarage
“only Bob can
change Jenkins”

130.
@holly_cummins#IBMGarage

131.
@holly_cummins#IBMGarage
“oh yes, that build
has been broken
for a few weeks…”

132.
@holly_cummins#IBMGarage
judge judge
judge

133.
@holly_cummins#IBMGarage
hybrid and multi-
cloud toolchains and
deployments
toolchains support
lean delivery
processes and
business agility
many, single-tenant
toolchains
modern devops
toolchains and processes reflect
cloud native apps and cultural transformation

134.
@holly_cummins#IBMGarage
on-premise
automation tools
release management
and dependency
coordination are hard
shared, multi-tenant
toolchain “backbone”
heritage devops
toolchains and processes reflect
heritage apps and cultural inertia

135.
@holly_cummins#IBMGarage
“you’ll be coding
on the mainframe”

136.
@holly_cummins#IBMGarage

137.
@holly_cummins#IBMGarage

138.
@holly_cummins#IBMGarage
this can
get tiring

139.
@holly_cummins#IBMGarage
transformation
endurance

140.
@holly_cummins#IBMGarage
remember
the why

141.
®
@holly_cummins
IBM Cloud Garage