DevOps: It’s Not Just For WebOps
And We Have The Metrics To Prove It
Gene
Kim
(IT
Revolu0on
Press)
Jez
Humble
(ThoughtWorks)
James
Turnbull
(Puppet
Labs)
John
Willis
(Enstra0us
/
Dell)

Presenters
James Turnbull, VP of Community, Evangelism, and Business
Development
• Prominent voice in DevOps community
• Author of several technical books including The LogStash
Book
Gene Kim, IT researcher and author
• Founder and former CTO of Tripwire for 13 years
• Author "The Phoenix Project: A Novel About IT, DevOps, and
Helping Your Business Win” and “Visible Ops Handbook”
Jez Humble, Principal, ThoughtWorks
• Co-author of Continuous Delivery
• Loud mouth on effective practices for software delivery

The Big Why?
• Desire
to
understand
the
health
and
habits
of
DevOps
community
• Understand
the
link
between
behaviors
and
performance
• Mi0gate
the
“echo
chamber”
effect:
what
is
the
evidence
of
improved
performance
• Prove
that
DevOps
isn’t
just
for
unicorns…
it’s
for
the
horses,
too.

Goals For This Presentation
• Show
you
how
to
do
cross-­‐sec0on
popula0on
studies
• Survey
Design
• Survey
Execu0on
• Survey
Analysis
• Share
our
top
findings
• Get
your
help
for
our
2013-­‐2014
study

Where Did The High Performers Come From?

Over Ten Years, We Benchmarked 1500+ IT Orgs

Intuitive Analytical
Academic
Study
Population
Sample
Population
Sample
Academic
Study
Population
Sample
What Is Cross-Section Population Study
Population
Sample
Academic
Study
Population
Study
Benchmark
ROI Story
Best
Practice
Peer
Recommendation
Anecdote
Prior Experience
We Are
Here
Vendors
Consultants
Industry
Analyst
Tradeshows
Industry
Press
Blogshere

High Performing IT Organizations
§ High
performers
maintain
a
posture
of
compliance
§ Fewest
number
of
repeat
audit
findings
§ One-­‐third
amount
of
audit
prepara0on
effort
§ High
performers
find
and
fix
security
breaches
faster
§ 5
/mes
more
likely
to
detect
breaches
by
automated
control
§ 5
/mes
less
likely
to
have
breaches
result
in
a
loss
event
§ When
high
performers
implement
changes…
§ 14
/mes
more
changes
§ One-­‐half
the
change
failure
rate
§ One-­‐quarter
the
first
fix
failure
rate
§ 10x
faster
MTTR
for
Sev
1
outages
§ When
high
performers
manage
IT
resources…
§ One-­‐third
the
amount
of
unplanned
work
§ 8
/mes
more
projects
and
IT
services
§ 6
/mes
more
applica0ons
Source: IT Process Institute, 2008

Visible Ops: Playbook of High Performers
• The IT Process Institute
has been studying high-
performing organizations
since 1999
• What is common to all the
high performers?
• What is different between
them and average and low
performers?
• How did they become great?
www.ITPI.org

The Story Begins…
Picture credit: Flickr: Floralgal

I Know!
12

Lesson:
You are only as smart as the
average
of the top 5 people you hang out with
13

2012 DevOps Survey
• Written by Puppet Labs, Gene Kim, and Jez Humble
• Ran through December of 2012
• Goal: Get a snapshot of the current state of the health
and habits of the DevOps community

How To Do A Cross Section Population Study
• Survey
design
• Survey
execu0on
• Survey
analysis

The Vision: DevOps Survey Of Practice II
• Performance
• Behaviors
• Demographics
• Psychographics
• Planned
target
was
30
ques0ons
• We
asked
for
a
budget
of
8
ques0ons
16

Survey Design
In
the
ideal,
we
want
to
be
able
to
say,
“the
more
you
do
[A,
B,
C],
the
more
performance
improves”
X axis: independent variables
DevOps behaviors hypothesized to improve performance
Y axis: dependent variables
DevOps performance measures

Y-Axis: What Performance Looks Like
• How
ofen
do
you
deploy
code?
(e.g.,
daily,
weekly,
monthly,
yearly)
• What
is
the
average
lead
0me
of
a
change?
(i.e.,
how
long
does
it
take
to
go
from
“code
commihed”
to
“code
successfully
running
in
produc0on?”)
• What
percentage
of
your
changes
require
rollbacks
or
hoiixes?
• On
average,
how
long
does
it
take
to
restore
service
when
something
goes
wrong?
(e.g.,
a
day,
a
week,
a
month,
a
year)
18

X-Axis: What Behaviors Result In Performance
• Are
environment
and
infrastructure
changes
(i.e.,
everything
except
for
code)
checked
into
revision
control?
• Is
there
an
automated
process
to
deploy
environment
and
infrastructure
changes?
• Who
performs
code
deployments?
(e.g.,
Dev,
Ops,
both)
• Who
is
on
the
hook
for
produc0on
support
(e.g.,
Dev,
Ops,
both)
19

Next Step: Survey Execution
• Once
the
survey
instrument
is
ready,
then
you
need
people
to
take
the
survey
• Michelle
Carroll
and
the
Puppet
Labs
marke0ng
machine
went
to
work
• Mailing
lists
• Twiher
• Reddit

21

The Results
• 4039
completed
survey
responses
in
30
days!!!
• This
is
amazing.
With
a
primary
research
firm,
this
type
of
popula0on
size
usually
would
cost
$100-­‐200K!
22

Lesson: Do Not Change Answers Mid-Survey

Uh, Oh… Now what?

Lesson:
Get data analyst on the team BEFORE you
do survey design!
25

Where Do You Find Your Own Wally?
• Steal
a
consultant
;)
• R
User
Groups
• University
students
in
search
of
a
thesis
• Economics
students
a
plus
(sta0s0cs,
applied)
• Ex-­‐Wall
Street
investors
(not
recommended)
Wally Zabaglio, Puppet Labs Data Analyst
– Former specialized analytics consultant at The Nielsen
Company; analyst for the US Department of Energy's
Office of Environmental Management
– Primary analyst on DevOps survey results

27

Data Cleanup
• Used
Google
Refine
to
clean
up
data
and
compensate
for
“Gene
screwup”
• Used
R
to
do
analysis
• Look
for
signal
• Explore
data
• Run
correla0ons

Key Findings

High Performing DevOps Teams
• They’re
more
agile
• 30x
more
frequent
deployments
• 8,000x
shorter
lead
0me
(minutes/hours
vs.
months/quarters)
• They’re
more
reliable
• 2x
the
change
success
rate
• 12x
faster
MTTR

31
Technical Practices Matter
• 89% are using infrastructure version control
• 82% are using automated code deployments

Deliberate Practice Matters
Organizations that implemented DevOps practices over 12
months ago were 5x more likely to be high performing than
organizations that weren’t implementing DevOps at all.

The Lost Hypotheses *
o Who performs code deployments?
o Who is on the hook for production support?

Measuring Culture
“I’ll tell you EXACTLY what devops means.
Devops means giving a shit about your job enough to not pass
the buck. Devops means giving a shit about your job enough to
want to learn all the parts and not just your little world.
Developers need to understand infrastructure. Operations
people need to understand code. People need to fucking work
with each other and not just occupy space next to each other.”
John Vincent | @lusis | http://bit.ly/12DkRhf

Measuring Culture
Trust (and Verify)
Continuous Improvement vs Risk Management Theater
“Human Error”
Heroes / "high performers"

The tribe is operationally strong *
70%
Opera0ons
people

The tribe is operationally strong
This
is
not
a
Dev
takeover

The tribe is enterprise too!
26%
-­‐
500
to
9999
16%
-­‐
10K
plus

Overcoming barriers to DevOps adoption
Biggest barriers were cultural
• Lack of manager buy-in
• Lack of team buy-in
• Value of DevOps not understood outside my
group

What are organizations looking for in DevOps skills?
• Coding/scripting (84%)
• People skills (60%)
• Process re-engineering skills (56%)
• Experience with specific tools (19%)

Challenges to culture
It's
easier
to
teach
tools
than
teach
people
skills

Preparing For DevOps Survey 2013-2014
• How
do
we
measure
culture?
• High
management
vs.
low
trust
management
styles
(and
link
to
team
size)
• Industry
code
• Beher
define
performance
variables
to
enable
regression
analysis
(Likert-­‐type
scale)
• Business
cri0cality:
system
of
engagement
vs.
system
of
record
• What
else?
We
want
to
hear
your
thoughts!

If you … Then …
… want to learn more about the findings • Read the full-length report:
http://puppetlabs.com/devops
• Share the infographic:
http://puppetlabs.com/2013-state-of-devops-
infographic
• Get these slides:
http://slideshare.net/realgenekim/
DevOpsSurveyOfPractice
… have ideas on the DevOps Survey Of
Practice 2013-2014
• Come talk with us!
• Join our Google+ Community:
https://plus.google.com/communities/
104363243715545285555
… share your stories of transformation • Go to FlowCon! http://flowcon.org
… get a free copy of The Phoenix Project:
A Novel About IT, DevOps, and Helping
Your Business Win
• Go to the AppFirst booth at 2:40pm. Gene
will be signing books!