SlideShare a Scribd company logo
@nicolefv
@jezhumble
What We Learned from
Four Years of Sciencing
the Crap Out of DevOps
Nicole Forsgren, PhD
Jez Humble
@nicolefv
@jezhumble
Outline
How to make your data suck less
• Writing good survey questions
• Making sure the survey questions are good
- with SCIENCE
• (These methods apply to your system and
log data)
What we found… that we did (AND didn’t) expect
Things about Continuous Delivery
Things about Management
@nicolefv
@jezhumble
Not all data is created equal
Who here thinks surveys are sh*t?
@nicolefv
@jezhumble
Not all data is created equal
Who here thinks surveys are sh*t?
Who here LOVES the data from their log files?
@nicolefv
@jezhumble
What is a Latent Construct?
@nicolefv
@jezhumble
We use
PSYCHOMETRICS
to make our survey data good*
*or give us a reasonable assurance that it’s telling us
what we think it’s telling us (& some of this can also apply
to your log data)
@nicolefv
@jezhumble
Psychometrics includes:
Construct creation (manual)
• When possible: use previously validated
constructs
• Based on definitions and theory, carefully and
precisely worded, card sorting task, pilot tested
Construct evaluation (statistics)
• Establishing Validity: discriminant and
convergent
• Establishing Reliability
@nicolefv
@jezhumble
Psychometrics Writing
Example: Culture
• Does it matter to our study?
• More than just intuition?
• What KIND of culture?
• National identity and norms
• Adaptive culture
• Value learning (2014 study)
• Value information flow and trust (2014 and 2015
studies -- Westrum culture)
@nicolefv
@jezhumble
Psychometrics Writing
Example: Culture
• Does it matter to our study?
• More than just intuition?
• What KIND of culture?
• National identity and norms
• Adaptive culture
• Value learning (2014 study)
• Value information flow and trust (2014 and 2015
studies -- Westrum culture)
@nicolefv
@jezhumble
Westrum typology
Pathological
Power-oriented
Bureaucratic
Rule-oriented
Generative
Performance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented
Try writing
items
yourself!
Use strong
statements
with clear
language.
Westrum,	R.	(2004).	A	typology	of	organisational cultures. Quality	and	safety	in	health	care, 13(suppl 2),	ii22-ii27.
@nicolefv
@jezhumble
Westrum Culture Items
• On my team, information is actively sought.
• On my team, failures are learning
opportunities, and messengers of them are not
punished.
• On my team, responsibilities are shared.
• On my team, cross-functional collaboration is
encouraged and rewarded.
• On my team, failure causes inquiry.
• On my team, new ideas are welcomed.
Found
to be
valid &
reliable
Predictive of
IT Performance &
Organizational
Performance
@nicolefv
@jezhumble
Psychometrics Analysis
Example: Notification of Failure
Original
in 2014,
but there
was a
surprise.
Can you
spot it?
At my organization…
• We are primarily notified of failures by reports from
customers.
• We are primarily notified of failures by the NOC.
• We get failure alerts from logging and monitoring
systems.
• We monitor system health based on threshold
warnings (ex. CPU exceeds 100%).
• We monitor system health based on rate-of-change
warnings (ex. CPU usage has increased by 25% over the
last 10 minutes).
@nicolefv
@jezhumble
At my organization…
• We are primarily notified of failures by reports from
customers.
• We are primarily notified of failures by the NOC.
• We get failure alerts from logging and monitoring
systems.
• We monitor system health based on threshold
warnings (ex. CPU exceeds 100%).
• We monitor system health based on rate-of-change
warnings (ex. CPU usage has increased by 25% over the
last 10 minutes).
Notification
from
NEAR
Notification
from
FAR
Psychometrics Analysis
Example: Notification of Failure
@nicolefv
@jezhumble
More data tests!
Plus, we test to make sure the survey doesn’t have
other problems.
• Common method variance (CMV) (aka CMB for
Bias)
• Early vs. late responders
• Survey drop-off rates and bias
@nicolefv
@jezhumble
Okay NOW we can look at the
data and how it relates to each
other
@nicolefv
@jezhumble
A note about analysis methods
One of three conditions must be met:
1. Longitudinal (no, this is cross-sectional)
2. Randomized, experimental design (no, this
is a non-experimental)
3. Theory-based design
When this condition was not met, only
correlations were tested and reported
@nicolefv
@jezhumble
KEY FINDING:
IT Performance and its
behavior
A combination of throughput and stability
• lead time for changes
• release frequency
• time to restore service
• change fail rate
Forsgren,	N.,	J.	Humble	(2016).	"DevOps:	Profiles	in	ITSM	Performance	and	Contributing	Factors."	In	the	
Proceedings	of	the	Western	Decision	Sciences	Institute	(WDSI)	2016,	Las	Vegas,	 NV.
@nicolefv
@jezhumble
KEY FINDING:
IT performance matters!
“Firms with high-performing IT organizations were
twice as likely to exceed their profitability, market
share and productivity goals.”
IT Performance is predictive of
organizational performance.
http://bit.ly/2014-devops-report/
http://bit.ly/2015-devops-report/
Forsgren,	N.,	J.	Humble	(2016).	“The	Role	of	Continuous	Delivery	in	IT	and	Organizational	
Performance.”	In	the	Proceedings	of	the	Western	Decision	Sciences	Institute	(WDSI)	
2016,	Las	Vegas,	NV.
@nicolefv
@jezhumble
@nicolefv
@jezhumble
@nicolefv
@jezhumble
some surprises
@nicolefv
@jezhumble
Which of these measure effective test
practices?
Developers primarily create & maintain acceptance tests
QA primarily create & maintain acceptance tests
Primarily created & maintained by outsourced party
When automated tests pass, I’m confident the software is releasable
Test failures are likely to indicate a real defect
It’s easy for developers to fix acceptance tests
Developers share a common pool of test servers to reproduce failures
Developers create on demand test environments
Developers use their own dev environments to reproduce failures
@nicolefv
@jezhumble
Which of these measure effective test
practices?
Developers primarily create & maintain acceptance tests
QA primarily create & maintain acceptance tests
Primarily created & maintained by outsourced party
When automated tests pass, I’m confident the software is releasable
Test failures are likely to indicate a real defect
It’s easy for developers to fix acceptance tests
Developers share a common pool of test servers to reproduce failures
Developers create on demand test environments
Developers use their own dev environments to reproduce failures
@nicolefv
@jezhumble
Key Finding: Unplanned work
High performers spend 20% more
time on new work than low
performers, and 22% less time on
unplanned work and rework.
@nicolefv
@jezhumble
Surprises with Culture
• We wanted to add additional measures of
culture
• Google study
• Identity
• Retain Westrum culture
@nicolefv
@jezhumble
Identity & Google items
• I am glad I chose to work for this
organization rather than another
company.
• I talk of this organization to my friends as a
great company to work for.
• I am willing to put in a great deal of effort
beyond what is normally expected to help
my organization to be successful.
• I find that my values and my organization's
values are very similar.
• In general, the people employed by my
organization are working toward the same
goal.
• I feel that my organization cares about me.
Adapted from adapted from Atreyi Kankanhalli, Bernard C.Y. Tan, and
Kwok-Kee Wei (2005), “Contributing Knowledge to Electronic Knowledge
Repositories: An Empirical Investigation,“ MIS Quarterly, 29, 113-143.
@nicolefv
@jezhumble
Identity & Google items
• I am glad I chose to work for this organization
rather than another company.
• I talk of this organization to my friends as a
great company to work for.
• I am willing to put in a great deal of effort
beyond what is normally expected to help my
organization to be successful.
• I find that my values and my organization's
values are very similar.
• In general, the people employed by my
organization are workingtoward the same
goal.
• I feel that my organization caresabout me.
Adapted from adapted from Atreyi Kankanhalli, Bernard C.Y.
Tan, and Kwok-Kee Wei (2005), “Contributing Knowledge to
Electronic Knowledge Repositories: An Empirical Investigation,“
MIS Quarterly, 29, 113-143.
Westrum items
@nicolefv
@jezhumble
Now for management stuff
We all know managing WIP is important, right?
@nicolefv
@jezhumble
Now for management stuff
We all know managing WIP is important, right?
Correlation between WIP and ITPerf is negligible
@nicolefv
@jezhumble
Now for management stuff
We all know managing WIP is important, right?
Correlation between WIP and ITPerf is negligible
What’s going on?
@nicolefv
@jezhumble
Lean management SEM
@nicolefv
@jezhumble
Also lean product management
@nicolefv
@jezhumble
Conclusions
• Even if you think it’s obvious, TEST WITH
DATA.
• (if the results don’t surprise you, you’re doing it wrong)
• (if you don’t also confirm some things you expected,
you’re doing it wrong)
• We CAN have it all, or at least throughput
AND stability.
• IT matters (but you have to do it right)
• DevOps culture & practices have a
measurable impact on IT & org perf
@nicolefv
@jezhumble
Sign	up	for	our	ROI	whitepaper	&	
get	peer-reviewed	research
devops-research.com
For more science-ing…
@nicolefv
@jezhumble
Thank you
devops-research.com

More Related Content

What's hot

DevOps: What's Buried in the Fine Print
DevOps: What's Buried in the Fine PrintDevOps: What's Buried in the Fine Print
DevOps: What's Buried in the Fine Print
Jeffery Smith
 
Vmware2021 why even devop nicolefv
Vmware2021 why even devop nicolefvVmware2021 why even devop nicolefv
Vmware2021 why even devop nicolefv
Nicole Forsgren
 
The Data on DevOps: Making the Case for Awesome
The Data on DevOps: Making the Case for AwesomeThe Data on DevOps: Making the Case for Awesome
The Data on DevOps: Making the Case for Awesome
Nicole Forsgren
 
Soaring in the Clouds - Don't be dragged down by ITIL bloat!
Soaring in the Clouds - Don't be dragged down by ITIL bloat! Soaring in the Clouds - Don't be dragged down by ITIL bloat!
Soaring in the Clouds - Don't be dragged down by ITIL bloat!
Navvia
 
What we learned from three years sciencing the crap out of devops
What we learned from three years sciencing the crap out of devopsWhat we learned from three years sciencing the crap out of devops
What we learned from three years sciencing the crap out of devops
Nicole Forsgren
 
The Data Behind DevOps: Becoming a High Performer
The Data Behind DevOps: Becoming a High PerformerThe Data Behind DevOps: Becoming a High Performer
The Data Behind DevOps: Becoming a High Performer
Nicole Forsgren
 
Successful writing at work copyright 2017 cengage learn
Successful writing at work copyright 2017 cengage learnSuccessful writing at work copyright 2017 cengage learn
Successful writing at work copyright 2017 cengage learn
ssusere73ce3
 
Continuous Delivery + DevOps = Awesome
Continuous Delivery + DevOps = AwesomeContinuous Delivery + DevOps = Awesome
Continuous Delivery + DevOps = Awesome
Nicole Forsgren
 
Why Even DevOp?
Why Even DevOp?Why Even DevOp?
Why Even DevOp?
VMware Tanzu
 
DOES 2016 Sciencing the Crap Out of DevOps
DOES 2016 Sciencing the Crap Out of DevOpsDOES 2016 Sciencing the Crap Out of DevOps
DOES 2016 Sciencing the Crap Out of DevOps
Nicole Forsgren
 
Sciencing the Crap Out of DevOps
Sciencing the Crap Out of DevOpsSciencing the Crap Out of DevOps
Sciencing the Crap Out of DevOps
Nicole Forsgren
 
Lean Product Management User-Centered App Design
Lean Product Management User-Centered App DesignLean Product Management User-Centered App Design
Lean Product Management User-Centered App Design
VMware Tanzu
 
Developer Nirvana
Developer NirvanaDeveloper Nirvana
Developer Nirvana
VMware Tanzu
 
Continuous Delivery: Making DevOps Awesome
Continuous Delivery: Making DevOps AwesomeContinuous Delivery: Making DevOps Awesome
Continuous Delivery: Making DevOps Awesome
Nicole Forsgren
 
DevOps Kaizen: Practical Steps to Start & Sustain a Transformation
DevOps Kaizen: Practical Steps to Start & Sustain a TransformationDevOps Kaizen: Practical Steps to Start & Sustain a Transformation
DevOps Kaizen: Practical Steps to Start & Sustain a Transformation
dev2ops
 
DevOps: The Key to IT Performance
DevOps: The Key to IT PerformanceDevOps: The Key to IT Performance
DevOps: The Key to IT Performance
Nicole Forsgren
 
Nf final chef-lisa-metrics-2015-ss
Nf final chef-lisa-metrics-2015-ssNf final chef-lisa-metrics-2015-ss
Nf final chef-lisa-metrics-2015-ss
Nicole Forsgren
 
The Key to High Performance - What the Data Says
The Key to High Performance - What the Data SaysThe Key to High Performance - What the Data Says
The Key to High Performance - What the Data Says
Nicole Forsgren
 
Secrets and surprises of high performance: What the data says
Secrets and surprises of high performance: What the data saysSecrets and surprises of high performance: What the data says
Secrets and surprises of high performance: What the data says
Nicole Forsgren
 
The History of DevOps (and what you need to do about it)
The History of DevOps (and what you need to do about it)The History of DevOps (and what you need to do about it)
The History of DevOps (and what you need to do about it)
dev2ops
 

What's hot (20)

DevOps: What's Buried in the Fine Print
DevOps: What's Buried in the Fine PrintDevOps: What's Buried in the Fine Print
DevOps: What's Buried in the Fine Print
 
Vmware2021 why even devop nicolefv
Vmware2021 why even devop nicolefvVmware2021 why even devop nicolefv
Vmware2021 why even devop nicolefv
 
The Data on DevOps: Making the Case for Awesome
The Data on DevOps: Making the Case for AwesomeThe Data on DevOps: Making the Case for Awesome
The Data on DevOps: Making the Case for Awesome
 
Soaring in the Clouds - Don't be dragged down by ITIL bloat!
Soaring in the Clouds - Don't be dragged down by ITIL bloat! Soaring in the Clouds - Don't be dragged down by ITIL bloat!
Soaring in the Clouds - Don't be dragged down by ITIL bloat!
 
What we learned from three years sciencing the crap out of devops
What we learned from three years sciencing the crap out of devopsWhat we learned from three years sciencing the crap out of devops
What we learned from three years sciencing the crap out of devops
 
The Data Behind DevOps: Becoming a High Performer
The Data Behind DevOps: Becoming a High PerformerThe Data Behind DevOps: Becoming a High Performer
The Data Behind DevOps: Becoming a High Performer
 
Successful writing at work copyright 2017 cengage learn
Successful writing at work copyright 2017 cengage learnSuccessful writing at work copyright 2017 cengage learn
Successful writing at work copyright 2017 cengage learn
 
Continuous Delivery + DevOps = Awesome
Continuous Delivery + DevOps = AwesomeContinuous Delivery + DevOps = Awesome
Continuous Delivery + DevOps = Awesome
 
Why Even DevOp?
Why Even DevOp?Why Even DevOp?
Why Even DevOp?
 
DOES 2016 Sciencing the Crap Out of DevOps
DOES 2016 Sciencing the Crap Out of DevOpsDOES 2016 Sciencing the Crap Out of DevOps
DOES 2016 Sciencing the Crap Out of DevOps
 
Sciencing the Crap Out of DevOps
Sciencing the Crap Out of DevOpsSciencing the Crap Out of DevOps
Sciencing the Crap Out of DevOps
 
Lean Product Management User-Centered App Design
Lean Product Management User-Centered App DesignLean Product Management User-Centered App Design
Lean Product Management User-Centered App Design
 
Developer Nirvana
Developer NirvanaDeveloper Nirvana
Developer Nirvana
 
Continuous Delivery: Making DevOps Awesome
Continuous Delivery: Making DevOps AwesomeContinuous Delivery: Making DevOps Awesome
Continuous Delivery: Making DevOps Awesome
 
DevOps Kaizen: Practical Steps to Start & Sustain a Transformation
DevOps Kaizen: Practical Steps to Start & Sustain a TransformationDevOps Kaizen: Practical Steps to Start & Sustain a Transformation
DevOps Kaizen: Practical Steps to Start & Sustain a Transformation
 
DevOps: The Key to IT Performance
DevOps: The Key to IT PerformanceDevOps: The Key to IT Performance
DevOps: The Key to IT Performance
 
Nf final chef-lisa-metrics-2015-ss
Nf final chef-lisa-metrics-2015-ssNf final chef-lisa-metrics-2015-ss
Nf final chef-lisa-metrics-2015-ss
 
The Key to High Performance - What the Data Says
The Key to High Performance - What the Data SaysThe Key to High Performance - What the Data Says
The Key to High Performance - What the Data Says
 
Secrets and surprises of high performance: What the data says
Secrets and surprises of high performance: What the data saysSecrets and surprises of high performance: What the data says
Secrets and surprises of high performance: What the data says
 
The History of DevOps (and what you need to do about it)
The History of DevOps (and what you need to do about it)The History of DevOps (and what you need to do about it)
The History of DevOps (and what you need to do about it)
 

Viewers also liked

DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...
DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...
DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...
Gene Kim
 
DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits
DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits
DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits
Gene Kim
 
DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...
DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...
DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...
Gene Kim
 
DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev
DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows DevDOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev
DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev
Gene Kim
 
DOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic Required
DOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic RequiredDOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic Required
DOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic Required
Gene Kim
 
DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams
DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams
DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams
Gene Kim
 
DOES SFO 2016 - David Habershon - Ministry of Social Development New Zealand
DOES SFO 2016 - David Habershon - Ministry of Social Development New ZealandDOES SFO 2016 - David Habershon - Ministry of Social Development New Zealand
DOES SFO 2016 - David Habershon - Ministry of Social Development New Zealand
Gene Kim
 
DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...
DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...
DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...
Gene Kim
 
DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...
DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...
DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...
Gene Kim
 
DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...
DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...
DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...
Gene Kim
 
DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...
DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...
DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...
Gene Kim
 
DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...
DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...
DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...
Gene Kim
 
DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...
DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...
DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...
Gene Kim
 
DOES SFO 2016 - Avan Mathur - Planning for Huge Scale
DOES SFO 2016 - Avan Mathur - Planning for Huge ScaleDOES SFO 2016 - Avan Mathur - Planning for Huge Scale
DOES SFO 2016 - Avan Mathur - Planning for Huge Scale
Gene Kim
 
DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...
DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...
DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...
Gene Kim
 
DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...
DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...
DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...
Gene Kim
 
DOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human Spirit
DOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human SpiritDOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human Spirit
DOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human Spirit
Gene Kim
 
DOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at Scale
DOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at ScaleDOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at Scale
DOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at Scale
Gene Kim
 
DOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the Enterprise
DOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the EnterpriseDOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the Enterprise
DOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the Enterprise
Gene Kim
 
DOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the Deal
DOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the DealDOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the Deal
DOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the Deal
Gene Kim
 

Viewers also liked (20)

DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...
DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...
DOES16 San Francisco - Damon Edwards - The Talent You Need is Already Inside ...
 
DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits
DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits
DOES SFO 2016 - Matthew Barr - Enterprise Git - the hard bits
 
DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...
DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...
DOES16 San Francisco - Charles Betz - Influencing Higher Education to Create ...
 
DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev
DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows DevDOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev
DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev
 
DOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic Required
DOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic RequiredDOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic Required
DOES SFO 2016 San Francisco - Julia Wester - Predictability: No Magic Required
 
DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams
DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams
DOES SFO 2016 - Paula Thrasher & Kevin Stanley - Building Brilliant Teams
 
DOES SFO 2016 - David Habershon - Ministry of Social Development New Zealand
DOES SFO 2016 - David Habershon - Ministry of Social Development New ZealandDOES SFO 2016 - David Habershon - Ministry of Social Development New Zealand
DOES SFO 2016 - David Habershon - Ministry of Social Development New Zealand
 
DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...
DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...
DOES16 San Francisco - Opal Perry - Technology Transformation: How Team Value...
 
DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...
DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...
DOES16 San Francisco - Carmen DeArdo, Cindy Payne, & Jim Grafmeyer - Episode ...
 
DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...
DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...
DOES16 San Francisco - Dominica DeGrandis - Time Theft: How Hidden and Unplan...
 
DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...
DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...
DOES16 London - Gareth Rushgrove - Communication Between Tribes: A Story of S...
 
DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...
DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...
DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...
 
DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...
DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...
DOES SFO 2016 - Aimee Bechtle - Utilizing Distributed Dojos to Transform a Wo...
 
DOES SFO 2016 - Avan Mathur - Planning for Huge Scale
DOES SFO 2016 - Avan Mathur - Planning for Huge ScaleDOES SFO 2016 - Avan Mathur - Planning for Huge Scale
DOES SFO 2016 - Avan Mathur - Planning for Huge Scale
 
DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...
DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...
DOES16 San Francisco - David Blank-Edelman - Lessons Learned from a Parallel ...
 
DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...
DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...
DOES16 San Francisco - Jan Schilt - DevOps is Not Going to Work…Unless! How T...
 
DOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human Spirit
DOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human SpiritDOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human Spirit
DOES SFO 2016 - Courtney Kissler - Inspire and Nurture the Human Spirit
 
DOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at Scale
DOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at ScaleDOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at Scale
DOES SFO 2016 - Greg Maxey and Laurent Rochette - DSL at Scale
 
DOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the Enterprise
DOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the EnterpriseDOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the Enterprise
DOES SFO 2016 - Daniel Perez - Doubling Down on ChatOps in the Enterprise
 
DOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the Deal
DOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the DealDOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the Deal
DOES SFO 2016 - Andy Cooper & Brandon Holcomb - When IT Closes the Deal
 

Similar to DOES16 San Francisco - Nicole Forsgren & Jez Humble - The Latest: What We Learned from the 2016 State of DevOps Report

The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...
The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...
The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...
Codemotion
 
What We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOpsWhat We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOps
SeniorStoryteller
 
What We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOpsWhat We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOps
SeniorStoryteller
 
The caseforawesome
The caseforawesomeThe caseforawesome
The caseforawesome
Chef
 
Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...
Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...
Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...
Matt Stubbs
 
Modeling and Measuring DevOps Culture
Modeling and Measuring DevOps CultureModeling and Measuring DevOps Culture
Modeling and Measuring DevOps Culture
Leland Newsom CSP-SM, SPC5, SDP
 
How do we fix testing
How do we fix testingHow do we fix testing
How do we fix testing
Peter Varhol
 
Adopting a Continual Improvement Mindset for IT
Adopting a Continual Improvement Mindset for ITAdopting a Continual Improvement Mindset for IT
Adopting a Continual Improvement Mindset for IT
Josh Atwell
 
School customer service presentation
School customer service presentationSchool customer service presentation
School customer service presentation
steve muzzy
 
UX London Collaborative Research Workshop
UX London Collaborative Research WorkshopUX London Collaborative Research Workshop
UX London Collaborative Research Workshop
Erika Hall
 
1.11 Data and Performance Simplified
1.11 Data and Performance Simplified1.11 Data and Performance Simplified
1.11 Data and Performance Simplified
National Alliance to End Homelessness
 
In-Depth Interviews: Techniques and Best Practices
In-Depth Interviews: Techniques and Best PracticesIn-Depth Interviews: Techniques and Best Practices
In-Depth Interviews: Techniques and Best Practices
Alexandra Howson MA, PhD, CHCP
 
Collaborative Research | uxlx 2014
Collaborative Research | uxlx 2014Collaborative Research | uxlx 2014
Collaborative Research | uxlx 2014
Erika Hall
 
Max Shron, Thinking with Data at the NYC Data Science Meetup
Max Shron, Thinking with Data at the NYC Data Science MeetupMax Shron, Thinking with Data at the NYC Data Science Meetup
Max Shron, Thinking with Data at the NYC Data Science Meetup
mortardata
 
Unit4 for st.pdf
Unit4 for st.pdfUnit4 for st.pdf
Unit4 for st.pdf
Poonkodi Jayakumar
 
Exploring Users' Values, Motivations and Emotions
Exploring Users' Values, Motivations and EmotionsExploring Users' Values, Motivations and Emotions
Exploring Users' Values, Motivations and Emotions
Northern User Experience
 
Navigating your career in the Shifting Winds of Academia
Navigating your career in the Shifting Winds of AcademiaNavigating your career in the Shifting Winds of Academia
Navigating your career in the Shifting Winds of Academia
Terri Friel
 
Synbio london 040214
Synbio london 040214Synbio london 040214
Synbio london 040214
Stanford University
 
Rebooting the Team - Surge 2013
Rebooting the Team - Surge 2013Rebooting the Team - Surge 2013
Rebooting the Team - Surge 2013
Fran Fabrizio
 
2.6 Expert Forum: Data and Performance Simplified
2.6 Expert Forum: Data and Performance Simplified2.6 Expert Forum: Data and Performance Simplified
2.6 Expert Forum: Data and Performance Simplified
National Alliance to End Homelessness
 

Similar to DOES16 San Francisco - Nicole Forsgren & Jez Humble - The Latest: What We Learned from the 2016 State of DevOps Report (20)

The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...
The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...
The Secrets of High Performance: Science Edition - Nicole Forsgren - Codemoti...
 
What We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOpsWhat We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOps
 
What We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOpsWhat We Learned from Three Years of Sciencing the Crap Out of DevOps
What We Learned from Three Years of Sciencing the Crap Out of DevOps
 
The caseforawesome
The caseforawesomeThe caseforawesome
The caseforawesome
 
Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...
Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...
Big Data LDN 2017: Preserving The Key Principles Of Academic Research In A Bu...
 
Modeling and Measuring DevOps Culture
Modeling and Measuring DevOps CultureModeling and Measuring DevOps Culture
Modeling and Measuring DevOps Culture
 
How do we fix testing
How do we fix testingHow do we fix testing
How do we fix testing
 
Adopting a Continual Improvement Mindset for IT
Adopting a Continual Improvement Mindset for ITAdopting a Continual Improvement Mindset for IT
Adopting a Continual Improvement Mindset for IT
 
School customer service presentation
School customer service presentationSchool customer service presentation
School customer service presentation
 
UX London Collaborative Research Workshop
UX London Collaborative Research WorkshopUX London Collaborative Research Workshop
UX London Collaborative Research Workshop
 
1.11 Data and Performance Simplified
1.11 Data and Performance Simplified1.11 Data and Performance Simplified
1.11 Data and Performance Simplified
 
In-Depth Interviews: Techniques and Best Practices
In-Depth Interviews: Techniques and Best PracticesIn-Depth Interviews: Techniques and Best Practices
In-Depth Interviews: Techniques and Best Practices
 
Collaborative Research | uxlx 2014
Collaborative Research | uxlx 2014Collaborative Research | uxlx 2014
Collaborative Research | uxlx 2014
 
Max Shron, Thinking with Data at the NYC Data Science Meetup
Max Shron, Thinking with Data at the NYC Data Science MeetupMax Shron, Thinking with Data at the NYC Data Science Meetup
Max Shron, Thinking with Data at the NYC Data Science Meetup
 
Unit4 for st.pdf
Unit4 for st.pdfUnit4 for st.pdf
Unit4 for st.pdf
 
Exploring Users' Values, Motivations and Emotions
Exploring Users' Values, Motivations and EmotionsExploring Users' Values, Motivations and Emotions
Exploring Users' Values, Motivations and Emotions
 
Navigating your career in the Shifting Winds of Academia
Navigating your career in the Shifting Winds of AcademiaNavigating your career in the Shifting Winds of Academia
Navigating your career in the Shifting Winds of Academia
 
Synbio london 040214
Synbio london 040214Synbio london 040214
Synbio london 040214
 
Rebooting the Team - Surge 2013
Rebooting the Team - Surge 2013Rebooting the Team - Surge 2013
Rebooting the Team - Surge 2013
 
2.6 Expert Forum: Data and Performance Simplified
2.6 Expert Forum: Data and Performance Simplified2.6 Expert Forum: Data and Performance Simplified
2.6 Expert Forum: Data and Performance Simplified
 

More from Gene Kim

DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...
DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...
DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...
Gene Kim
 
DOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at Verizon
DOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at VerizonDOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at Verizon
DOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at Verizon
Gene Kim
 
DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOps
DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOpsDOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOps
DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOps
Gene Kim
 
DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...
DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...
DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...
Gene Kim
 
DOES SFO 2016 - Greg Padak - Default to Open
DOES SFO 2016 - Greg Padak - Default to OpenDOES SFO 2016 - Greg Padak - Default to Open
DOES SFO 2016 - Greg Padak - Default to Open
Gene Kim
 
DOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, Initiative
DOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, InitiativeDOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, Initiative
DOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, Initiative
Gene Kim
 
DOES SFO 2016 - Alexa Alley - Value Stream Mapping
DOES SFO 2016 - Alexa Alley - Value Stream MappingDOES SFO 2016 - Alexa Alley - Value Stream Mapping
DOES SFO 2016 - Alexa Alley - Value Stream Mapping
Gene Kim
 
DOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding Edge
DOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding EdgeDOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding Edge
DOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding Edge
Gene Kim
 
DOES SFO 2016 - Topo Pal - DevOps at Capital One
DOES SFO 2016 - Topo Pal - DevOps at Capital OneDOES SFO 2016 - Topo Pal - DevOps at Capital One
DOES SFO 2016 - Topo Pal - DevOps at Capital One
Gene Kim
 
DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?
DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?
DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?
Gene Kim
 
DOES SFO 2016 - Chris Fulton - CD for DBs
DOES SFO 2016 - Chris Fulton - CD for DBsDOES SFO 2016 - Chris Fulton - CD for DBs
DOES SFO 2016 - Chris Fulton - CD for DBs
Gene Kim
 
DOES SFO 2016 - Marc Priolo - Are we there yet?
DOES SFO 2016 - Marc Priolo - Are we there yet? DOES SFO 2016 - Marc Priolo - Are we there yet?
DOES SFO 2016 - Marc Priolo - Are we there yet?
Gene Kim
 
DOES SFO 2016 - Steve Brodie - The Future of DevOps in the Enterprise
DOES SFO 2016 - Steve Brodie - The Future of DevOps in the EnterpriseDOES SFO 2016 - Steve Brodie - The Future of DevOps in the Enterprise
DOES SFO 2016 - Steve Brodie - The Future of DevOps in the Enterprise
Gene Kim
 
DOES SFO 2016 - Ray Krueger - Speed as a Prime Directive
DOES SFO 2016 - Ray Krueger - Speed as a Prime DirectiveDOES SFO 2016 - Ray Krueger - Speed as a Prime Directive
DOES SFO 2016 - Ray Krueger - Speed as a Prime Directive
Gene Kim
 
DOES SFO 2016 - Steve Mayner - Transformational Leadership
DOES SFO 2016 - Steve Mayner - Transformational LeadershipDOES SFO 2016 - Steve Mayner - Transformational Leadership
DOES SFO 2016 - Steve Mayner - Transformational Leadership
Gene Kim
 
DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...
DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...
DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...
Gene Kim
 
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
Gene Kim
 

More from Gene Kim (17)

DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...
DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...
DOES SFO 2016 - Kaimar Karu - ITIL. You keep using that word. I don't think i...
 
DOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at Verizon
DOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at VerizonDOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at Verizon
DOES SFO 2016 - Ross Clanton and Chivas Nambiar - DevOps at Verizon
 
DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOps
DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOpsDOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOps
DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOps
 
DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...
DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...
DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...
 
DOES SFO 2016 - Greg Padak - Default to Open
DOES SFO 2016 - Greg Padak - Default to OpenDOES SFO 2016 - Greg Padak - Default to Open
DOES SFO 2016 - Greg Padak - Default to Open
 
DOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, Initiative
DOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, InitiativeDOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, Initiative
DOES SFO 2016 - Michael Nygard - Tempo, Maneuverability, Initiative
 
DOES SFO 2016 - Alexa Alley - Value Stream Mapping
DOES SFO 2016 - Alexa Alley - Value Stream MappingDOES SFO 2016 - Alexa Alley - Value Stream Mapping
DOES SFO 2016 - Alexa Alley - Value Stream Mapping
 
DOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding Edge
DOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding EdgeDOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding Edge
DOES SFO 2016 - Mark Imbriaco - Lessons From the Bleeding Edge
 
DOES SFO 2016 - Topo Pal - DevOps at Capital One
DOES SFO 2016 - Topo Pal - DevOps at Capital OneDOES SFO 2016 - Topo Pal - DevOps at Capital One
DOES SFO 2016 - Topo Pal - DevOps at Capital One
 
DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?
DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?
DOES SFO 2016 - Cornelia Davis - DevOps: Who Does What?
 
DOES SFO 2016 - Chris Fulton - CD for DBs
DOES SFO 2016 - Chris Fulton - CD for DBsDOES SFO 2016 - Chris Fulton - CD for DBs
DOES SFO 2016 - Chris Fulton - CD for DBs
 
DOES SFO 2016 - Marc Priolo - Are we there yet?
DOES SFO 2016 - Marc Priolo - Are we there yet? DOES SFO 2016 - Marc Priolo - Are we there yet?
DOES SFO 2016 - Marc Priolo - Are we there yet?
 
DOES SFO 2016 - Steve Brodie - The Future of DevOps in the Enterprise
DOES SFO 2016 - Steve Brodie - The Future of DevOps in the EnterpriseDOES SFO 2016 - Steve Brodie - The Future of DevOps in the Enterprise
DOES SFO 2016 - Steve Brodie - The Future of DevOps in the Enterprise
 
DOES SFO 2016 - Ray Krueger - Speed as a Prime Directive
DOES SFO 2016 - Ray Krueger - Speed as a Prime DirectiveDOES SFO 2016 - Ray Krueger - Speed as a Prime Directive
DOES SFO 2016 - Ray Krueger - Speed as a Prime Directive
 
DOES SFO 2016 - Steve Mayner - Transformational Leadership
DOES SFO 2016 - Steve Mayner - Transformational LeadershipDOES SFO 2016 - Steve Mayner - Transformational Leadership
DOES SFO 2016 - Steve Mayner - Transformational Leadership
 
DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...
DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...
DOES SFO 2016 - Sam Guckenheimer & Ed Blankenship "Moving to One Engineering ...
 
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
 

Recently uploaded

Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
FilipTomaszewski5
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
DianaGray10
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
Sease
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
BibashShahi
 

Recently uploaded (20)

Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
 

DOES16 San Francisco - Nicole Forsgren & Jez Humble - The Latest: What We Learned from the 2016 State of DevOps Report

  • 1. @nicolefv @jezhumble What We Learned from Four Years of Sciencing the Crap Out of DevOps Nicole Forsgren, PhD Jez Humble
  • 2. @nicolefv @jezhumble Outline How to make your data suck less • Writing good survey questions • Making sure the survey questions are good - with SCIENCE • (These methods apply to your system and log data) What we found… that we did (AND didn’t) expect Things about Continuous Delivery Things about Management
  • 3. @nicolefv @jezhumble Not all data is created equal Who here thinks surveys are sh*t?
  • 4. @nicolefv @jezhumble Not all data is created equal Who here thinks surveys are sh*t? Who here LOVES the data from their log files?
  • 5. @nicolefv @jezhumble What is a Latent Construct?
  • 6. @nicolefv @jezhumble We use PSYCHOMETRICS to make our survey data good* *or give us a reasonable assurance that it’s telling us what we think it’s telling us (& some of this can also apply to your log data)
  • 7. @nicolefv @jezhumble Psychometrics includes: Construct creation (manual) • When possible: use previously validated constructs • Based on definitions and theory, carefully and precisely worded, card sorting task, pilot tested Construct evaluation (statistics) • Establishing Validity: discriminant and convergent • Establishing Reliability
  • 8. @nicolefv @jezhumble Psychometrics Writing Example: Culture • Does it matter to our study? • More than just intuition? • What KIND of culture? • National identity and norms • Adaptive culture • Value learning (2014 study) • Value information flow and trust (2014 and 2015 studies -- Westrum culture)
  • 9. @nicolefv @jezhumble Psychometrics Writing Example: Culture • Does it matter to our study? • More than just intuition? • What KIND of culture? • National identity and norms • Adaptive culture • Value learning (2014 study) • Value information flow and trust (2014 and 2015 studies -- Westrum culture)
  • 10. @nicolefv @jezhumble Westrum typology Pathological Power-oriented Bureaucratic Rule-oriented Generative Performance-oriented Low cooperation Modest cooperation High cooperation Messengers shot Messengers neglected Messengers trained Responsibilities shirked Narrow responsibilities Risks are shared Bridging discouraged Bridging tolerated Bridging encouraged Failure leads to scapegoating Failure leads to justice Failure leads to inquiry Novelty crushed Novelty leads to problems Novelty implemented Try writing items yourself! Use strong statements with clear language. Westrum, R. (2004). A typology of organisational cultures. Quality and safety in health care, 13(suppl 2), ii22-ii27.
  • 11. @nicolefv @jezhumble Westrum Culture Items • On my team, information is actively sought. • On my team, failures are learning opportunities, and messengers of them are not punished. • On my team, responsibilities are shared. • On my team, cross-functional collaboration is encouraged and rewarded. • On my team, failure causes inquiry. • On my team, new ideas are welcomed. Found to be valid & reliable Predictive of IT Performance & Organizational Performance
  • 12. @nicolefv @jezhumble Psychometrics Analysis Example: Notification of Failure Original in 2014, but there was a surprise. Can you spot it? At my organization… • We are primarily notified of failures by reports from customers. • We are primarily notified of failures by the NOC. • We get failure alerts from logging and monitoring systems. • We monitor system health based on threshold warnings (ex. CPU exceeds 100%). • We monitor system health based on rate-of-change warnings (ex. CPU usage has increased by 25% over the last 10 minutes).
  • 13. @nicolefv @jezhumble At my organization… • We are primarily notified of failures by reports from customers. • We are primarily notified of failures by the NOC. • We get failure alerts from logging and monitoring systems. • We monitor system health based on threshold warnings (ex. CPU exceeds 100%). • We monitor system health based on rate-of-change warnings (ex. CPU usage has increased by 25% over the last 10 minutes). Notification from NEAR Notification from FAR Psychometrics Analysis Example: Notification of Failure
  • 14. @nicolefv @jezhumble More data tests! Plus, we test to make sure the survey doesn’t have other problems. • Common method variance (CMV) (aka CMB for Bias) • Early vs. late responders • Survey drop-off rates and bias
  • 15. @nicolefv @jezhumble Okay NOW we can look at the data and how it relates to each other
  • 16. @nicolefv @jezhumble A note about analysis methods One of three conditions must be met: 1. Longitudinal (no, this is cross-sectional) 2. Randomized, experimental design (no, this is a non-experimental) 3. Theory-based design When this condition was not met, only correlations were tested and reported
  • 17. @nicolefv @jezhumble KEY FINDING: IT Performance and its behavior A combination of throughput and stability • lead time for changes • release frequency • time to restore service • change fail rate Forsgren, N., J. Humble (2016). "DevOps: Profiles in ITSM Performance and Contributing Factors." In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV.
  • 18. @nicolefv @jezhumble KEY FINDING: IT performance matters! “Firms with high-performing IT organizations were twice as likely to exceed their profitability, market share and productivity goals.” IT Performance is predictive of organizational performance. http://bit.ly/2014-devops-report/ http://bit.ly/2015-devops-report/ Forsgren, N., J. Humble (2016). “The Role of Continuous Delivery in IT and Organizational Performance.” In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV.
  • 22. @nicolefv @jezhumble Which of these measure effective test practices? Developers primarily create & maintain acceptance tests QA primarily create & maintain acceptance tests Primarily created & maintained by outsourced party When automated tests pass, I’m confident the software is releasable Test failures are likely to indicate a real defect It’s easy for developers to fix acceptance tests Developers share a common pool of test servers to reproduce failures Developers create on demand test environments Developers use their own dev environments to reproduce failures
  • 23. @nicolefv @jezhumble Which of these measure effective test practices? Developers primarily create & maintain acceptance tests QA primarily create & maintain acceptance tests Primarily created & maintained by outsourced party When automated tests pass, I’m confident the software is releasable Test failures are likely to indicate a real defect It’s easy for developers to fix acceptance tests Developers share a common pool of test servers to reproduce failures Developers create on demand test environments Developers use their own dev environments to reproduce failures
  • 24. @nicolefv @jezhumble Key Finding: Unplanned work High performers spend 20% more time on new work than low performers, and 22% less time on unplanned work and rework.
  • 25. @nicolefv @jezhumble Surprises with Culture • We wanted to add additional measures of culture • Google study • Identity • Retain Westrum culture
  • 26. @nicolefv @jezhumble Identity & Google items • I am glad I chose to work for this organization rather than another company. • I talk of this organization to my friends as a great company to work for. • I am willing to put in a great deal of effort beyond what is normally expected to help my organization to be successful. • I find that my values and my organization's values are very similar. • In general, the people employed by my organization are working toward the same goal. • I feel that my organization cares about me. Adapted from adapted from Atreyi Kankanhalli, Bernard C.Y. Tan, and Kwok-Kee Wei (2005), “Contributing Knowledge to Electronic Knowledge Repositories: An Empirical Investigation,“ MIS Quarterly, 29, 113-143.
  • 27. @nicolefv @jezhumble Identity & Google items • I am glad I chose to work for this organization rather than another company. • I talk of this organization to my friends as a great company to work for. • I am willing to put in a great deal of effort beyond what is normally expected to help my organization to be successful. • I find that my values and my organization's values are very similar. • In general, the people employed by my organization are workingtoward the same goal. • I feel that my organization caresabout me. Adapted from adapted from Atreyi Kankanhalli, Bernard C.Y. Tan, and Kwok-Kee Wei (2005), “Contributing Knowledge to Electronic Knowledge Repositories: An Empirical Investigation,“ MIS Quarterly, 29, 113-143. Westrum items
  • 28. @nicolefv @jezhumble Now for management stuff We all know managing WIP is important, right?
  • 29. @nicolefv @jezhumble Now for management stuff We all know managing WIP is important, right? Correlation between WIP and ITPerf is negligible
  • 30. @nicolefv @jezhumble Now for management stuff We all know managing WIP is important, right? Correlation between WIP and ITPerf is negligible What’s going on?
  • 33. @nicolefv @jezhumble Conclusions • Even if you think it’s obvious, TEST WITH DATA. • (if the results don’t surprise you, you’re doing it wrong) • (if you don’t also confirm some things you expected, you’re doing it wrong) • We CAN have it all, or at least throughput AND stability. • IT matters (but you have to do it right) • DevOps culture & practices have a measurable impact on IT & org perf