The document discusses selecting metrics for DevOps processes. It outlines key metrics to measure for different stages including development, testing, deployment, release and operations. Examples of metrics include deployment frequency, lead time, change fail rate, and mean time to recovery. The document provides recommendations on where to start measuring for each stage and resources for further information on DevOps metrics.

1. © Electric Cloud | electric-cloud.com
From Measurement to Insight
Putting DevOps Metrics To Work

2. © Electric Cloud | electric-cloud.com
Speakers
Nicole Forsgren
CEO, DORA
@nicolefv
Anders Wallgren
CTO, Electric Cloud
@anders_wallgren

3. © Electric Cloud | electric-cloud.com
Agenda
Measuring DevOps – The Key Metrics That Matter
• Why Measure?
• Understanding the Impact
• Choosing The Right Metrics

4. © Electric Cloud | electric-cloud.com
Why Measure?
The Art and Science of Continuous Improvement

5. © Electric Cloud | electric-cloud.com
Every Business is a
Software Business

6. © Electric Cloud | electric-cloud.com
Today’s software delivery challenges
• High cost (risk, time) per product release
• Manually operated, non-integrated tool chains
• No repeatability, predictability
• No traceability, auditability
• Inefficient infrastructure, low utilization
• Non-standard practices
• Lack of shared visibility across Dev-QA-Ops
@samueldfell

7. © Electric Cloud | electric-cloud.com
If you're having
trouble already,
lack of visibility
won't help.
Anders Wallgren

8. © Electric Cloud | electric-cloud.com
How does visibility help? Because SCIENCE!
OBSERVATION is fundamental to the scientific method.
REPEAT!

9. © Electric Cloud | electric-cloud.com
What does this
have to do with
DevOps?
DevOps uses the
Scientific Method
to enable software
innovation!

10. © Electric Cloud | electric-cloud.com
Measuring the Impact
DORA, State Of DevOps Report

11. © Electric Cloud | electric-cloud.com
Why do I know any of this?
@nicolefv

12. © Electric Cloud | electric-cloud.com
DevOps Direction?
DevOps: the use of technology,
process, and culture to deliver
value to an organization.
Is there “one metric that
matters”?

13. © Electric Cloud | electric-cloud.com
Our direction in DevOps
IT Performance: developing and delivering
software with both speed and stability
- Deploy frequency
- Lead Time for Changes
- Mean Time to Recover (MTTR)
- Change fail rate

14. © Electric Cloud | electric-cloud.com
More frequent
Code deployments
46x
That’s the difference between multiple
times per day and once a week or less.
Faster lead time from
commit to deploy
440x
That’s the difference between less than
an hour and more than a week.
@nicolefv

15. © Electric Cloud | electric-cloud.com
faster mean time to
recover from downtime
96x
That means high performers recover in
less than an hour instead of several days.
as likely that changes will
fail
1/5x
That means high performers’ changes fail 7.5%
of the time instead of 38.5%.
@nicolefv

16. © Electric Cloud | electric-cloud.com
Choosing the Right Metrics
DevOps Metrics to Track

17. © Electric Cloud | electric-cloud.com
One measurement to rule them all?

18. © Electric Cloud | electric-cloud.com
What to measure in the pipeline
Idle time
Defects discovered/
escaped, impact of
defects
MTTD
Release frequency,
% Automated
Time/cost per
release
Predictability,
Planned vs. Actual
Deployment lead
time
Deployment
frequency, duration
Change success rate
MTTR
MTTR
Cost/frequency of
outages
On-call after
business hours
Performance /
utilization
Development lead
time
Rework required by
defects, build
breakage, downtime
Idle time
Work-in-progress
and technical debt
Cycle time
-- Cycle Time --
-- Visibility --
-- Scale --
DEV/CI QA Deploy Release Operate

19. © Electric Cloud | electric-cloud.com
What to measure in the pipeline
Idle time
Defects discovered/
escaped, impact of
defects
MTTD
Release frequency
Time/cost per
release
Predictability
Deployment lead
time
Deployment
frequency, duration
Change success rate
MTTR
MTTR
Cost/frequency of
outages
On-call after
business hours
Performance /
utilization
-- Cycle Time --
-- Visibility --
-- Scale --
QA Deploy Release OperateDEV/CI
Development lead
time
Rework required by
defects, build
breakage, downtime
Idle time
Work-in-progress
and technical debt
Cycle time

20. © Electric Cloud | electric-cloud.com
Dev/CI: Where to Start?
• Adopt CI and trunk-based development
(or very-short-lived branches)
• Treat “Everything-as-code”
• Enable self-service automation for
environment provisioning
• Reduce unplanned work downstream
▪ “Build-in” Quality
▪ “Build-in” Security
▪ “Build-in” Performance

21. © Electric Cloud | electric-cloud.com
What to measure in the pipeline
Release frequency
Time/cost per
release
Predictability
Deployment lead
time
Deployment
frequency, duration
Change success rate
MTTR
MTTR
Cost/frequency of
outages
On-call after
business hours
Performance /
utilization
-- Cycle Time --
-- Visibility --
-- Scale --
Deploy Release OperateDEV/CI
Development lead
time
Rework required by
defects, build
breakage, downtime
Idle time
Work-in-progress
and technical debt
Cycle time
QA
Idle time
Defects discovered/
escaped, impact of
defects
MTTD

22. © Electric Cloud | electric-cloud.com
QA: Where to Start?
• Implement Test Automation
• Evolve from “Tester” to “Test Writer”
• Enable self-service automation for
environment provisioning
• Ensure fidelity of env, artifact, and
deployment process (align with PROD)

23. © Electric Cloud | electric-cloud.com
What to measure in the pipeline
Release frequency
Time/cost per
release
Predictability
MTTR
Cost/frequency of
outages
On-call after
business hours
Performance /
utilization
-- Cycle Time --
-- Visibility --
-- Scale --
Release OperateDEV/CI
Development lead
time
Rework required by
defects, build
breakage, downtime
Idle time
Work-in-progress
and technical debt
Cycle time
QA
Idle time
Defects discovered/
escaped, impact of
defects
MTTD
Deployment lead
time
Deployment
frequency, duration
Change success rate
MTTR
Deploy

24. © Electric Cloud | electric-cloud.com
Deploy: Where to Start?
• Automate deployments
• Treat “Everything-as-code”
• Maintain inventory of existing
environments
• Ensure fidelity of env, artifact, and
deployment process

25. © Electric Cloud | electric-cloud.com
What to measure in the pipeline
MTTR
Cost/frequency of
outages
On-call after
business hours
Performance /
utilization
-- Cycle Time --
-- Visibility --
-- Scale --
OperateDEV/CI
Development lead
time
Rework required by
defects, build
breakage, downtime
Idle time
Work-in-progress
and technical debt
Cycle time
QA
Idle time
Defects discovered/
escaped, impact of
defects
MTTD
Deployment lead
time
Deployment
frequency, duration
Change success rate
MTTR
Deploy
Release frequency,
% Automated
Time/cost per
release
Predictability,
Planned vs. Actual
Release

26. © Electric Cloud | electric-cloud.com
Release: Where to Start?
• Integrate, centrally orchestrate and view the
entire pipeline across all teams, projects,
pipelines
• Ensure Fidelity of environment, artifact,
deployment process - across all stages
• Provide security, governance and reporting for
all application promotion
• Coordinate scheduling and dependencies at
scale (microservices make this harder!)

27. © Electric Cloud | electric-cloud.com
What to measure in the pipeline
-- Cycle Time --
-- Visibility --
-- Scale --
DEV/CI
Development lead
time
Rework required by
defects, build
breakage, downtime
Idle time
Work-in-progress
and technical debt
Cycle time
QA
Idle time
Defects discovered/
escaped, impact of
defects
MTTD
Deployment lead
time
Deployment
frequency, duration
Change success rate
MTTR
Deploy
Release frequency
Time/cost per
release
Predictability
Release
MTTR
Cost/frequency of
outages
On-call after
business hours
Performance /
utilization
Operate

28. © Electric Cloud | electric-cloud.com
Operate: Where to Start?
• Version control all artifacts
(rollback, governance, visibility)
• Monitor health of systems and
applications
• Self-service provisioning of
environments
• Shared infrastructure to drive
down opex/capex

29. © Electric Cloud | electric-cloud.com
Questions?

30. © Electric Cloud | electric-cloud.com
Resources
• http://electric-cloud.com/wp-content/uploads/DOES15_forum_metrics_102015.pdf
• https://puppet.com/resources/white-paper/2015-state-of-devops-report
• http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1765804/pdf/v013p0ii22.pdf
• http://devops.com/2014/11/10/devops-scorecard/
• http://www.datical.com/blog/9-metrics-devops-teams-tracking/
• http://devops.com/2015/01/26/metrics-devops/
• https://blog.appdynamics.com/devops/quantified-devops/
• http://www.slideshare.net/jedi4ever/devops-metrics
• http://www.slideshare.net/ITRevolution/does15-troy-magennis-and-julia-wester-metrics-and-
modeling-helping-teams-see-how-to-improve

31. © Electric Cloud | electric-cloud.com
Thank You!