1) The document discusses how metrics intended to measure continuous delivery (CD) success, like the DORA metrics, can be misused if not properly understood in context. 2) It provides examples of how goals focused solely on metrics like deployment frequency can negatively impact quality and sustainability. 3) The author advocates for a balanced set of metrics that measure efficiency, effectiveness, quality, sustainability and employee/customer satisfaction. Leading indicators should support business goals rather than become goals themselves.

1. HOW TO MISUSE AND
ABUSE DORA METRICS

2. BRYAN FINSTER
Deploy more & sleep better
Distinguished Engineer
Value Stream Architect
Board Advisor

3. PILOTING CD
“Why can’t we release every day?”
Walmart Logistics SVP’s Challenge:
“Can we increase release frequency from
quarterly to bi-weekly?”

4. Architecture: Cohesive, loosely coupled
teams aligned to business capabilities
Enablement: Delivery platform team
Established KPIs
EXPERIMENTS

5. COMPLETED/COMMITTED %
• Improved throughput
Goal
• Teams focused on planning
and estimate accuracy, not
value delivery
Outcome

6. CODE TEST COVERAGE %
• Improved testing
Goal
• Tests with no assertions
• Tests with meaningless
assertions
• High coverage of untrustworthy
tests
Outcome

7. LESSONS
Highly cohesive, loosely coupled teams with a shared
mission worked better than scaling frameworks
“Why can’t we deliver today?” was an effective tool for
uncovering roadblocks.
CD improved business outcomes and morale
And…

8. WE NEEDED
BETTER METRICS
Metrics change behavior

9. SCALING TO THE ENTERPRISE
CTO Goals
Use CD to improve engineering excellence
Enable more effective, and sustainable delivery

10. EXPERIMENT
Opinionated CD platform
Gamified CD signals
• Trunk-based development
• Continuous integration
• Code quality
• Delivery frequency

11. LESSONS
Gamified metrics helped early and middle adopters
Opinionated pipeline encouraged late adopters to explore CD
We needed better ways to communicate “why” to leadership.

12. CONFIRMED BY INDUSTRY DATA
“Continuous delivery improves both delivery
performance and quality, and also helps
improve culture and reduce burnout and
deployment pain.”
-- Accelerate
2018

14. WE CAUSED A
PROBLEM
Complex
Emergent
Complicated
Good Practices
Chaos
Novel
Obvious
Best Practice
Landed here
Measuring
Behavior
The 4 metrics make sense in the context
of the rest of the book
People don’t read books
We over-simplified the metrics
Their purpose got lost in translation

16. GOALS
“To be a high performing
organization, we need better
DORA metrics!”
• Correlation != Causation
• High performing organizations
focused on improving how they
deliver value
• Goals should focus on value, not
DORA metrics
Fallacy Reality

17. PRODUCTIVITY
“Now we can compare the
productivity of our teams!”
• Each team has their own context
• We should measure to identify
constraints, not productivity
• comparing teams is destructive to
our goals
Fallacy Reality

18. SPEED
“We need to deploy more
frequently to deliver faster!”
• Measure of batch size, not speed
• Smaller batches delivered more
frequently to expose waste and
improve our quality processes
• Improved quality process
improves speed
Fallacy Reality

19. VANITY RADIATORS
Deploys
Lead
Time
MTTR
Change
Fail %
437
3 5 8%
hours hours
Deploys
Lead
Time MTTR
Change
Fail %
645
2.3 2 10%
days hours

20. DELIVERY HEALTH INDICATORS
Batch size shrinking?
Quality improving?
Less toil?
Faster feedback?
Happier customers?
Happier teams?

21. Lead Time for Change
Change Fail % MTTR
Deploy Frequency
Flow
• Total lead time
• Work in progress
Continuous Integration
• Branch duration
• Integration frequency
• Mean time to detect
Culture
• Westrum score
• Employee NPS
• After hours work
Customer Outcomes
• NPS
• Downtime for deploy
FOUR METRICS?
An incomplete view

22. INFORMATION
RADIATORS
“It’s only when they’re combined with the
use of visual displays… that we see a strong
effect.” -- Accelerate
bit.ly/usaf-holocron

23. INFORMATION
RADIATORS
“A goal without a method is nonsense.”
- W. Edward. Deming
IMPROVEMENT
RADIATORS
bit.ly/usaf-holocron

24. Trust
-
Mission
-
Learning
-
Improving
Culture
Business Goals, Customer Value, & Sustainability
• Profitable, Happy customers, Happy teams
Business Objectives
• Measurable outcomes (Key Results)
Flow Metrics
• WIP, Lead time, Development cycle time, Throughput, Flow efficiency
Continuous Delivery Metrics
• Deploy frequency, Pipeline cycle time, Defect rate, MTTR
Continuous Integration Metrics
• Code integration frequency, Branch duration
Low
Data
Fidelity
High
LEADING & LAGGING INDICATORS

25. Customer &
Business Outcomes
Objectives & Key
Results
Flow
Continuous
Delivery
Continuous
Integration
EACH LEVEL OF
METRICS DEPENDS
ON UNDERLYING
METRICS
To improve flow, we must
improve CI

26. Continuous Integration
Education
We shouldn’t wager business goals on hope
that teams will self-train on security, testing,
modern delivery, etc.
IMPROVEMENT IS
DEPENDENT ON
INVESTMENT IN
PEOPLE

27. METRICS NEED BALANCE
Efficiency Effectiveness
Moving towards smaller, more frequent
batches uncovers delivery pain

28. METRICS NEED BALANCE
Efficiency Effectiveness
Moving towards smaller, more frequent
batches uncovers delivery pain
Quality is a guardrail against prioritizing
speed

29. METRICS NEED BALANCE
Efficiency
Effectiveness &
Sustainability
Defect Rates
& Customer
NPS
Employee
NPS
Integration
Frequency
Delivery
Frequency
Moving towards smaller, more frequent
batches uncovers delivery pain
Quality is a guardrail against prioritizing
speed
Team feedback is a guardrail against false
improvement

30. CLOSING THOUGHTS
 The 4 outcome metrics are only the tip of the iceberg.
 Product development is a complex interaction of people, process,
and products. There are no simple metrics.
 Measures require guardrails to avoid perverse incentives.

31. CLOSING THOUGHTS
 Metrics are a critical part of the improvement toolbox, but…
 We cannot measure our way to improvement.
 We use them to monitor and inform the next improvement
experiment.
 Don’t measure people, invest in them. They are our most valuable
asset.

32. RESOURCES
bit.ly/continuous-videos

33. LET’S COMPARE NOTES!
bryan-finster
@BryanFinster
blog.bryanfinster.com
bit.ly/dora-metrics