The document discusses the complexities and potential harms of using metrics in organizations, particularly emphasizing that many commonly used metrics can be harmful if they are the wrong ones. It outlines various types of metrics and essential test metrics while proposing rules for identifying good metrics that promote positive behavior without creating undue pressure. The importance of thoughtful metric application is highlighted, along with principles to ensure metrics serve their intended purpose without leading to negative consequences.

1. MINDFUL METRICS
DMITRY SHARKOV

6. METRICS!
1. metrics are good
2. metrics are bad

7. "Most people have no idea how to measure well. They make
their organization run a marathon with a thermometer up
its rear end, and then they wonder why it's running so
slowly (and awkwardly)."
Jurgen Appelo, “Managing for Happiness”
METRICS ARE HARD…

9. MOST METRICS ARE HARMFUL IF…
The wrong metrics are harmful.
Most metrics are the wrong metrics.

10. DEFINE: METRICS
metric: (n) measurement of characteristics that are
countable or quantifiable

11. TYPES OF METRICS
• Process health metrics
• Product development metrics
• Release metrics
• Technical / code metrics
• People / team metrics
source: Jason Tice @theagilefactor

12. TYPES OF METRICS (2)
• Productivity
• Predictability
• Responsiveness
• Quality
source: asynchrony.com

13. 64 ESSENTIAL TEST METRICS
1. Total number of test cases
2. Number of test cases passed
3. Number of test cases failed
4. Number of test cases blocked
5. Number of defects found
6. Number of defects accepted
7. Number of defects rejected
8. Number of defects deferred
9.Number of critical defects
10. Number of planned test hours
11. Number of actual test hours
12. Number of bugs found after shipping
13. Passed test cases ratio
14. Failed test cases ratio
15. Blocked test cases ratio
16. Fixed defects ratio
17. Defects accepted ratio
18. Defected rejected ratio
19. Defects deferred ratio
20. Critical defects ratio
21. Average time to repair defect
22. Number of tests run per time period
23. Test design efficiency
24. Test review efficiency
25. Defects found per test hour ratio
26. Defects found per test ratio
27. Average time to test bug fix
28. Defect containment efficiency
29. Test effectiveness using team assessment
30. Tests run percentage
31. Requirement coverage
32. Test cases by requirement
33. Defects per requirement
34. Requirements without test coverage
35. Total allocated costs for testing
36. Actual cost of testing
37. Budget variance
38. Schedule variance
39. Cost per bug fix
40. Cost of not testing
41. Distribution of defects returned per team member
42. Distribution of open defects for retest per test team member
43. Test cases allocated per test team member
44. Test cases executed by the test team member
45. Test execution status chart
46. Test execution and defect find rate tracking
47. Effect of testing changes
48. Defect injection rate
49. Defect distribution by cause
50. Defect distribution by functional area
51. Defect distribution by severity
52. Defect distribution by priority
53. Defect distribution by type
54. Defect distribution by tester
55. Defect distribution by test type
56. Defect distribution by platform/environment
57. Defect distribution over time by cause
58. Defect distribution over time by module
59. Defect distribution over time by severity
60. Defect distribution over time by platform
61. Defects created vs Defects resolved
62. Defect removal efficiency (defect gap)
63. Defect density
64. Defect age
source: https://www.qasymphony.com/blog/64-test-metrics/

14. WHAT IS HARM? ( THE WRONG METRICS ARE HARMFUL )

17. AMAZON, WHOLE FOODS, AND METRICS
source: gizmodo.com
“They say many employees are terrified
of losing their jobs under the new system
and that they spend more hours mired in
OTS-related paperwork than helping
customers.”
“Seeing someone cry at work is
becoming normal.”
( THE WRONG METRICS ARE HARMFUL )

18. AMAZON, WHOLE FOODS, AND METRICS
source: gizmodo.com
( THE WRONG METRICS ARE HARMFUL )

19. INCREASING VELOCITY IN BOSTON ( THE WRONG METRICS ARE HARMFUL )

20. INCREASING DEFECTS IN BOSTON ( THE WRONG METRICS ARE HARMFUL )

21. IN HARM’S WAY
Icons made by Freepik from www.flaticon.com
Product Process People
( THE WRONG METRICS ARE HARMFUL )

22. WHAT METRICS DO…
“What gets measured, gets done.”
“What gets measured, improves.”
( THE WRONG METRICS ARE HARMFUL )

23. WHAT METRICS DO…
source: network-graphics.com, chevyfirst.com
( THE WRONG METRICS ARE HARMFUL )
1998
2018

24. INCENTIVES…
“When a measure becomes a target, it ceases to be a good metric.”
Goodhart’s Law
“As soon as you set a target for others, they will pursue the target
instead of the original purpose.”
Jurgen Appelo, “Managing for Happiness”
( THE WRONG METRICS ARE HARMFUL )

25. DISINCENTIVES…
bill rate
pay rate
recruiter account manager
( THE WRONG METRICS ARE HARMFUL )

26. EXTRINSICALLY MOTIVATED, INDIVIDUAL METRICS
1. Extrinsic motivation does not have a positive
correlation with productivity.
2. Morale is more likely to be negatively affected than
with other metrics.
sources:
Srivastava, S.K. and Barmola, K.C., 2012. Role of motivation in higher productivity.
Management Insight, 7(1).
Dan Pink: The puzzle of motivation | TED Talk | TED.com
http://www.lse.ac.uk/website-archive/newsAndMedia/news/archives/2009/06/
performancepay.aspx
http://www.management-issues.com/news/5640/performance-related-pay-doesnt-
encourage-performance/
( THE WRONG METRICS ARE HARMFUL )

27. "Gathering and analyzing data is a burden the team bears
at the expense of other work"
Me, today
THE WRONG METRICS ARE HARMFUL.

28. “If you torture the data long enough, it will confess.”
Ronald H. Coase
(20th century British economist)
MOST METRICS ARE THE WRONG METRICS.

29. RULES OF “GOOD” METRICS
‣ A good metric is comparative
‣ A good metric is understandable
‣ A good metric is a ratio
‣ A good metric changes the way
you behave
( MOST METRICS ARE THE WRONG METRICS )

30. 12 RULES OF GOOD METRICS ( MOST METRICS ARE THE WRONG METRICS )

31. 12 RULES OF GOOD METRICS
1. Measure for a purpose. (Inform decisions. Measure what you want to improve.)
2. Shrink the unknown. (Recognize limitations of measures but use what you can.)
3. Seek to improve. (Avoid vanity metrics.)
4. Delight all stakeholders. (Account for intersection and difference in needs.)
5. Distrust all numbers. (Account for observer effect and risk compensation.)
6. Set imprecise targets. (Avoid deflecting from goals with tangential measures.)
( MOST METRICS ARE THE WRONG METRICS )

32. 12 RULES OF GOOD METRICS
7. Own your metrics. (Don't measure subordinates; measure yourself.)
8. Don’t connect metrics to rewards. (What is rewarded is never the organization's
true purpose.)
9. Promote values and transparency. (Nourish self-motivation; avoid pay-for-
performance.)
10.Visualize and humanize. (Don't get caught up in numbers, forgetting people.)
11.Measure early and often.
12.Try something else.
( MOST METRICS ARE THE WRONG METRICS )

33. VANITY METRICS ( MOST METRICS ARE THE WRONG METRICS )

34. VANITY METRICS ( MOST METRICS ARE THE WRONG METRICS )
number of test
scenarios
coverage ratio of
critical featuresVS

35. APPLYING METRICS TO A SAMPLE TEAM… ( MOST METRICS ARE THE WRONG METRICS )

36. 64 “ESSENTIAL” TEST METRICS
1. Total number of test cases
2. Number of test cases passed
3. Number of test cases failed
4. Number of test cases blocked
5. Number of defects found
6. Number of defects accepted
7. Number of defects rejected
8. Number of defects deferred
9.Number of critical defects
10. Number of planned test hours
11. Number of actual test hours
12. Number of bugs found after shipping
13. Passed test cases ratio
14. Failed test cases ratio
15. Blocked test cases ratio
16. Fixed defects ratio
17. Defects accepted ratio
18. Defected rejected ratio
19. Defects deferred ratio
20. Critical defects ratio
21. Average time to repair defect
22. Number of tests run per time period
23. Test design efficiency
24. Test review efficiency
25. Defects found per test hour ratio
26. Defects found per test ratio
27. Average time to test bug fix
28. Defect containment efficiency
29. Test effectiveness using team assessment
30. Tests run percentage
31. Requirement coverage
32. Test cases by requirement
33. Defects per requirement
34. Requirements without test coverage
35. Total allocated costs for testing
36. Actual cost of testing
37. Budget variance
38. Schedule variance
39. Cost per bug fix
40. Cost of not testing
41. Distribution of defects returned per team member
42. Distribution of open defects for retest per test team member
43. Test cases allocated per test team member
44. Test cases executed by the test team member
45. Test execution status chart
46. Test execution and defect find rate tracking
47. Effect of testing changes
48. Defect injection rate
49. Defect distribution by cause
50. Defect distribution by functional area
51. Defect distribution by severity
52. Defect distribution by priority
53. Defect distribution by type
54. Defect distribution by tester
55. Defect distribution by test type
56. Defect distribution by platform/environment
57. Defect distribution over time by cause
58. Defect distribution over time by module
59. Defect distribution over time by severity
60. Defect distribution over time by platform
61. Defects created vs Defects resolved
62. Defect removal efficiency (defect gap)
63. Defect density
64. Defect age
https://www.qasymphony.com/blog/64-test-metrics/
( MOST METRICS ARE THE WRONG METRICS )

38. OK THEN…
How do we identify those metrics that are useful and unlikely to
"go bad" or be misused?
How do we prevent those that seem useful from "going bad" or
being misused?

39. "..it's no wonder that an organization which measures very
little and just runs around blindly usually goes much
faster, until it runs into a tree, hopefully not with a
thermometer in its mouth."
Jurgen Appelo, “Managing for Happiness”

40. GOOD METRICS
1. A good metric is comparative
2. A good metric is understandable
3. A good metric is a ratio
4. A good metric changes the way you
behave
1. Measure for a purpose.
2. Shrink the unknown.
3. Seek to improve.
4. Delight all stakeholders.
5. Distrust all numbers.
6. Set imprecise targets.
7. Own your metrics.
8. Don’t connect metrics to rewards.
9. Promote values and transparency.
10.Visualize and humanize.
11.Measure early and often.
12.Try something else.
Croll & Yoskovitz, “Lean Analytics”
Jurgen Appelo, “Managing for Happiness”

41. GOOD METRICS — 4 VALUES
1. Value data that provides actionable knowledge over data
that does not.
2. Value metrics aimed at the team over metrics aimed at
management.
3. Value measuring the end over measuring the means.
4. Value inspecting trends over points in time.

42. VALUE 1
Value data that provides actionable
knowledge over data that does not.
new release system
downtime trendline
accuracy of work
estimates

43. VALUE 2
Value metrics aimed at the team
over metrics aimed at management.
integration test
coverage within
codebase
pace of delivery
across teams

44. VALUE 3
Value measuring the end over
measuring the means.
test cycle time
number of
automated vs
manual tests
number of escapes time spent testing

45. VALUE 4
Value inspecting trends over points in time.
test coverage trend line
points delivered in
last sprintnumber of escapes
trend line
escapes last sprinttest suite total
runtime trend line
team happiness trend line
team happiness today

46. GOOD METRICS — 5 PRINCIPLES
1. Don't over-invest in a metric; don't overmeasure.
2. Don't let management care more than the team cares; let the
team drive interest and action items.
3. Don't ignore potential side effects; be aware of zero-sum
relationships in the process.
4. Don't use metrics to measure team productivity against others.
5. Don't introduce additional rewards, let alone penalties.

47. PRINCIPLE 1
Don't over-invest in a
metric; don't overmeasure.

48. PRINCIPLE 2
Don't let management
care more than the team
cares; let the team drive
interest and action items.
scrumalliance.org

49. PRINCIPLE 3
Don't ignore potential
side effects; be aware of
zero-sum relationships in
the process.
carthrottle.com

50. PRINCIPLE 4
Don't use metrics to measure team productivity against others.
gizmodo.com

51. PRINCIPLE 5
Don't introduce additional
rewards, let alone penalties.

52. APPLYING THE VALUES AND PRINCIPLES

53. APPLYING THE VALUES AND PRINCIPLES

54. APPLYING THE VALUES AND PRINCIPLES

55. APPLYING THE VALUES AND PRINCIPLES
https://www.flickr.com/photos/gen/2648500033/

56. APPLYING THE VALUES AND PRINCIPLES
Lisa Mckelvie / Getty Images

57. MINDFUL METRICS
@DmitrySharkov