The document discusses various agile metrics that can be used to improve quality, shorten product cycle times, decrease costs and risks, and increase predictability. It describes metrics like defects trends, test automation coverage, flow diagrams to measure lead and cycle times, earned value analysis to track costs and wastage. Control limits and variances can be set for work in progress, defects, team size and velocity. The focus should be on analyzing trends rather than individual values to identify special causes of variation and lead indicators.

1. Agile Metrics and Quality

2. Metrics -> Improvements
Use metrics to
• Improve quality
• Shorten product cycle time
• Decrease costs
• Decrease risks
• Increase predictability and reliability

3. Control Limits and Variances
We can have limits for
• WIP
• Defects
• Team size
• Velocity
• Time-boxes
• Others?
We can analyze variances to understand
the special causes of variation

4. Focus on Trends
• Trends matter more than values
• Trends and spikes are only the symptoms
• Trends are leading indicators
Images from MountainGoat Software

5. Improve quality
◦ Quality is measured by passed tests and
customer acceptance
◦ Quality is defined by Definition of Done
and Acceptance Criteria
◦ Defect trends to use in Agile projects
 Number of open defects
 Escaped defects
 Defect Cycle Time
 Defect spill-over
◦ Test Automation is critical!

6. Defect Injection and
Resolution
What is the trend of total Active Defects?
Image from MountainGoat Software

7. Analyzing and Preventing
Pareto Chart Defects
Fishbone / 5 Why Technique

8. Escaped Defects
• Used to measure the quality of delivered
code
 Calculation:
9
◦ For all releases, find 8
7
all defects related to
6
the release found
5
after the release date
4
3
◦ Add up all the defects 2
1
◦ Can be captured per
0
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
day / week / month /
… or per sprint / Escaped Defects over Time
release

9. Defect Cycle Time
• Rapid resolution of defects is important
• Average bug-fix time can also be tracked for
different priority bugs
Defect Cycle Time
20
18
16
Desired Threshold
14
12
Hours
10
8
6
4
2
0
1 2 3 4 5 6 7 8 9 10
Sprint No.

10. Defect Spill-over
• Resolution of defects for a sprint story within the sprint is
important
• Definition of Done can include a criteria like “No P1 defects”
• We can also tracked whether spilled-over defects are being
closed in the next sprint
Defect Spill Over
9
8
7
No of defects
6
5
4
3
2
1
0
1 2 3 4 5 6 7 8 9 10
Sprint No

11. Test Automation
 Critical factor for regression testing
 Coverage should be as high as possible

12. Shorten product cycle time
 Cumulative Flow Diagrams
◦ Lead Time
◦ Cycle Time

13. Features CFD – An Elaboration
Time

14. What does this CFD say?
Not started
Started
Completed
Too much Work-In-Progress!
Image from MountainGoat Software

15. Identifying bottlenecks with a
CFD
1 2 3 4 5 6 7 8 8 10 11 12 13
Day
Where is the bottleneck?
In DB Procs – After the widening Analysis phase
Image from http://www.soliantconsulting.com

16. Decrease costs
• Earned Value Management
• Wastage

17. Earned Value Analysis
PV – Planned Value
EV - Earned Value
AC - Actual cost
BAC - Budget @
Completion
EAC - Estimate @
Completion
ETC - Estimate to
CompletionCost Variance (CV) = EV – AC
VAC- Variance @ Completion (SV)
Schedule Variance = EV – PV
Cost Performance Index (CPI) = EV / AC
Schedule Performance Index (SPI) = EV / PV
Estimate at Completion (EAC) = BAC / CPI or
= AC + BAC – EV
Estimate To Complete (ETC) = EAC – AC
Variance at Completion (VAC) = BAC – EAC

18. EVM - Interpretation
PV EV AC Schedule Cost
$1 $1 $1 On schedule On cost
$2 $2 $1 On schedule Under cost
$1 $1 $2 On schedule Over cost
$1 $2 $2 Ahead of schedule On cost
$1 $2 $3 Ahead of schedule Over cost
$1 $2 $1 Ahead of schedule Under cost
$3 $2 $1 Behind schedule Under cost
$2 $1 $3 Behind schedule Over cost
$2 $1 $1 Behind schedule On cost

19. EVM - Progress
EV PV AC
CV
SV
Ref: Earned Value and Agile Reporting
Anthony Cabri, Mike Griffiths

20. Wastage
 Any time not dedicated to working on the
highest priority story / task on the sprint
backlog
 Should be tracked over time and by cause
Wastage (in hrs) Wastage (in hrs)
60 160
140
50 120
100
40 80
60
30 40
20
20 0
10
0
1 2 3 4 5 6 7 8 9 10 11 12

21. Decrease risks
• WIP Limits
• Small Iterations
• More predictability

22. Increase Predictability /
Reliability
• Velocity
• Lead Time
• Cycle Time

23. Metrics to Avoid
• Velocity comparisons
• Productivity e.g. Story Points per day
• Effort Estimation accuracy
• Number of stories completed
• LOC/Function Points
• Code quality

24. Questions?
Contact
Anu.khendry@gmail.com