This is a presentation on how to create and implement meaningful defect metrics for software projects.

1. By Harmony Brenner
January 21, 2014

2. “Measurement is the first step that leads to control
and eventually to improvement.
If you can’t measure something, you can’t understand it.
If you can’t understand it, you can’t control it.
If you can’t control it, you can’t improve it.”
― H. James Harrington

3. Agenda
 Goals
 Definitions
 Pre-Requisites
 Process Requirements
 Metrics and Exclusions
 Calculate DRE Metrics
 Calculate Rework Metrics
 Presenting Results
 Metrics Usage and Sharing

4. Goals
 Increase attention given to all phases of testing
(Unit, System, User Acceptance) to improve quality.
 Decrease number of defects detected in later
phases of testing to reduce rework and overall costs.
 Implement metrics as an objective method to measure
outcomes and improvements over time.

5. Definitions
 DRE – Defect Removal Effectiveness.
 Metrics – Use of numerical ratings to determine
quality and performance of item / phase being
measured.
 Also define any Testing phases and describe who /
which team(s) own accountability.
 Examples – SIT is usually done by IT Team;
UAT is usually performed by Business Team

6. Pre-Requisites
 Ensure your Defect Tracking system contains discrete
categories for the following:
 Statuses
 Reasons / outcomes for defect closure
 Impact Levels
 Root Causes
 Ability to track Cycles of Rework

7. Process Requirements
 Encourage consistent creation and updates of defects
by IT and Business teams immediately upon discovery.
 Mentor teams how to create effective defect reports.
 Foster communication between teams to ensure that
all release defects are worked through to completion.
 Defect Tracking Training
 Resolve other barriers to acceptance

8. Metrics and Exclusions
 Decide and describe whether any Defect Statuses
should be excluded from measurement.
 Example – statuses indicate the defect should not have
been created may merit exclusion.

Cancelled, No Action Required, etc.

9. Metrics and Exclusions (cont’d)
 Decide and describe whether any Defect Root Causes
should be excluded from measurement.
 Example – categories that indicate the defect should not
have been created may merit exclusion.

Change Request, User Error, etc.

10. Calculate DRE Metrics
 Calculate the DRE (Defect Removal Effectiveness) for
each test phase in the project to be measured:
 Phase DRE =
Phase Defects / (Phase Defects + All Defects)

SIT Defects (10) / (SIT Defects (10) + UAT Defects (5) )
10 / 15 = 67% is SIT DRE
5 / 15 = 33% is UAT DRE

11. Calculate Rework Metrics
 Decide how to track Rework instances for defects.
 Example – Audit trail may count changes between
Retest vs. Rework statuses.

Where possible, automate Rework counts per defect so these
can be calculated vs. manually determined.

12. Presenting Results
 Share results in a non-judgmental framework.
 Anonymize and de-personalize outcomes wherever
possible. This is team, not individual, performance.
 Give credit where due and be a change champion!

13. Metrics Usage and Sharing
 Always tell yourself the truth!
 Use metrics to raise awareness on overall quality –
not as punishment!
 Share metrics at regular intervals – no surprises.
 Compare metrics over time on the same project or
among different projects to measure performance.

14. Want to Learn More?
 I enjoy professional discourse on a variety of subjects
pertaining to Software Quality and Testing.
 Harmony Brenner – harmony.brenner@gmail.com
 LinkedIn – www.linkedin.com/in/harmonybrenner