Aleksandra Petrovic's (Senior QA engineer specializing in regression and performance testing) presentation at www.OnlineTestConf.com

1. endava.com
QUALITY. PRODUCTIVITY. INNOVATION.
TESTING METRICS
And Why Managers
Like Them
Aleksandra Petrovic
Nov 2016

2. 2 QUALITY. PRODUCTIVITY. INNOVATION.
 Software Quality
 What are Testing Metric?
 Why we need metrics and why
managers like them?
 Important Metrics and how to
identify the ones we need
 Metrics Report Example
 Can Metrics Cheat?
Testing Metrics

3. 3
3 QUALITY. PRODUCTIVITY. INNOVATION.
What makes software development projects successful?
• Product quality
• Optimize implemented processes
• Increase Team efficiency
• Reduce cost and time needed to deliver
• Keep customers happy
Software Quality

4. 4
4 QUALITY. PRODUCTIVITY. INNOVATION.
“When you can measure what you are speaking about, and can express it in
numbers, you know something about it; but when you cannot measure it,
when you cannot express it in numbers, your knowledge is of a meager and
unsatisfactory kind.”
-- Lord Kelvin, a physicist.
Software testing metric is a quantitative measure of the degree to which
a system, component, product or process possesses a given attribute. In
other words, metrics helps estimating the progress, quality and health of a
software testing effort.
Example of software testing metric: Total number of defects found
What is Software Testing Metric?

5. 5
5 QUALITY. PRODUCTIVITY. INNOVATION.
• "We cannot improve what we cannot measure“
• Display past and present performance
• Predict future
• Understand what needs to improve
• Decide what to do next
Why do Test Metrics?

6. 6
6 QUALITY. PRODUCTIVITY. INNOVATION.
“All they (managers) care about are numbers…”
• Your team is doing a great job – show it!
• Inefficiency affects schedules and product quality
• Need to have a way to measure QA process quality
• Metrics help you generate good test reports
Why Managers Like Them?

7. 7
7 QUALITY. PRODUCTIVITY. INNOVATION.
Metrics can be sorted into groups based on different criteria:
• Project-, department- and company-level metrics
• Process (process efficiency), product (improve software quality) and
project metrics (measure team or tools efficiency)
• Base (raw data collected during execution) and calculated metrics
(derive from data in base metrics usually for test reporting purpose)
Metric Types

8. 8
8
Important Metrics (1)
1. Test cases:
• How many test cases designed per requirement?
• How many test cases executed/pending?
• How many test cases passed/failed/blocked? How many failed due to
one particular defect?
2. Defects:
• Total number of defects identified
• Defect distribution per Severity, per Priority, per Component/Module
• Defect Aging
• Defect Density and Defect Density in released code
3. Automation:
• Number of defects found in automated tests (Are they still useful?
Watch for pesticide paradox!)
• Percent of automated test coverage
• How long it takes to run test plan and how often we do it?
4. People/QA Team:
• Execution by user: How many test cases executed per person?
QUALITY. PRODUCTIVITY. INNOVATION.

9. 9
9
Important Metrics (2)
1. Defect distribution by status and phase
• Expected to decrease as the software quality improves toward the
project end
2. Defect open and close rates
• Insight into the ability of testers and developers to work together to
identify and address software issues
3. Defect removal efficiency (DRE)
• Rate at which team is able to adequately fix identified program flaws
4. Burn down chart
• Visual representation of the amount of work yet to be completed
5. Defect severity index
• Shine some light on the effectiveness of development team
QUALITY. PRODUCTIVITY. INNOVATION.

10. 10
10
Important Metrics (3)
1. Number of issues reported by customers
• Determine effectiveness of QA departments (risk-based testing to reduce
gaps)
2. Mean Time to Detect (MTTD) and Mean Time to Repair (MTTR)
• MTTD - how long it takes QA professionals to find a problem
• MTTR - amount of time needed to effectively address it
3. Number of system outages and downtime
• The number/frequency of system outages and length of downtime
experienced by end user
4. COQ (Cost of Quality) and COPQ (Cost of Poor Quality)
• COQ: Total effort put into quality-related activities (development, testing,
reviews)
• COPQ: Cost of fixing defects, updating docs, re-testing, patch distribution
QUALITY. PRODUCTIVITY. INNOVATION.

11. 11
11 QUALITY. PRODUCTIVITY. INNOVATION.
A few tips how to identify correct metrics:
• Set the target audience
• Define the goal for metrics
• Add only relevant metrics based on your project needs (less is more)
• Analyze the cost benefits aspect of each metric
How to identify the right metrics?

12. 12
12 QUALITY. PRODUCTIVITY. INNOVATION.
Steps to create Metrics Report
Identify the key
processes to be
measured
Set the target value for
metrics (baseline)
Identify how often to
collect data and who
should do it
Collect actual data,
calculate and interpret
the results
Create report and
present to stakeholder
with reasonable
conclusions
If things don’t look
good, investigate
reasons and suggest
how to improve

13. 13
13
Metrics Report Example
QUALITY. PRODUCTIVITY. INNOVATION.
METRIC ACTUAL VALUE TARGET
Total number of open defects 18 20
Incoming defect rate 7
Defects found by automation
tests 3
Breakage introduced by new
features (regressions) 5
Outgoing defect-fix rate 5
Severity 1 (catastrophic outage) 0 0
Severity 2 (severe breakage) 2 0
Severity 3 (moderate issue) 3 2
Severity 4 (cosmetic issue) 12 Best Effort
Documentation defects 40 Best Effort
Defects found by customers 4 Best Effort
MTTR (days) 7 10
Customer satisfaction 4.3 4.8

14. 14
14
Metrics – Graphical Representation (1)
QUALITY. PRODUCTIVITY. INNOVATION.
Defects per Severity
Total number of open defects

15. 15
15 QUALITY. PRODUCTIVITY. INNOVATION.
Metrics – Graphical Representation (2)
Open vs Resolved Defects
Defects per Component:
1. Cumulative Chart
2. JIRA Pie Chart

16. 16
16 QUALITY. PRODUCTIVITY. INNOVATION.
• Yes, metrics can cheat too! Usually when they are taken out of
context…
• Need reliable input data
• Cheat accidentally (incorrect input data) or cheat on purpose
(‘tweak’ your queries to get better stats)
• True story example (on the next slide)
• Project metric that did not take into account that there
was a new defect tracking tool introduced!
Can Metrics Cheat?

17. 17
17 QUALITY. PRODUCTIVITY. INNOVATION.
What my manager saw
0
20
40
60
80
100
120
140
Incoming Defect Rate (ALL Defects)
S1-S3 S1-S5

18. 18
18 QUALITY. PRODUCTIVITY. INNOVATION.
After closer look (behind the numbers)
0
10
20
30
40
50
60
70
80
90
Incoming Defect Rate (New, Resolved, Verified defects only)
S1-S3 S1-S5

19. 19
19 QUALITY. PRODUCTIVITY. INNOVATION.
• The bigger the number – the better we are ?
• The component where we find most defects is the worst coded
• Metrics are there for managers to point fingers at people
• “All my filters and queries are correct but I still get very strange
output results”
• There are no "universal metrics“
Common Deceptions about Metrics

20. 20
20 QUALITY. PRODUCTIVITY. INNOVATION.
• http://www.getzephyr.com/resources/whitepapers/qa-metrics-value-
testing-metrics-within-software-development
• Useful Automated Software Testing Metrics – Research Gate “Implementing
Automated Software Testing,” by Elfriede Dustin, Thom Garrett, Bernie Gauf
• http://www.ifpug.org/Documents/Jones-
SoftwareDefectOriginsAndRemovalMethodsDraft5.pdf
• https://www.se.rit.edu Quality Management Overview – Defect Removal
Metrics
• http://www.guru99.com/software-testing-metrics-complete-tutorial.html
• Software Quality Metrics Overview – Pearson:
https://www.pearsonhighered.com/samplechapter/0201729156.pdf
• https://www.equinox.co.nz/blog/software-testing-metrics-defect-removal-
efficiency
• http://softwaretestingfundamentals.com/defect-density/
References

21. 21
21
Q & A
QUALITY. PRODUCTIVITY. INNOVATION.

22. 22
22 QUALITY. PRODUCTIVITY. INNOVATION.
• If you have any
questions,
please contact
the presenter:
Thank You!
Aleksandra Petrovic
Test Consultant
aleksandra.petrovic@endava.com