One of the most challenging problems that test managers face involves implementing effective, meaningful, and insightful test metrics. Data and measures are the foundation of true understanding, but the misuse of metrics causes confusion, bad decisions, and demotivation. Rex Black shares how to avoid these unfortunate situations by using metrics properly as part of your test management process. How can we measure our progress in testing a project? What can metrics tell us about the quality of the product? How can we measure the quality of the test process itself? Rex answers these questions, illustrated with case studies and real-life examples. Learn how to use test case metrics, coverage metrics, and defect metrics in ways that demonstrate status, quantify effectiveness, and support smart decision making. Exercises provide immediate opportunities for you to apply the techniques to your own testing metrics. Join Rex to jump-start a new testing metrics program or gain new ideas to improve your existing one.

1. Metrics for Software Testing
Measuring Project, Process, and Product

2. Metrics for Testing: Managing with Facts
By using metrics, we can manage with
facts and reality
How can we use metrics to manage testing?
What metrics can we use to measure the test
process?
What metrics can we use to measure our
progress in testing a project?
What do metrics tell us about the quality of
the product?
Let’s see if we can find some answers…
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 2

3. Part 1: Why and How of Metrics

4. Why Metrics?
Managing with metrics—for testing or
anything—allows us to manage with
facts
Subjective, uninformed opinions are not
a sound basis for management
What sounds reasonable can be wrong
(e.g., Aristotle)
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 4

5. Why Testing Metrics?
Testing by itself has no value…but it produces
potentially valuable information
Test information must be generated and
communicated effectively to be valuable
Common communication goals
Notify (“We have 24 bugs remaining to close”)
Enlighten (“See the time lost due to reopens”)
Influence (“We propose a bug triage meeting”)
The metrics should be used as part of regular
status reports (dashboards) as well as periodic
analyses
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 5

6. How to Develop Metrics?
Identify objectives
Create questions about effectiveness,
efficiency, and elegance of achievement of
objectives
Devise a measurable metric (or find a
surrogate metric) to address the question
Set a goal for each metric
Implement improvements to enhance
achievement of objectives
Let’s look at two examples…
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 6

7. Example One
Objective: Find defects
Effectiveness question: Have we
finished finding new defects?
Metric: Track trend in defect discovery
during testing
Goal: Watch for convergence (more
later)
Improvement: Find bugs earlier
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 7

8. Example Two
Objective: Build confidence
Note that we need a surrogate metric for
confidence; one way is to measure coverage
Coverage has multiple dimensions
Effectiveness question: Do any requirements have
identified failures?
Metric: What % of requirements are completely
tested without any failures, what % requirements
have failures, what % are untested
Goal: 100% requirements tested without must-fix
failures
Improvement: Reduce % of requirements that fail
in testing
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 8

9. Good Metrics
Define a useful, pertinent, and concise
set of quality and test metrics
Avoid too large a set of metrics
Too difficult to measure later
Confusing to participants
Ensure uniform, agreed interpretations
of these metrics, to minimize disputes
and divergent opinions about measures
of outcomes, analyses, and trends
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 9

10. Tracking Metrics
Automated tool support can reduce the time required
to capture, track, analyze, report, and measure
metrics
Apply objective and subjective analysis for specific
metrics over time, especially when trends emerge
that could allow for multiple interpretations of
meaning
Be aware of and manage the tendency for people’s
interests to affect the interpretation they place on a
particular metric or measure
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 10

11. Reporting Metrics and Measurements
Reporting of metrics and measures should enlighten
management and other stakeholders, not confuse or
misdirect them
Good testing reports based on metrics should be
easily understood, not overly complex or ambiguous
Good testing reports based on metrics should help
management improve processes, guide the project to
success, and manage product quality
Metrics include snapshots of status at a moment in
time, trends emerging over time, and analysis of
causes and relationships between factors that
influence testing and quality outcomes
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 11

12. Part 1: Why and How of Metrics
Exercise

13. Develop a Metric
Pick any personal, professional, or
business objective
Pose a question about the effectiveness,
efficiency, and elegance of achievement
of this objective
Devise a measurable metric (or find a
surrogate metric) to address the
question
Set a goal for each metric
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 13

14. An Example
Objective: train for a marathon
Effectiveness question: Can I complete
the marathon?
Metric: starting with a 10 km run, run a
long run every weekend, adding 2 km
every weekend
Goal: Continue for 14 weeks until
reaching 38 km two weeks before actual
marathon
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 14

15. Part 2: Process Metrics

16. Process Metrics for Testing
Uses
To measure the effectiveness, efficiency, and
elegance of the test process
To provide the insights to guide where
process improvements should occur
Part of certain test process assessments
(e.g., using CTP framework)
Not used as frequently as should be the
case
Not typically part of any dashboard (but
can be)
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 16

17. Developing Good Process Metrics
Determine key objectives for the test
process
Typical objectives
Finding bugs, especially important bugs
Reducing quality risk
Decide which important effectiveness,
efficiency, and elegance questions apply
Create metrics that address those
questions
Set goals that reflect current or desired
capability
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 17

18. Bug Finding Effectiveness
Objective: Finding bugs
Question: What percentage of bugs found?
Metric: Defect detection percentage
Goal: A typical number is 85%, though
some teams do much better
DDP
bugs detected
bugs present
DDP ( for testing)
Metrics for Testing
test bugs
test bugs production bugs
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 18

19. Bug Finding Focus
Objective: Finding important bugs
Question: Do we find more important
bugs than less important bugs?
Metric: Relation of DDPs (see below)
Goal: Relationship shown below is
always true
DDP (all bugs) DDP (critical bugs)
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 19

20. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 20

21. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 21

22. Part 2: Process Metrics
Exercise

23. Test Process Metric
Define an objective for your test process
Define an effectiveness, efficiency, or
elegance question
Create a metric that addresses that
questions
Discuss
Measure your own test process when
you get back to your office
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 23

24. Part 3: Project Metrics

25. Project Metrics for Testing
Uses
To measure the progress of the testing effort
on a project
To provide the insights to guide where project
optimizations might be needed/useful
Part of certain test process assessments
(e.g., using CTP framework)
The most commonly used test metrics
Usually included in test dashboards (often
over-relied upon)
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 25

26. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 26

27. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 27

28. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 28

29. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 29

30. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 30

31. A Troubled Project’s Defect Metrics
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 31

32. Part 3: Project Metrics
Exercise

33. Your Current Test Project Metrics
Sketch or describe your current test project
metrics
Classify your presentation of each metric
as:
Snapshot of status (e.g., tables)
Trend chart
Analysis/causality
What do you think is missing from your
metrics?
Discuss
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 33

34. Part 4: Product Metrics

35. Product Metrics for Testing
Uses
To measure the quality and residual quality
risk in the product
To provide the insights to guide where
product improvements should occur
Good testing should measure the quality
of the system
Not always measured during testing
(using project metrics instead)
Not always on testing dashboards (but
should be)
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 35

36. Example: Reporting Residual Levels of Risk
Test execution period
Region in green represents risks for which all tests were run and passed and
no must-fix bugs were found. Region in red represents risks for which at
least one test has failed and at least one must-fix bug is known. Region in
black represents other risks, which have no known must-fix bugs but still
have tests pending to run.
www.rbcs-us.com
Metrics for Testing
Copyright (c) RBCS 2005-2012
Page 36

37. Example: Testing and Risk
Defects
Quality Risk Areas
Tests
#
%
Performance
304
27
3,843
1,512
39
Security
234
21
1,032
432
42
Functionality
224
20
4,744
2,043
43
Usability
160
14
498
318
64
Interfaces
93
8
193
153
79
Compatibility
71
6
1,787
939
53
Other
21
2
0
0
0
Total
1,107
100
12,857
5,703
44
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Planned Executed
%
Page 37

38. Use of Standards
Functionality: Suitability, accuracy,
interoperability, security, compliance
Reliability: Maturity (robustness), faulttolerance, recoverability, compliance
Usability: Understandability, learnability,
operability, attractiveness, compliance
Efficiency: Time behavior, resource
utilization, compliance
Maintainability: Analyzability,
changeability, stability, testability,
compliance
Portability: Adaptability, installability, coexistence, replaceability, compliance
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Addressed by
functional tests
Addressed by
non-functional
tests
Page 38

39. Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 39

40. Part 4: Product Metrics
Exercise

41. Your Current Product Metrics
Sketch or describe how you currently measure
the product during testing
Coverage of requirements, configurations, etc.
Residual level of risk
Confidence in the product
Sketch or describe how your organization
currently measures product quality during
actual use
How well do your pre-release metrics predict
product quality as experienced by the users
and customers?
Discuss
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 41

42. Conclusions
Metrics can be useful to measure, understand, and
manage process, project, and product attributes
Testing metrics are essential for a complete
understanding of status
Use a small number of metrics
Use surrogate metrics where necessary
Ensure consistent stakeholder understanding of
metrics
Define objectives, then questions, then metrics, then
goals
Use the goals to drive meaningful process
improvements
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 42

43. To Contact RBCS
For almost twenty years, RBCS has delivered consulting, outsourcing and training
services to organizations that want to improve their software and hardware testing.
Employing the industry’s most experienced and recognized consultants, RBCS
assesses test teams, manages testing projects, builds and improves testing groups,
and provides test experts for hundreds of clients worldwide. Ranging from
Fortune 20 companies to start-ups, RBCS clients save time and money through
improved system development, decreased failures in production, improved
corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.
Address:
RBCS, Inc.
31520 Beck Road
Bulverde, TX 78163-3911
USA
Phone:
+1 (830) 438-4830
E-mail:
info@rbcs-us.com
Web:
www.rbcs-us.com
Metrics for Testing
www.rbcs-us.com
Copyright (c) RBCS 2005-2012
Page 43