Testers are often evaluated by metrics that don’t really quantify the value of their work. Metrics such as tests planned, tests executed, coverage achieved, and defects reported all measure effort rather than results. Since people generally want to meet metrics goals, measurements that focus on activity rather than effectiveness often encourage unintended behaviors. Since the true value of testers lies in their ability to analyze and communicate risks and impacts, we must change the focus of metrics from numbers to insights. Scott Barber shares what stakeholders are really looking for when they request specific metrics, how the metrics they request frequently fail, and how to help your organization create metrics that do provide real insight. Discover the tools you need to explain what can be measured, what those measurements mean, and how to combine measurements into metrics that tell insightful stories about your testing.

1.
nt Session
Presented by:
Scott
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073
888‐2
W8
Concurre
4/9/2014
12:45 PM
“How Metrics Programs Can Destroy Your Soul”
Barber
SmartBear
300,
68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Scott Barber
SmartBear
Chief performance evangelist for SmartBear Scott Barber is a respected leader in the
advancement of software testing practices, an industry activist, and load testing
celebrity of sorts. Scott authored several books―Performance Testing Guidance for
Web Applications, Beautiful Testing, How to Reduce the Cost of Testing, and Web
Load Testing for Dummies―and more than 100 articles and blog posts.
Founder/president of PerfTestPlus, Scott co-founded the WOPR, served as director of
the AST and CMG, and is a founding member of ISST. His industry writing, speaking,
and activism focus on improving the effectiveness and business alignment of software
development practices. Learn more about Scott Barber.

3. 2/25/2014
1
How Metrics Programs
Can Destroy Your Soul
Created for:
How Metrics Programs
Can Destroy Your Soul
By:
Scott Barber
Product Owner – Load Testing Tools
SmartBear Software, Inc.

4. 2/25/2014
2
Product Owner – Load Testing Tools
SmartBear Software, Inc.
scott barber@smartbear comscott.barber@smartbear.com
@sbarber
Co‐Founder: Workshop On Performance and Reliability
www.performance‐workshop.org
Co‐Author:Author: Contributing Author:
Books: www.perftestplus.com/pubs
About me: about.me/scott.barber
Have valid needs to:
• Assess & compare performance & quality of:
di id l
Managers & Execs
• Individuals
• Teams
• Projects
• Products
• Compute costs
• Identify risks & trends

5. 2/25/2014
3
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i• Quantitative Metrics
• Scalable
• Transferable
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Tied to individuals
C ’ b i d• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i
• Can’t be trained
• Predicated on trust
(not data)
• Virtually impossible to
defend
• Generally reserved for
“hands on” managers
• Quantitative Metrics
• Scalable
• Transferable
hands‐on managers
with one project or
team

6. 2/25/2014
4
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Very difficult to do
ll• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i
well
• Expensive
• Time consuming
• Not good for detailed
comparisons
• Not good for
estimation
• Quantitative Metrics
• Scalable
• Transferable
estimation
• Few managers/execs
trained in using them
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Scale well
P id i li i &• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i
• Provide simplicity &
consistency
• Easy to use
• Good for detailed
comparisons
• Good for estimation
• Good for trends
• Quantitative Metrics
• Scalable
• Transferable
• Good for trends
• Most managers/Execs
trained
• Validated in many
areas over long
periods of time

7. 2/25/2014
5
Are based on measurements that are:
• Verifiable
St d di d
Effective Metrics
• Standardized
Are explicitly linked to goals
• Correlated/causal
• Compliance
F t d b l tFavor trends over absolutes
Evolve
Are not used in isolation
“Most metrics are created, collected, and reported to satisfy a
leader’s request. The leader’s role is to supply clarity and direction
by providing the proper questions. Middle management’s role is to
th ti M t i ff f idi th
What is a Metric?
answer the questions. Metrics offer a means of providing the
answers so that all involved can have faith in them.
Unfortunately, leaders often don’t know exactly what they want.
Chances are you have played the Guessing Game with a leader,
where the data you provided wasn’t what he needed, so he asked
for different data, figuring he would know the right data when he
saw it Despite repeated failures you continued to chase data as ifsaw it. Despite repeated failures, you continued to chase data as if
all the effort invested in collecting the wrong data would eventually
prove worth your perseverance.”
From: “Do-It-Yourself Metrics”
by Martin Klubeck, Michael Langthorne, and Don Padgett
http://www.educause.edu/ero/article/do-it-yourself-metrics

8. 2/25/2014
6
Provide insight into the health of:
• Individuals
• Teams
Metrics Should
• Teams
• Projects
• Products
Help to:
• Assess efficiency & progress
• Inform business decisions
• Enhance communication
• Maintain focus
However;
Even in well studied &
historically validated areas,
quantitative metrics are…quantitative metrics are…

9. 2/25/2014
7
Quantitative Metrics:
• Obscure Details
ll & i
Far From Perfect
• Roll‐ups & summaries
• Focus on “big picture”
• Risk Measurement Dysfunction
• People tend to optimize metrics
• Metrics optimization invites “bad stuff”
• Are “invitations to conversations”
• Don’t tell the whole story in isolation
• Can be wildly misleading
… and…
Common quantitativeCommon quantitative
management metrics
models are not a good fit
for software development

10. 2/25/2014
8
… and…
Within software
development, testing is the
*worst* fit for common
quantitative managementquantitative management
metrics models
Inconsistent Units:
• Test/Test Cases
Why?
• Size/Importance of defects
Unknown Unknowns:
• How many defects are there?
• How many tests would it take to…?
Quality is Subjective:
• How good is “good enough”?
• What do Pass/Fail *mean*?

11. 2/25/2014
9
Test/Test Cases Planned?
Test/Test Cases Executed?
These metrics indicate… ?
Defects Found?
Defect Removal Rate?
Coverage?
Test/Test Cases Planned?
Test/Test Cases Executed?
These metrics encourage… ?
Defects Found?
Defect Removal Rate?
Coverage?

12. 2/25/2014
10
Communicate impacts to revenue:
• Help the CEO get a bigger yacht, sooner.
What do we *want* to do?
Identify risks to the business:
• Generally revenue centric.
Identify risks to the buyers/users:
• If they don’t want to buy, we lose.
Streamline Development:
• Get to “good enough” faster.
Do your test metrics help with…
• Risks to revenue?
Test Metrics Effectiveness
• Risks to the business?
• Risks to the buyers/users?
• Increasing delivery velocity?

13. 2/25/2014
11
Change Your Perspective
Risk as a common language
Risk
Security
FunctionalCompliance
Whether explicitly or
implicitly, all forms
of testing revolve
around the
reduction and
PerformanceUsability
reduction and
management of
risk.
© 2011 PerfTestPlus, Inc. All rights reserved.
The Secret to Managing Risk
To effectively
manage risk, you
must effectively
manage knowledge.
© 2011 PerfTestPlus, Inc. All rights reserved.

14. 2/25/2014
12
Control Model Testing
Control Model Testing is
a business-a business
aligned approach to
software testing that
derives “test cases” from
knowledge
models of the system
© 2011 PerfTestPlus, Inc. All rights reserved.
based on a risk-based
taxonomy.
Basic Entities
© 2011 PerfTestPlus, Inc. All rights reserved.

15. 2/25/2014
13
Risk Assessment Framework
The Open Group (http://www3.opengroup.org/):
Risk Taxonomy Technical Standard - https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156
Testing For Threats & Risk
Controls prevent or
mitigate risk whichmitigate risk which
may impact business
objectives.
Control Model
Testing helps identify
© 2011 PerfTestPlus, Inc. All rights reserved.
Testing helps identify
and assess these
controls.

16. 2/25/2014
14
Possibly Better Test Metrics
Risks/Threats Identified?
Risk/Threat Tests Planned/Executed?
Risks/Threats Dismissed?
Risks/Threats Uncontrolled/Unmitigated?
Risks/Threats Controlled/Mitigated?
Questions?

17. 2/25/2014
15
Contact Info
about me/scott barberabout.me/scott.barber
Product Owner – Load Testing Tools
SmartBear Softwear, Inc.
E-mail:
Scott barber@smartbear com
Web Site:
SmartBear.comScott.barber@smartbear.com
Blog:
scott-barber.blogspot.com
SmartBear.com
Twitter:
@sbarber