Joel Montvelisky discusses the pros and cons of using metrics in testing, emphasizing their effectiveness in communication, clarity, and benchmarking while cautioning against the misuse and overreliance on metrics. He outlines key principles for planning and maintaining a metrics program, including identifying objectives, choosing appropriate metrics, and regularly reviewing their relevance. Tips for presenting metrics effectively highlight the importance of tailoring information for different audiences and ensuring clarity to avoid confusion.

1. Joel Montvelisky
Chief Solution Architect
Simplify Test Management
How to Avoid Cutting Yourself
with the Double Edge Sword of
Testing Metrics?

2. What do we want to cover today?
• Pros & Cons of working with Metrics
• Planning your metrics program and
maintaining it over time
• Tips and tricks for working with metrics with
internal and external Stakeholders
• Questions and Answers

3. A little about me…
 Professions :
(last 19 years)
- Tester
- QA Manager
- Blogger
- Speaker
- Solution Architect
 PractiTest:
- Chief Solution Architect
- QA Manager

4. About
 SaaS ALM Solution
 Simple & Intuitive – yet – Methodological & Flexible
 Integrates with Jira, Selenium, Jenkins, Pivotal Tracker, QTP,
Bugzilla, Ranorex, SoapUI, etc…
 Best Darn Support in the Business 

5. Show of hands
“Raise your hand”
if Your Team or
Your Company has
a Standard Set of
Metrics.

6. Show of hands No.2
“Raise your hand”
if these Metrics
have caused
more harm than help
sometime in the past…

7. Fact (?)
Metrics, just like knives
(or fire, or TNT, or salt ),
are a great tool that can also be
dangerous in the wrong hands

8. Pros & Cons of
working with Metrics

9. Pro 1: Metrics are an effective
means of communication
It is trivial to talk about metrics to communicate
status, but using metrics you can also convey to the
team what is important and where and effort is needed.
Pro 2: Metrics are (usually) clear
Good metrics are easy to understand both for the people inside
your team and outside of it

10. Pro 3: Metrics allow us to compare things
You can benchmark and compare:
- Different teams / process
- Same process in different times / version
etc.
Pro 4: Metrics help us track progress
You can see how things and change over time, make
predictions, corrections, etc.

11. Pro 5: Metrics are easy to distribute
It is easy to create a short set of metrics and send them by mail,
publish them in a portal or post them on a Kitchen Monitor.
Pro 6: People like metrics
People & even managers tend to open metrics and
review them because they are compact.

12. There are many
additional Pros…
Let’s talk a little about the
Cons of working with Metrics!

13. 1. People don’t like being measured
(Many times this is for good reasons)

14. 2. Metrics tend to present only one side
of the story

15. 3. Metrics are going to be used by
managers who…
 Think their personal experience is valid
in all situations.
 Jump to conclusions without waiting to
listen to all the information.
 Forget problems in paper are easier to
solve than those in the real world.
 Like applying quick fixes to complex issues.
 Believe what they see in the metrics can tell them all
the information they need to know.

16. Planning and Maintaining your
Metrics Program

17. The Laws of (Conservation of) Metrics
(Apologies for being such a geek…)
1. Anything that is measured will improve over time.
2. Whenever something improves, it will be at the expense
of something else that will worsen accordingly.
3. You can only measure 5 (sometimes only 3) aspects of
anything. Trying to measure more will lead to confusion
and failure.
4. Metrics will become ineffective after a number of
iterations (kind of like the pesticide paradox).

18. Planning your Metrics Program
Define what are the objectives of your metrics?
 Who is your audience?
 Your management?
 Your team?
 Your end users?
 Regulation?
 HR?
 Each of them need different information to make
different decisions.

19. Planning your Metrics Program
Make sure you have the right metrics.
 5 Why’s - to make sure you reached the core metrics
 SMART
 Simple
 Measurable
 Actionable
 Repetitive
 Timely

20. Planning your Metrics Program
Some of your most important metrics may be be
available ONLY after your project is done…
 Escaping Defects
 Rejected Defects
 Server Exception Levels
 Support calls post Release
etc.

21. 1. Metrics need to be constantly reviewed for:
- Accuracy
- Relevancy
2. As your project, technology and people change so will
your metrics.
Maintaining your Metrics Program

22. 3. You need to change your metrics once in a while, bugs
and people learn to work around them.
4. Be ready to change metrics, even
half way during your project
Maintaining your Metrics Program

23. Tips and Tricks
for working with Metrics

24. Presentation is as important as content
Have diff metrics for diff people
Tips & Tricks
Vs.

25.  Capture more data than you need today,
you may need it later.
 Provide analysis of your results with every graph
(don’t trust the people to understand what they see).
Tips & Tricks

26.  Make sure to provide
multiple angles.
 Too many metrics confuse
tell a story, but keep it
interesting.
Tips & Tricks

27.  Push metrics, don’t make people ask for them since it
won’t be relevant by the time you provide them.
Tips & Tricks

28. Questions & Answers
Developer Testing
Do’s and Don’ts

29. About
 SaaS ALM Solution
 Simple & Intuitive – yet – Methodological & Flexible
 Integrates with Jira, Selenium, Jenkins, Redmine, Bugzilla, QTP,
Ranorex, SoapUI, etc…
 Best Darn Support in the Business 

30. Thank You!
joel@practitest.com
http://qablog.practitest.com