The document presents key performance indicators (KPIs) that should be used to drive an efficient test organization. It recommends starting by defining goals before selecting KPIs to measure progress towards those goals. Both qualitative and quantitative KPIs are important. Specific KPIs discussed include operational costs, overhead costs, environment utilization, fixed issues, issue leakage, stakeholder satisfaction, leadership, employee satisfaction, and innovation. Target levels should be set for each KPI and deviations analyzed to understand causes and potential corrective actions. Simplicity is key - measures should focus on added value rather than internal optimization.

1. Critical KPI for an Efficient Test Organization
Simplicity adds value

2. Purpose
• The purpose of this presentation is to present a number of KPI that should
be used to drive an efficient test organization
• If test and development are integrated in the same organization, then this
setup is not applicable, even though it could inspire to do some similar
measurements in that organization

3. What is a KPI?
• “Measures that help decision makers define and measure progress toward
business goals. KPI metrics translate complex measures into a simple indicator
that allows decision makers to assess the current situation and act quickly.” [2]

4. Start with Why, not What or How!
• Remember to start by asking the question:
“Why do I need to do measurements?”
• Then continue with:
“What do I need to measure to mitigate those reasons?”
• And finally:
“How do I measure that?”
• Don’t start measuring something just because you can
• And don’t ask for measurements without thinking about the goal of those
measurements first

5. Measuring the Right Thing
• “In business, organizations should strive to create metrics that encourage
behaviors that benefit the enterprise as a whole. These metrics and their
reporting should provide a guiding light toward achieving the “3 Rs” of business—
everyone doing the right things, and doing them right, at the right time. “ [3]

6. Occam’s Razor
• By definition, all assumptions introduce possibilities for error; if an assumption
does not improve the accuracy of a measurement, its only effect is to increase the
probability that the overall measurement is wrong. [4]
• “… most systems work best if they are kept simple rather than made complex;
therefore simplicity should be a key goal in design and unnecessary complexity
should be avoided.” [5]

7. Simplicity not Complexity
• Because assumptions decrease KPI accuracy it is often better
to create a simple KPI that correlates well enough with the
objective of the measurement than to try and create the
perfect KPI
• Complex KPI can also create the illusion of validity because
many of the factors in the KPI are highly correlated with
each other and a number of highly correlated inputs should
not increase confidence much more than only one of the
inputs [10]

8. Qualitative & Quantitative KPI
• “Qualitative properties are properties that are observed and can generally not be
measured with a numerical result. They are contrasted to quantitative properties
which have numerical characteristics.” [1]

9. Critical KPI for an Efficient Test Organization
Cost Value Organization

10. Cost: Operational Costs
• Quantitative KPI
• Frequency: Monthly
• For a commercial enterprise, operating costs fall into two broad categories:
– fixed costs, which are the same whether the operation is closed or running at
100% capacity
– variable costs, which may increase depending on whether more production is
done, and how it is done (producing 100 items of product might require 10
days of normal time or take 7 days if overtime is used. It may be more or less
expensive to use overtime production depending on whether faster
production means the product can be more profitable). [6]

11. Cost: Overhead Costs
• Quantitative KPI
• Frequency: Annually
• Overhead costs for a business are the cost of resources used by an organization
just to maintain its existence. Overhead costs are usually measured in monetary
terms, but non-monetary overhead is possible in the form of time required to
accomplish tasks.
• Examples of overhead costs include:
– payment of rent on the office space a business occupies
– cost of electricity for the office lights
– some office personnel wages [5]

12. Cost: Environment Utilization
• Quantitative KPI
• Frequency: Quarterly
• Measures utilization of tools and environments – a low value indicates that many
tools are not used that often
• Low utilization may indicate that tools should instead be rented for the short
period of time they are used, or could perhaps be used by someone else as
well, since the tools are available
• Low utilization may be necessary in some organizations, but should always be
considered a waste

13. Efficiency: Fixed Issues
• Quantitative KPI
• Frequency: Monthly
• Measures the number of issues found by the test organization that are actually
fixed
• Issues that are not fixed might have informational value and will then be captured
in the Stakeholder Satisfaction KPI, but otherwise hold no actual business value
• It could also be good to measure Fixed Issues / Submitted Issues, to see how many
of all the issues submitted that are fixed, since there are costs associated with
each submitted issue in form of reporting and analysis
• It is possible to translate the number of fixed issues to a monetary value in form of
return rates and loss of sales that would have occurred because of those
issues, but it requires some effort

14. Efficiency: Issue Leakage
• Quantitative KPI
• Frequency: Monthly
• Measures how many issues that were not found during testing and thus unknown
to the product project, that was later found by customers
• If it is reported by customers it is important to the customers and should have
been found during testing
• It is possible to translate the number of leaked issues to a monetary value in form
of return rates and loss of sales that occurred because of those issues, but it
requires some effort

15. Efficiency: Stakeholder Satisfaction
• Qualitative KPI
• Frequency: Quarterly
• How good is the test organization at providing the correct decision material and
information to it’s stakeholders?
• This should be measured by a survey that goes out to all stakeholders
• If the stakeholders perceive that the test organization provides value to the
company by the reports and decision material generated this KPI will indicate that
• This is harder to quantify in monetary terms

16. Organization: Leadership
• Qualitative KPI
• Frequency: Annually
• There are many ways to measure leadership qualities in an organization, more or
less complex
• One way is to conduct 360 degree assessments with the employee’s
subordinates, peers, and supervisor(s), as well as a self-evaluation [7]

17. Organization: Employee Satisfaction
• Qualitative KPI
• Frequency: Annually
• Affective job satisfaction is the extent of pleasurable emotional feelings individuals
have about their jobs overall, and is different to cognitive job satisfaction which is
the extent of individuals’ satisfaction with particular facets of their jobs, such as
pay, pension arrangements, working hours, and numerous other aspects of their
jobs. [8]
• While cognitive job satisfaction might help to bring about affective job
satisfaction, the two constructs are distinct, not necessarily directly related, and
have different antecedents and consequences. [8]
• One way to measure this is to let all employees answer a suitable survey

18. Organization: Innovation
• Quantitative KPI
• Frequency: Quarterly
• This measurement should cover both intellectual property and innovation in
general
• You could focus on patent applications and invention disclosures …
• But 90 to 95 percent of all patents lack any market relevance and 99 percent fail to
bring any profit to the firm (Stevens & Burley, 1997). Still, patenting is often
necessary for many firms to keep their competitive advantage so clearly patents
indicate something about the innovativeness of a firm. [9]
• Innovation in general is more difficult, but there are a number of measurement
methods with different advantages and drawbacks [9]

19. Cost vs. Value

20. Target Levels
• Target levels for all these KPI will of course vary greatly between organizations
because of different pre-requisites
• It is important the set reasonable target levels to motivate and drive the right
behaviour
• Example: In some test organizations it may be ok to have a low number of fixed
issues, as long as the stakeholder satisfaction is high, because the objective of that
test organization is not to find bugs, but to validate and provide decision material

21. KPI Analysis
• When KPI deviate from target levels an analysis of the
situation is critical
• Sometimes it may be warranted to deviate from
previously set target level
• It is critical to NOT start corrective actions until the
situation has been analyzed properly
• The KPI value alone does not give a complete picture
– it gives indications if things are according to plan or
not

22. Summary
• Keep it simple!
• Measure added value to the company, not internal optimization and company
politics!
• Measure the right thing for the right reasons!
• Know when to use qualitative and quantitative measurements!

23. Reference
[1] Qualitative Property
http://en.wikipedia.org/wiki/Qualitative_property
[2] Defining KPI
http://www.kaizen-analytics.com/2008/11/defining-actionable-business-driven.html
[3] Measuring the Right Thing
http://www.qualitydigest.com/inside/quality-insider-article/are-your-business-metrics-measuring-right-thing.html#
[4] Occam’s Razor
http://en.wikipedia.org/wiki/Occam%27s_razor#Probability_theory_and_statistics
[5] KISS
http://en.wikipedia.org/wiki/KISS_principle
[6] Operating Costs
http://en.wikipedia.org/wiki/Operating_cost
[7] 360 degree feedback
http://en.wikipedia.org/wiki/360-degree_feedback
[8] Job Satisfaction
http://en.wikipedia.org/wiki/Job_satisfaction
[9] Measuring Innovation
http://www.innovationmanagement.se/2013/02/15/measuring-innovation-part-1-frequently-used-indicators/
[10] Illusion of Validity
http://en.wikipedia.org/wiki/Illusion_of_validity