Test case potency assessment is primarily a diagnostic that assess the quality of test cases. This is one of the applications of HBT (Hypothesis Based Testing). This is boutique service offering from STAG Software Private Limited.

1. Test Case Potency Assessment
(TCPA)

2. Introduction
Test case potency assessment is primarily a diagnostic that that assess the quality of test cases.
This is one of the applications of HBT (Hypothesis Based Testing).
Typically the quality of the test cases i.e. it ability to uncover defects that matter is understood
as being heavily dependent on experience. And therefore assessment of the same is seen as one that
can be done by a person with deep domain knowledge.
No doubt domain knowledge is useful the act of assessment in TCPA is done using test case
architecture of HBT as the basis and therefore done scientifically.
The key objectives of this
1. Are the right types of defects being targeted to be uncovered?
2. Are the test cases covering the functional and non-functional aspects of the system?
3. Are the designed test cases sufficient?
4. Is the mix of test cases of +ve/-ve good enough?
5. Is the baseline for test complete?
The assessment is done by examining the test case document(s) and test strategy statically after
understanding the system.
© 2012. STAG Software Private Limited. All rights reserved. 2

3. Value Proposition
TCPA is intended to assess the quality of test scenarios/cases and help in improving the coverage of
the same.
The key value proposition is to ensure that the test scenarios/cases can indeed cover the system
under test well thereby ensuring that release quality of the system is indeed good.
© 2012. STAG Software Private Limited. All rights reserved. 3

4. Application
Potency Assessment 1
... is done by “Area” Requirements
‣ using a test case architecture that traceability
“External area
‣ characterises properties of test cases and
that I am
‣ examine/assess potency related properties Entity covering”
e t
t arg Code coverage
to
e re “Internal area
Wh
“Drug”
that I am
covering”
Test Cases Potency Wh
o t “Bug”
o targ
et?
Fault Coverage
Countability PDT “What PDTs
“Proving sufficiency of test cases” Potential Defect Type are uncovered by
the test cases”
Conformance:Robustness
“Distribution of +ve/-ve test cases” Test case
immunity
Level-isation Immunity
“No defect yield
“Optimal targeting” Resistant to bugs
from test cases”
i.e. hardened entities
© 2012. STAG Software Private Limited. All rights reserved. 4

5. HBT Test Architecture
Organised by Cleanliness levels
sub-ordered by items (features/modules..),
Level segregated by type,
ranked by importance/priority,
Item sub-divided into conformance(+) and robustness(-),
classified by early (smoke)/late-stage evaluation,
Type tagged by evaluation frequency,
linked by optimal execution order,
classified by execution mode (manual/automated)
Priority
Focus
Stage
The external form and internal structure of test cases
are seen as key to ensuring that the test cases are
Frequency complete and effective.
Order
Examining the organisation of test cases allows one to
uncover “holes” in the test cases.
Mode
© 2012. STAG Software Private Limited. All rights reserved. 5

6. Brief Flow of TCPA
2. Setup test baseline 3. Identify attributes that are
1. Understand SUT
i.e “what to test” key
4. Check if appropriate
5. Hypothesise PDTs and 6. Check if the baseline used
types of test have been
types of test of interest for strategy is compete
considered
7. Check if test cases exist for 8. Analyse test case count for
9. Analyse +ve:-ve distribution
these PDTs completeness
10. Document gaps in the test 12. Explain gaps using the
11. Produce TCPA report
scenarios/cases TCPA report
© 2012. STAG Software Private Limited. All rights reserved. 6

7. Note
1. Since TCPA is assessment based on documented test cases, this cannot be done for system that is
tested using no/very minimal test documentation.
2. Also any defect information is welcome as this would accelerate the PDT list creation. This can
include customer defects and those found the vendor’s test team.
3. Note that in the case of a large system, this may be done by identifying similar components and
assessing only sample of similar features/requirements to get a good indicative view of areas of
improvement.
© 2012. STAG Software Private Limited. All rights reserved. 7

8. Inputs/Deliverables
Test Strategy
Test Scenarios/Cases
Potency Assessment Report
TCPA This highlights the gaps in the
test scenarios/cases discovered by TCPA.
Defects found by
QA team
Defects reported by
customer
© 2012. STAG Software Private Limited. All rights reserved. 8