TEA - Test Escape Analysis

1. Ian McDonald
Test Escape Analysis - Presentation
© 2010 Ian McDonald
1

2. Cost of Defects Fixing
 It can take one person 10 minutes to find a
defect in requirements, 10 minutes to fix the
document. Cost £10 per defect.
 Finding the same defect in Functional Test
and then Fixing. Cost £211 per defect.
 Finding the same single defect in System
Integration Testing. Cost £587 per defect.
 Cost following delivery £? + £Reputation.
2

3. Add Success to Testing
 There are usually quick wins, however
do we review the success of our testing?
3

4. Quick Wins
 How many defects could have been
avoided by changing the review
practices and acceptance of
Requirements into Development?
 Measure by Audit of 1 product:
1. Audit Defect Logs
2. Audit Customer Service Reports.
3. Audit Requirements and their Review.
4

5. RCA vs TEA
 Root Cause Analysis (RCA) allows us to
learn, rectify and avoid future defect
injection. Providing we use the data.
 Test Escape Analysis (TEA) in contrast
allows us to become more efficient at
testing and avoid future defects from
escaping detection. Provided we use
the data.
5

6. TEA Applied to Defect History
 Applying TEA to defect history can help
to analyse where resources and
changes in procedure should be applied
for maximum return on investment.
 Normally one would focus upon:
 a single typical representative project.
 High and Critical defects, to reduce sample size.
 TEA can be integrated into the defect
tracking system.
6

7. Purpose of TEA
 Root Cause Analysis is aimed at the code
programming – why was the defect
introduced.
 Analogy why did the tightrope walker fall
 TEA in contrast is about why a defect was not
detected.
 Analogy why did we not catch the falling tightrope
walker.
 We do TEA to learn how to detect more
defects and how to detect them sooner and
smarter. The ultimate goal is to drive down
the number of defects leaking to the customer
by finding those defects ourselves first.
7

8. Benefits
 The benefits of early detection are:
 Reduced costs.
 Improved reputation.
 Reduced loading on engineers.
 Improved output from testing team.
8

9. Trends
 With TEA we are interested in trends.
 Is there a pattern in the defects that are being let through?
 Is there a type of test that could help catch more defects.
 We target TEA at a sample of defects. This sample
includes:
 Defects raised for example by the Customer Which are either
 Critical or
 High
 The knowledge that we gain from TEA is applied to future
Test Plans:
 Trend Reports in product TEA - produced by the QA Engineers.
 Product Teams are expected to produce their own reports on their
individual trends and apply lessons learned to their Test Plans.
9

10. Resources
 TEA Meetings are linked to Company Targets
 Are to occur at least monthly for each team.
 Expectations
 For this activity teams usually need a developer and
tester to attend the monthly meeting.
 In practice a TEA meeting usually takes an hour per
month.
 In looking at a TEA report one may initially spend
around 15 minutes per record, however it is possible
to do these in just 5 minutes each.
 Remember we are looking at a sample – not all.
10

11. Managing TEA
 There are 2 ways that TEA
meetings run. Either:
 Hold a meeting at which the TEA
reports are completed.
 Complete the reports before the
meeting and at the meeting these are
audited.
 The TEA meetings ensure that there
is a consistent approach and it also
acts as a mentoring exercise in
supporting the rollout of TEA.
11

12. TEA Assumptions
 In carrying out TEA there is an assumption that:
 The review team have some minimal familiarity with
the code impacted by the defect.
 The review team are familiar with testing techniques.
This can be obtained by either:
 Reading BS7925-2;
 Laying on an internal course 2 or 3 days;
 Attending an ISEB/IQTB Test Foundation Course.
12

13. Management of TEA and RCA
 Root Cause Analysis if employed
improves development through the
application of lessons learned. RCA is
usually managed through the defect
tracking tool employed.
 TEA in contrast improves testing.
However it to can be managed through
the defect life cycle.
13

14. Summary of TEA Data Fields
 The following Fields are used within TEA:
 TEA Status
 Reason Introduced
 Development Phase
 Code Designed From
 Test Environment
 Test Tools Required
 Test Technique
 Test Type
 Free Text Notes Field
 All fields are mandatory, but there is a “not
known” option available.
14

15. Reason Introduced
 A shared field with the main Defect and RCA. The record can be changed
from any of the reports.
 Fields include:
 Code Missing
 Code not fully built or tested Stage Defect Introduced:
 Coding Error “Earliest stage in the processes
 Design Incorrect that the defect could have been
 Design Missing or Incomplete prevented”.
 Design Unclear Reason Introduced:
 Environment Unavailable “Provide finer detail as to the initial
root cause of the defect”.
 Initial Fix Incomplete
 Initial Fix Incorrect
 Requirement Incorrect
 Requirement Missing or Incomplete
 Requirement Unclear
 Standards Not Followed
 Typing Mistake
 Other (free text)
15

16. Development Phase
 Requirements Review,
 Design Review, What was the appropriate appraisal
process (testing or reviews) that
 Code Review, would have detected this defect
earlier?
 Unit Test,
 Component Test,
 Component Integration Test,
 System Test
16

17. Code Designed From
 API Spec,
What was the source used,
 Detail Design, from which the code with the
defect was created?
 Requirement,
 Standards,
 Functional Specification,
 Developer Led Requirements,
 Other (Selecting Other allows a free text
field to be used).
17

18. Test Environment
 Sufficient,
 Customer Device,
 External Server,
 System Component,
 Stub,
 HRP (Hardware Reference Platform) Feature,
 OS / HRP Config,
 Platform, What other Test Environment would have
helped to detect the bug earlier? (other than
 External Device, what was planned or available)
 Live Environment,
 Other (Selecting Other allows a free text field to be
used).
18

19. Test Tools Required
What other Test tools would have helped to
 Sufficient, detect the bug earlier? (other than what was
planned or available)
 Test Execute Framework Extension, Test
Driver Extension,
 Code Analyser,
 Protocol Tester,
 Other (Selecting Other allows a free text
field to be used).
19

20. Test Technique
 Boundary Analysis (See BS7925-2)
 Cause Effect Graphing (See BS7925-2)
 Classification Tree
 Equivalence Partitioning (See BS7925-2)
 Out Of Memory
 Random Testing
 Risked Based Targeted Exploratory Testing
 State Transition Testing (See BS7925-2)
 Static Analysis (Reviews and the use of Static
Analysis Tools)
20

21. Test Type
 API Validation  Test In Real Environment
 Binary Compatibility testing  Test on Emulator
 Compliance Testing
 Dependency Testing
 Test Tools – Code Coverage
 Design Verification  Test Tools –Static Binary
 Error Guessing Compatibility
 Exploratory Testing  Test Tools – Codenomicon
 Functional Testing  Test Tools – Copyright
 Interoperability Testing  Test Tools – Coverity
 Intrusive Testing
 Performance Testing
 Test Tools – Lint
 Platform Verification Testing  Test Tools – Win Runner
 Product Configuration Testing  Test Tools – Load Runner
 RAM / ROM Testing  Test Tools – Other
 Recovery (Robustness) Testing  Open Source Check
 Requirements Validation
 Security Vulnerability Testing
 Static Analysis
 Static Testing
 Negative Testing
21

22. Evidence from TEA
TEA can be useful for highlighting unforeseen problems. We might have
predicted that inadequate negative tests are being written (and this is
supported by TEA in this example) but there is also much evidence to
suggest that we need to write more positive test cases.
450
Test Techniques - Test Technique that would find the Defect
418
400
350
Nos. of Defects
300
244
250
200
150
96 93
100 61 56 53 46 44 37 35
50 26 17 16 15 7 5 5 2 0
0
Re st
ua s
Eq ery
lity
Re r if
a t a rt
cy
fo e r)
st
n
e
ce
ga est
BC
ns
dA M
pe ity
S e ls
pe e
gr is
M view
St a nc
Co s sio
e
A c l Te
nc
Ve
o
O
Re lys
en
Pe lia n
vP
bi
ra
De cu r
eT
Pl (o th
v
Ne e T
To
O
In epta
co
ra
rm
eT
nd
n
rm
t iv
ui
e
iv
ic
p
m
at
an
sit
rfo
c
ro
un
at
St
Po
te
Bo
22