1. OAK~TQ Seminars 2008
Celebrating 10 years of Quality Life
Test Defect Metrics and Analysis
Pradeep P. Oak
Bangalore
[January 24, 2008]

2. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/2
“Works are always
associated with the
defects ..
just as the fire is
enveloped with
smoke”
Bhagavad-Gita Ch. XVIII, 48

3. 2989/1B, 12th
Main, 80 Feet Road,
HAL II Stage,
BANGALORE – 560 008
http://www.oaksys.net/

4. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/4
Who we are
• Specialist Independent Software
QA/Testing company
• Established in 1998
• ISO 9001:2000 certified since 2003
• Over 150 Test specialists
• Multiple domain expertise
• Offices in Bangalore, Pune
• Operations in India, USA, Europe

5. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/5

6. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/6
Key Members
• Mr. Pradeep Oak - M Tech., IIT Kanpur
– worked with TCS, Index, …
– consulted with HPISO, Philips, DRDO, Satyam, …
• Mr. Narendranath Pai - B Tech
– expertise in Business & Web Applications
– worked with ANZIT, SAP Labs, PSI, …
• Ms Tejaswini - B Tech
– expertise in Embedded, Real Time & Telecom projects
– consulted with DRDO, Motorola, …
• Ms Asmita Oak - B Tech
– expertise in Quality Management Systems
– worked on V & V projects at DRDO, Synopsys, …
AND .. 150+ others

7. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/7
Services Offered
Managed Testing Service
Software Test Planning & Strategy
development
QA for the complete SDLC
Test case design and driver development
Test Environment setup & Test Execution
Development of Test Automation Framework
User Acceptance Testing
In short, everything in Software Testing

8. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/8
What makes us different?
• Flexible approach
To us, each client is unique. So is each client solution.
• Adaptable delivery
We know that the key to a successful project is the ability to adapt. If
anything is likely to have an impact on a project, we identify and
communicate the issue quickly and take a constructive approach to
problem resolution.
• Attention to detail
• Superior technical ability
• Team-based delivery
We don’t supply individuals - we supply the knowledge and backup of a
specialist organisation. We have the depth to offer a full range of specialist
skills, ranging from project management to automation scripting, focused on
your requirements.
• Long term relationships, not "Touch and Go"

9. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/9
Our research initiatives
• OAK-BVB Center of Excellence in V&V (2003)
• In association with BVB college of Engineering, Hubli
• Focusing on research and propoagation of new techniques in software
Verification & Validation, deriving new methods of analysing test
metrics, ..
• OAK-PDA Center of Excellence in Software
Automation (2005)
• In association with PDA College of Engineering, Gulbarga
• Focusing on test automation, tool evaluation, productivity
improvement etc.

10. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/10
Projects’ Snapshot ..
Testing & Test Automation in Insurance, Banking, CRM domains
System Testing, Automation and UAT of products under Insurance domain
• Cross-platform Testing of a Network Management Product
• Testing WAP based internet software in Medical services domain
Testing and reverse engineering of a B2B software (Automotives)
Test design, development and execution of ISDN v110 protocol
Internet/Intranet based Pension administration software
System Testing of Internet Payment Security System
Test Automation of Telecom applications
IV & V of of Mission Critical Avionics Software (LCA and others) under 2167A,
DO-178B, IEEE 12207, …
Test automation of telecom data validation system
Requirement analysis of Avionics system using formal methods (ESTEREL)
UAT of HRMIS product
Managed Testing of a Global Core Banking Application with CMMi L5 processes

11. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/11
Partial Client List• Aeronautical Development Agency, Min. of Defence, Govt. of India
• ANZ Information Technology Ltd.
• CEMILAC (Centre for Military Airworthiness & Certification)
• CGI India
• CyberCash India
• Dell International Services Ltd
• Electronics & Radar Development Establishment (LRDE)
• First Apex Technologies
• GE India Technology Center (GEITC)
• IBM Global Services India Pvt. Ltd.
• iFlex Solutions Ltd
• India Life Hewitt Ltd. (leading Pension consulting ASP)
• Infosys Technologies Limited
• Price Waterhouse Coopers (PwC)
• SupportSoft
• Symantec (Veritas) India
• Yodlee Inc.

12. OAK~TQ Seminars 2008
Celebrating 10 years of Quality Life
Test Defect Metrics and Analysis
Pradeep P. Oak
Bangalore
[January 24, 2008]

13. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/13
Seminar Agenda
• Tester – y’day – today - tomorrow
• Defect metrics & Analysis
Discussions & experience-sharing of all
• Questions & Comments

14. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/14
Where are we in Testing
• 1980 .. Nothing called s/w tester
• 1990 – QC/QA sneaks in
• Late 90’s – Tester as a role
• 2000’s – Independent testing
• 2005 – Testing as a career option
• many conferences, seminars,
trainings, symposia, ...
• …
• 2010 - ????
. . . .

15. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/15
Industry Expectations
In the last few years .. We have seen lot of
changes ….Expectations are building
up
• Need the right attitude
• Act as Virtual Users
• Job is quite demanding as the profile is
yet to get defined
. . . .

16. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/16
Tester is required to ..
• Find all defects
• Be responsible to own all the
onsite bugs
• Know programming very well
• Be a domain expert
• Be analytical
• Be flexible
• Be available for a 9-6 job
(9 PM to 6 AM !!)
• ….

17. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/17
Can we be a Thinking Tester ?
Goals
• Be an efficient tester/test group
• Improve effectiveness of testing
• Do complete testing [goal accomplishment]
• Be predictable [repeatable]
Let us try to discuss these thru sample metrics

18. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/18
Goal – Improve Test efficiency
Question: What is Test efficiency ?
• Faster
• Cheaper

19. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/19
Test Efficiency
Cost of finding a defect
• No. of TCs designed v/s effort spent
• No. of TCs executed v/s effort spent
• No. of TCs automated v/s effort spent
• Test effort v/s Devp effort
• Waiting time v/s Test time

20. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/20
Cost of a defect
• Cost = effort spent/defects found
• Avg. Effort per defect per project is plotted
Analysis..
Cost of a defect
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
PersonHours

21. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/21
Cost of a defect - Analysis
Why some projects have high costs of testing
• Is it because of good quality of development?
• Is it ineffective testing / inefficient testing ?
• Is the product complex ?
• Is it new technology ?
• Is it new domain / platform ?
• Is the team new ?
• Is a different process followed / process not followed

22. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/22
Test Effort distribution
0
5
10
15
20
25
30
35
Test
Strategy
STP ITP UTP UT IT ST
Testing Activity
%TestEffort
P1
P2
P3
Analysis..

23. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/23
Test Effort distribution - Analysis
• Is there a relationship between planning & execution ?
• Is more planning leading to reduced execution time
(telescopic)?
STP v/s ST
ITP v/s IT
UTP v/s UT
Related Question: Is there a relationship between test effort &
test effectiveness

24. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/24
Cost v/s Value of a Test
• What is the COST of each test ?
• What is the VALUE of each test ?

25. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/25
Cost v/s Value of a Test
• When you test what you gain ?
• When you do not test, what you save/gain ?
• Cost benefit analysis or ROI
– Tangible benefits
– Intangible benefits –
• increased confidence, better image
Why not .. Value of Quality (VoQ) instead of CoQ !!

26. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/26
Goal – Improve effectiveness
Question: What is Test effectiveness ?
• Finding all defects [till current stage]
• No defects reported in next stages
• Finding critical defects more
• Presenting Testing work well to others

27. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/27
Goal – Improve effectiveness
What are the metrics ?
• Defect leakage
• Defect distribution
• Defect prediction
• Defect acceptability by affected teams
• Feedback / Survey
• …..

28. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/28
Defect Leakage Matrix (sample)
SRS Dsgn CUT IT ST UAT Total
SRS 23 5 1 8 2 39
Dsgn -- 56 7 1 64
CUT -- -- 234 6 34 4 278
IT -- -- -- 32 3 2 37
ST -- -- -- -- 128 26 154
UAT -- -- -- -- -- 11 11
Total 23 61 242 39 173 45 583
Defectinduction
 Defect detection (reviews/testing)  Analysis..

29. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/29
Defect Leakage Matrix - Analysis
• Ideal matrix should have values only on the diagonal cells
• Any values above the diagonal indicate defect leakage
Your Comments on defect leakage please ?? ..
• How is the testing process ? UAT / ST / IT / UT

30. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/30
Defect classification
Defects by Severity
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
Showstopper High Medium Low Misc
Severity
%defects
Analysis..

31. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/31
Defect classification - Analysis
Are important defects being identified ?
It is also important to look at ..
Who is assigning severity ?
Important in whose eyes ?
Are defect classification guidelines defined and agreed

32. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/32
Defect Prediction Model
Defect distribution across SDLC
11
13.5
16.5
17.25
10.5
21
10.25
7
17
0
5
10
15
20
25
SRS Design Coding UT IT ST UAT
Phase
%Defects
Avg %
No. of defects NOW
U R
here
UAT
Analysis..

33. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/33
Defect Prediction Model - Analysis
From past experience, 10% defects got skipped to UAT
Suppose, We have committed to deliver with less than 10 defects
at UAT.
And, found 7 defects at SRS & 17 defects at Design
• What do we infer ?
• What is expected at UAT ?
• Quality of SRS development v/s quality of SRS review
• Quality of Design v/s quality of design review
Suppose, Till the time of delivery, we had already found 400
defects.
• What do we predict at UAT ?

34. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/34
Defect Acceptability
0
50
100
150
200
250
#ofdefects
Code Rev UT IT ST
Phase
TotalDefects
Accepted
Rejected
Withdrawn
Analysis..

35. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/35
Defect Acceptability - Analysis
Defects accepted
Defects rejected
Probable Defects withdrawn
• Comments please ..
• How early defects are reported ? Is there enough time to
fix them?
• Tester’s Image & Pride are at stake

36. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/36
Goal –Test completeness
Question: What is Test completeness ?
• Functional & nonFunctional
• Against standards / objectives

37. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/37
Goal Accomplishment (FURPS+)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Functionality
Usability
Reliability
Perform
ance
SecurityM
aintainabilitySupportability
TC not Appl
TC not Exec
TC Failed
TC Passed
Analysis..

38. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/38
Goal Accomplishment - Analysis
Test cases failed & not executed are important
to look at
Is there a need to replan ?

39. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/39
Completeness / Coverage
Feature coverage
Code coverage – statement, condition, loop, ..
Requirements coverage
Standards/protocol coverage

40. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/40
Goal – Create right visibility
• Planned presentations to management
– what
– How often
• Conduct survey among affected groups
• Collect feedback – internal / external

41. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/41
Justify Ourselves
Developers feel-
“ Testers write lot of tests, but it is a wasted
effort. Real bugs are not uncovered.”
Is this true?
– May be!
– May be not!!
How do we justify?
Measurement - Metrics - Analysis

42. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/42
Testers must …
• Know what they are doing
• Have pride in their work
• Present their work to the right people
• Plan, Measure, Analyse, Refine, ….
Industry responds with rewards
It is an opportunity to be a leader

43. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/43
Going Forward ...
• Industry will learn & mature
• Demand & value of Testing is
bound to increase
• Will have matured models to support
• Focus may then shift more
towards preventive from
corrective

44. Thank You
Your feedback is valuable to us
- We can be contacted at -
OAK SYSTEMS PVT. LTD.
http://www.oaksys.net/
oakTQ@oaksys.net
Phone- (91)-(80)-4126 7681

45. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/45
Q & CQuestions and Comments

46. OAK~TQ Seminars 2008
Celebrating 10 years of Quality Life
Test Defect Metrics and Analysis
Pradeep P. Oak
Bangalore
[January 24, 2008]