This document provides an overview of defect management processes. It discusses defining defects, defect prevention, discovery, resolution and process improvement. The key aspects covered are: - Defining goals as preventing defects, early detection, minimizing impact and process improvement. - Activities like root cause analysis, escape analysis and process metrics. - The defect lifecycle of prevention, discovery, resolution and continuous improvement. - Examples of defect analysis and status reporting including metrics like density, backlog and mean time to repair.

1. Defect Management Process
Yolanda Williams
ybowdrywilliams@gmail.com
http://www.slideshare.net/ybowdrywillia
ms
A practical guide for handling opportunities in IT
Quality Assurance

2. AGENDA
Today's session is divided into two main
Sections
2
What is a Defect
Management Process?
• Knowledge sharing
• Discussion
Defect Analysis and
recommendations
• Critical Analysis
• Issue
• Change Control
Coordination
• UAT an dry run Defects
• Recommendations
• Discussion

3. Process Improvement : What
are they thinking?
Developers
“While the defect itself may not be a big
deal, the fact that there was a defect is
a big deal.”
Testers
“If this defect could have gotten this far
into the process before it was
captured, what other defects may be
present that have not been discovered.”
Customers
“A defect is anything that causes
customer dissatisfaction, whether in the
requirements, design, code or testing or
not”
Program
Leadership‟s
advice:
Efforts should be made to analyze the
process that originated the defect to
understand what caused the defect.
3

4. Management Reporting
What Leaders need?
To make more
informed
decisions -- e.g.,
redesign of error
prone modules,
the need for
more testing,
etc.
To provide
strategic
information and
metrics to senior
management --
defect
trends, problem
systems, etc.
To provide
insight into
areas where the
process could
be improved to
either prevent
defects or
minimize their
impact.
To provide
insight into the
likelihood that
target dates and
cost estimates
will be achieved
or overran.
4

5. Goals of DMP
Agreement on the Goals of the Defect Management
Process
• The foremost goals are to
• Prevent the Defect
• Early detection
• Minimize the impact
• Decrease in change requests
• The process should be Risk driven.
• Integrated with Software Development
• Continuous effort to improve the process
• Most defects are caused by due process
5

6. Defect Management Process
Defect
Prevention
Defect
Discovery
Defect
Resolution
Process
Improvement
6

7. Objectives
To enable participants understand and apply defect
prevention concepts
Defect Prevention Objectives
 Identify and analyze the causes of defects
 Reduction in number of defect categories
 Reduction in frequency of common defects
 Reduction in the extent of defect escape between test phases and
releases
7

8. Defining the Defect Prevention
Defect Prevention is a process of
improving quality and productivity by
preventing the injection of defects into a
software work product.
Definition: “…an activity of continuous
institutionalized learning during which
common causes of errors in work products
are systematically identified and process
changes eliminating those causes are
made.”
[Eickelmann]
8

9. Defect Prevention Activities
9
Establish practice of Root Cause Analysis within
projects for Analysis of Identified Defects
Identify critical processes as part of root cause
analysis
Set goals for improving critical processes with
team-level accountability
Reduce most frequent type of defects such as
“ not following coding guidelines” or Ambiguity
within Requirements and Specifications
Analyze opportunities for improvement by
conducting escape analysis.

10. Defect Prevention
 It is virtually impossible
to eliminate the defects
altogether.
 Testers/Developers
should collaborate for
quick detection of
defects and minimize
the risk.
 Assess the critical risks
associated with the
system and identify the
defects
10
Minimize expected Impact
Prioritize
based on
Impact
Estimate
Expected
Impact
Identify
Critical
Risks

11. Deliverable Baselines
When should it occur in the Development
timeline?
 When a Predefined milestone is
reached then Product is Baselined
 Further development work continues
from one stage to another.
 A product should be considered
baselined when developers pass it on
for integration testing.
11

12. Defect Discovery
 A defect is said to be discovered when it is
brought to the attention of the developers
and acknowledged (i.e., “Accepted”) to be
valid one.
 A defect discovery team should comprise of
respectable and knowledgeable individuals
and should be lead by a Facilitator.
12
Find/Discover Defect
• Discover defects
before they
become major
problems.
Report Defect
• Report defects to
developers so that
they can be
resolved.
Acknowledge/Accept
Defect
• Agreed that the reported
defect is valid by the
developers

13. DEFECT RESOLUTION PROCESS
A resolution process needs to be
established for use in the event there
is a dispute regarding a defect. For
example, if the group uncovering the
defect believes it is a defect but the
developers do not, a quick-
resolution process must be in
place. The 2 recommended
processes:
• Arbitration by the software owner -
- the customer using the software
determines whether or not the
problem is a defect.
• Arbitration by a software
development manager -- a senior
manager of the software
development department will be
selected to resolve the dispute.
Prioritize
Risk
Schedule
Fix
Fixing
Defects
Report the
Resolution
13

14. Process Improvement : General
Feedback
 Senior Management must understand, support, and be a part of the
Defect Management Program.
 The Defect Management process should be integrated into the overall
software development process.
 To the extent practical, the process should be repeatable and
automated.
 Specific development approaches
(e.g., testing, inspections, reviews, exit criteria etc.) should be chosen
based on project objectives and risks that must be addressed.
 Process improvement efforts should be driven by the measurement
process: metrics, reporting & decision making.
 It should be noted, the process does not require a significant
investment, yet will likely result in significant payback
14

15. Implementing Process
Improvements
Continuous
Improvement
through
Collaboration
Define Critical
Metrics
Identify
Critical Risks
Identify
Process
Improvements
Develop
Project
Management
Plan
Pilot
“Improved”
Defect
Management
Process
15

16. Define Critical Metrics
 Critical Metrics Joint Definition Session:
◦ What can we do to ensure that more of these are found earlier
in the cycle?
◦ Have the defects from these „last minute‟ discoveries been
analyzed to determine why they were missed in our normal
SIT test cycle?
◦ When these issues are found late in the cycle, what is done to
ensure that we catch them next time?
◦ This is stemming from the way our business team is doing
their testing (ad-hoc testing coming late, late in the process).
 Correction of Errors (COE) analysis
◦ the number of defects which were found in the final days of
UAT and all of the ones we found during the dry runs.
◦ Why were they not found earlier?
◦ What phase of testing that uncovers the majority of them?
16

17. Calculated Metrics & Phases
 % Complete
 % Defects Corrected
 % Test Coverage
 % Rework
 % Test Cases Passed
 % Test Effectiveness
 % Test Cases Blocked
 % Test Efficiency
 1st Run Fail Rate
 Defect Discovery Rate
 Overall Fail Rate
17

18. Identify Critical Risks
 Risks which negatively impact the Critical Metrics
set
◦ Some of the few Risks may be as follows:
 Stable mature technology vs. new technology
 Technical platform
 Large integrated system vs. small stand-alone
system
 Tight target dates
 Limited budget
 Bottlenecks within the Development & Testing
cycles
 Resource constraints: Overview of all resources
within Dev and testing to decrease:
 Resource access request/on-boarding bottlenecks
 Resource need competition between projects
18

19. Identify Process
Improvements
Escapes
This is done by driving the defect discovery as far
back into the software development process as
possible.
 Escape Analysis process benefits:
◦ improve product quality and customer satisfaction
◦ lower costs spent on resolving customer-found defects
◦ Reduce defect escape between test phases and
releases
 Test scenario – test case reviews
 Exit Criteria for Developer testing
 QA acceptance of Development
 Environment/Non-functional Requirements
 Technical requirements
 Environment baselines
 GUI checklists prior to test case execution
 No Ad-hoc testing: document all variants of testing and their 19

20. Types of Errors
20
Human Errors
Omission
• Knowledge
• Ignorance
• Informational
• Process-related
Translational
• Miscommunication
• Mismatch in solution
and requirement
• Mismatch with
requirement and test
case
Design and Coding
• Affect Data Integrity
• Alter Correctly stored
data
• Affect downstream
system dependencies
• Affect testing outcomes
Testing
• Failure to notice a
problem
• Misreading the screen.
• Failure to execute a
planned test.
• Criteria disagreements
• CRs, Rejections

21. Lessons Learned – Feedback
Testing
Integration Testing should be
scheduled so that the core
modules are initially tested.
Rigorous unit testing to be
done to avoid logical errors.
Basic level review and testing
should be done at developer's
level before handing over the
code to the testers and
reviewers.
Communication
Client responsibilities should
be clearly communicated in
the beginning of the project.
Design documents should
contain all the necessary
validations to avoid validation
error.
QA should create an
accessible communication
model to increase
transparency, collaboration
and trust
Process
Checklist should be used to
avoid GUI errors
Test cases should be formed
with test data.
Developer team should accept
defect while researching to
move defect through pipeline.
HPQC should be utilized at all
test phases :
DIT, FIT, SIT, UAT, Prod to
maintain visibility of all
program defect and for trend
assessment
21

22. SAMPLE ISSUE # 1
Corrections include
• Increased downstream
communication for Tier 1
software owners
• Create a GUI Checklist to
baseline what is acceptable
viewing per page type
• Increase Dev visibility into high
impact change control at pre/post
go live Checkpoints
• Include Change Coordinator in
install planning that can increase
risk to prod environment
• Escalation method:
• IT Change coordinator for release management
team to Program Manager, RM Manager and QA
Managment
• Concerns
• Increased workload
• the multiple installs of the OM700 program that
needed review and approvals.
• Issues were found during a dry run last week,
• Issues found in a load test from past weekend
• the change for next week was found last night.
• Not enough planning and test cases performed on
the screen that this program runs in.
• Reactive response
• The change for Monday went to CAB meeting to
ensure proper testing and planning have been
done.
• Why?
• To prevent any impacting issues going forward as
occurred with the go live in Production installs.
22

23. SAMPLE DEFECT ANALYSIS #1
DEFECTS
• 13 Open Defects.
• 60% defects with Source Code and
Design as the root cause.
• 107 Rejected defects for this triage
• 30% of all reported defects were
rejected.
OPPORUNITIES
• Implement a Rejection Resolution
process to understand why so
many defects are reported and
rejected.
• Increase visibility to Design
Traceability and Unit
testing/Development Integration
testing for increased quality in
upstream Program processes.
23
DEFECT CAUSE /
DEFECT STATUS

24. Status: IT Release Testing As Of 21-Oct-2011
11.90 17.53
Sev 2 MTTR "Days" Sev 3 MTTR "Days"Sev 1 MTTR "Days"
1.74
Sev 4 MTTR "Days"
0.00
Defect Mean Time To Repair for System Integration Test
AVG MTTR "Days"
10.39
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0
22/Aug/11
29/Aug/11
05/Sep/11
12/Sep/11
19/Sep/11
26/Sep/11
03/Oct/11
10/Oct/11
17/Oct/11
24/Oct/11
31/Oct/11
07/Nov/11
Days
Defect "MTTR" Mean Time To Repair
Sev 1 MTTR Sev 2 MTTR Sev 3 MTTR Sev 4 MTTR

25. Status: IT Release Testing As Of 21-Oct-2011
SIT 30-Aug-11 SIT 21-Oct-11 SIT 30-Aug-11 SIT 4-Nov-11
Regression 24-Oct-11 Regression 4-Nov-11 Regression 24-Oct-11 Regression
Planned Start Planned Finish Actual Start Trend Finish
0
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150 22/Aug/11
29/Aug/11
05/Sep/11
12/Sep/11
19/Sep/11
26/Sep/11
03/Oct/11
10/Oct/11
17/Oct/11
24/Oct/11
31/Oct/11
07/Nov/11
TestCases
Test Case "Pass" Powercurve
SIT Actual Pass SIT Projection SIT Trend
Regression Actual Passed Regression Projection Regression Trend

26. Test Case defect density
Total number of errors found in test scripts vs. developed and
executed.
 (Defective Test Scripts /Total Test Scripts) * 100
Example: Total test script developed 1360, total test script executed
1280, total test script passed 1065, total test script failed 215
So, test case defect density is
215 X 100
---------------------------- = 16.8%
1280
This 16.8% value can also be called as test case efficiency %, which
is depends upon total number of test cases which uncovered
defects.
26
Smaller
test case
density %
Increased
test case
efficiency
High test
case
execution
rates

27. Status: IT Release Testing
Sev 4 DensitySev 2 Density
18.18% 0.83%
Defects Density for System Integration Test
Sev 1 Density
2.48%
Total Density
68.60%47.11%
Sev 3 Density
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
22/Aug/11
29/Aug/11
05/Sep/11
12/Sep/11
19/Sep/11
26/Sep/11
03/Oct/11
10/Oct/11
17/Oct/11
24/Oct/11
31/Oct/11
07/Nov/11
Defect Density (Defects/Pass TCs)
Overall Defect Density Defect Density (Industry Avg)
Sev 1-2 Defect Density Sev 3- 4 Defect Density

28. Status: IT Release Testing
3 74
Sev 3-4 Defect Backlog
Defect Discovery and Severity 1-2 Backlog for System Integration Test
Sev 1-2 Defect Backlog
0
5
10
15
20
25
30
0
10
20
30
40
50
60
70
80
90
22/Aug/11
29/Aug/11
05/Sep/11
12/Sep/11
19/Sep/11
26/Sep/11
03/Oct/11
10/Oct/11
17/Oct/11
24/Oct/11
31/Oct/11
07/Nov/11
DefectBacklog
Defects
Defect Discovery and Severity 1-2 Backlog
Defects (Sev 1-2) Defects (Sev 3-4) Defect Backlog (Sev 1-2)

29. IT TESTING ITERATION 2 UAT AND DRY RUN ANALYSIS
DEFECTS
OPPORUNITIES
• Implement a Rejection
Resolution process to
understand why so many
defects are reported and
rejected.
• Increase visibility to Design
Traceability and Unit
testing/Development
Integration testing for
increased quality in
upstream Program
processes.
29

30. Status: Sample Release Testing Pass Trend Analysis
SIT 19-Sep-11 SIT 28-Nov-11 SIT 21-Sep-11 SIT 30-Jan-12
Regression 29-Nov-11 Regression 5-Dec-11 Regression 29-Nov-11 Regression
Planned Start Planned Finish Actual Start Trend Finish
0
50
100
150
200
250
300
350
400
450
500
550
600
650
700
750
800
850
900
950
1000
1050
1100
1150
1200
1250
1300
15-Sep-11
22-Sep-11
29-Sep-11
06-Oct-11
13-Oct-11
20-Oct-11
27-Oct-11
03-Nov-11
10-Nov-11
17-Nov-11
24-Nov-11
01-Dec-11
08-Dec-11
15-Dec-11
22-Dec-11
29-Dec-11
05-Jan-12
12-Jan-12
19-Jan-12
26-Jan-12
02-Feb-12
09-Feb-12
TestCases
Test Case "Pass" Powercurve
SIT Actual Pass SIT Projection SIT Trend
Regression Actual Passed Regression Projection Regression Trend

31. Status: Sample Mean Time To Repair Data
Sev 4 MTTR "Days"
16.70
Sev 1 MTTR "Days"
17.45 17.99
Sev 2 MTTR "Days" Sev 3 MTTR "Days"
0.69
Defect Mean Time To Repair for System Integration Test
AVG MTTR "Days"
13.21
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0 15/Sep/11
22/Sep/11
29/Sep/11
06/Oct/11
13/Oct/11
20/Oct/11
27/Oct/11
03/Nov/11
10/Nov/11
17/Nov/11
24/Nov/11
01/Dec/11
08/Dec/11
15/Dec/11
22/Dec/11
29/Dec/11
05/Jan/12
12/Jan/12
19/Jan/12
26/Jan/12
02/Feb/12
09/Feb/12
Days
Defect "MTTR" Mean Time To Repair
Sev 1 MTTR Sev 2 MTTR Sev 3 MTTR Sev 4 MTTR

32. Status: Sample Defect Disovery and Resolution
Backlog
32 118
Sev 3-4 Defect BacklogSev 1-2 Defect Backlog
Defect Discovery and Severity 1-2 Backllog for System Integration Test
0
10
20
30
40
50
60
70
80
90
100
0
50
100
150
200
250 15-Sep-11
22-Sep-11
29-Sep-11
06-Oct-11
13-Oct-11
20-Oct-11
27-Oct-11
03-Nov-11
10-Nov-11
17-Nov-11
24-Nov-11
01-Dec-11
08-Dec-11
15-Dec-11
22-Dec-11
29-Dec-11
05-Jan-12
12-Jan-12
19-Jan-12
26-Jan-12
02-Feb-12
09-Feb-12
DefectBacklog
Defects
Defect Discovery and Severity 1-2 Backlog
Defects (Sev 1-2) Defects (Sev 3-4) Defect Backlog (Sev 1-2)

33. Defect Prevention Support Criteria and Actions
33
• Management Approval and Support: Management must support the project team and the
objectives of the defect management effort if it is to be successful.
• Receptivity of Project Team: The project team should wholeheartedly embrace the
program. Project management, in particular, must take ownership for the successful
implementation of the program.
• Project Duration: The project should not be so long that results will not be known for a long
period of time. A project duration of three to six months is probably ideal.
• Project Risk: The project should be sufficiently large and complex that the results will be taken
seriously, but not so large and complex that success of the pilot project is unlikely.
• Early in the Life Cycle: The defect management program should be integrated into project
plans and not retrofitted into a project whose approach is well established.
Criteria for implementing Defect Prevention efforts
• Conduct Training: The project team should be trained in the Defect Management concepts, if
necessary. Example: on-boarding new team members.
• Incorporate Defect Management Into Project Plan: Each pilot project should adapt the Defect
Management approach to its particular circumstances. Emphasis should be placed on
automating as much of the process as practical, and measuring the process. Data collected
from the pilot projects should form the baseline for the Critical Metrics Set.
• Review Results and Refine Approach: Progress implementing the Defect Management
Program should be monitored and the approach refined as appropriate.
• Report Results: Periodic status reports, which report pilot project results, should be issued.
Defect Prevention Actions

34. Problem Closure & Congratulate Team
• Acknowledge significance and value of problem
solution
• Recognize team‟s collective efforts in solving the
problem as well as individual contributions
• Document what was learned in solving the
problem; lessons learned not only about the
problem which was solved but also about the
problem solving process
• Consider investigating other potential causes as
preventive actions
• Create case study/problem solving reports
34

35. Thank You
35
If you have any questions or concerns about this presentation
Contact
Yolanda Williams ybowdrywilliams@gmail.com