Advanced Defect Management

Advanced Defect Management – The IBM Way
Sabarinath Venugopalan
Practice Leader, Nordic – IBM Test Services

© 2009 IBM Corporation
ERUC 2009
Agenda
 Defect Reduction Methord
 Test Planning and Optimization Workbench (TPOW)
 Summary
 Client Examples
 Introduction to Advanced Defect Management

ERUC 2009
Key Messages
 The Next Generation of Testing is
emerging to meet the demands of the
intelligent enterprise
 The key to the Next Generation of
Testing is informed actions, enabled by
predictive analytics
 Advanced Defect Management is
powered by informed actions. It delivers
significant benefits to the major factors
governing the Next Generation of
Testing

ERUC 2009
Today’s leading Enterprises are Intelligent. They
are…
Anticipating
Precise
Aware
Questioning
Empowering
Gathers, senses, and uses
structured and unstructured
information from every
node, person, and sensor
within the environment
Predicts and prepares for the
future and doesn’t only react or
correct actions, but also steers
and evaluates trade-offs
Linked
Connects internal and
external functions front to
back across geographies in
a way that aligns to desired
business outcomes
Enables and extends
employees' memory,
insight and reach, as well
as the authority to decide
and act
Reserves the right to get
smarter by challenging its
status quo while creating
new opportunities
Uses only the most relevant
information to support timely
decisions/actions closer
to the point of impact and
consequence
Source: IBM Study: Business Analytics and Optimization for the Intelligent Enterprise, March 2009

ERUC 2009
Old ways of decision making and management are breaking
down. Organizations will need to adopt new ways of
working to improve speed to insight and speed to impact
Traditional Approach New Approach
Instinct and intuition Fact driven
Corrective Directive
Efficient Optimized
Years, months, weeks Hours, minutes, seconds
Human insight Applied semantics
Decision support Action support
Source: IBM Study: Business Analytics and Optimization for the Intelligent Enterprise, March 2009

ERUC 2009
The Next Generation of Testing will be driven by four
factors. Advanced Defect Management substantially
improves these factors.
Quality Improvement
DATA
INSIGHTS
Cost Reduction Risk Management
Schedule Acceleration
3
42
1
Analytical I Insightful I “Smarter” I Linked
Total View of Test
Managing Change will be paramount

ERUC 2009
IBM’s Advanced Defect Management capabilities are based
on two related analytical predictive capabilties. These are
enabled by significant base of data collected over 25 years.
 IBM® Defect Reduction Method™ is a patent pending, statistical,
objective defect classification and analysis process that helps
organizations find and fix defects as early as possible and prevent
future defects from being injected.
 IBM® TPOW™ is an advanced, empirically risk-based planning tool
that delivers an optimized test strategy, supporting master test plan
and ongoing project planning. At the root of TPOW is the defect
prediction capabilities that that gives you insights about your
application projects.

ERUC 2009
Advanced Defect Management drives dramatic reductions
in cost and schedule while improving overall quality and
lowering business risk
Cost Overview
(Total: $13,277,407)
Source: Test Planning and Optimization Workbench example
Cost Overview
(Total: $20,653,285)
Schedule: 89 days
Environment: 14 days
Schedule: 37 days
Environment: 5 hrs
$911,260
$10,240,335
$2,125,812
Defect fix cost
Business cost
Test cost
Defect fix cost $14,439,176
Test cost $1,579,838
Business cost $4,634,271
Cost Reduction
DATA
INSIGHTS
35%+
Cost Reduction

ERUC 2009
Car Wash
Car Wash
Car Wash
Car Wash
Car Wash
IBM is leveraging on Advanced Defect Management internally and in
client engagements to substantially improve the four factors of Next
Generation Testing.
TPOW Macro
Planning
(Reference
Project)
TPOW Micro
Planning
Test
Optimizers
Execution
Managed Testing Services
Test
Execution
Implement advanced Test
practices, processes and
optimizers with the full-life cycle.
Execute managed test services
leveraging globally deployed
resources executing test
according to the newly
implemented processes.
 1000 – 5000
Function Points
 Currenty in
development or
under major
enhancement
1B
2C
Execute detailed bottom-up test planning
leveraging the Macro Planning feature of the
Test Planning and Optimization Workbench.
3
Execute Pilot
TPOW Macro
Planning
(Client
Project Data)
1A
2A
2B
2C
Conduct a Defect Analysis Starter which
executes IBM’s Defect Reduction Method to
gather quantitative data needed to customize a
client Reference Project
Execute IBM’s Test Planning and Optimization
Workbenchby starting with Macro Planning. IBM
assesses an existing application and, provides
recommended improvements.
1A
1B
Execute Test Optimizers recommended thru
Macro Planning.
2B
DAS
Execute IBM’s Test Planning and Optimization
Workbench Macro Planning using baseline
data gathered by the Defect Analysis Starter.
2A 3
Select target programs (s)
Applications

ERUC 2009
IBM® Defect Reduction Method™ (DRM) provides measurable insight
into software quality and development processes
When does it start/end?
 Used continually throughout the test life cycle
 Defect classification can begin at any point that
defects have been recorded
 Comprehensive analysis reports typically run at test
exit for each phase or for the release, but interim
reports should be generated periodically throughout
test execution to gain the most value from DRM
IBM® Defect Reduction Method™ is a patent
pending, statistical, objective defect classification and
analysis process that helps organizations find and fix
defects as early as possible and prevent future defects
from being injected.
DRM Classification DRM Analysis
►Systems engineering life cycle
Test Assess and Planning Test Execution and Reporting
Testing project
startup
Test Design Test Implementation and Close
Ongoing DRM activity

ERUC 2009
The IBM® Defect Analysis Starter™ quickly identifies risks and helps improve the
quality of system and application development. Application of DAS is typically
completed in 3 weeks!
X X X
X X X
XX XX XX
XX XX XX
The IBM® Defect Analysis Starter™ provides business value by:
What is it?
A defect based analysis, using IBM®’s Defect Reduction Method™ (DRM), to
identify high priority cost reduction opportunities associated with software
test practices and defect prevention.
The IBM® Defect Analysis Starter™ utilizes:
 An IBM approach that encompasses classification and analysis
 Benchmarks based on defect data collected for nearly two decades,
mapped to industry, quality level, and test phase/activity
 Metrics to objectively quantify and prioritize risk and opportunity, and to
measure quality
The IBM® Defect Analysis Starter™ includes:
 Classification of client provided defects
 Analysis and measurement of risks (General Test Effectiveness, Test
Design Effectiveness, System Stability and Completeness) and Quality
(Evaluation against Quality Benchmarks, Focus Area Identification, and
Artifact Value Assessment)
 A Rough Order of Magnitude business case reflecting cost reduction
opportunity (Invalid Reduction, Production Defect Reduction, Earlier Defect
Removal, Cycle Time Reduction, and Prevention of Defect Injection)
 Identifying and recommending specific actions to reduce risk and cost associated with developing and testing
systems or applications
 Acknowledging and reinforcing client strengths
 Guiding the client to utilize test practices that can increase their ROI while delivering improved quality

ERUC 2009
TPOW helps to answer challenging questions …
 Am I doing too much/too little testing?
 Am I getting my money’s worth out of testing?
 Will an increased testing investment drive any further
improvement to quality?
 How can I shift more of my investment to earlier project
phases/development?
 What is the overall cost, schedule, and quality impact of
moving testing offshore?
 How can I determine the quality of my application based
on my testing effort?
 I just got a budget cut; what testing should I eliminate?
What impact will it have on application production quality?
 How early in the life cycle can I develop an overall test
strategy, including effort, schedule, cost, and quality
predictions?
 What if I….?

ERUC 2009
TPOW works at multiple levels
Model initial cost
and benefits
Provides guidance in test
design and execution
Achieve the organization’s
quality, schedule and cost goals
Analytical
Project
Oriented
Micro
Planning
Macro
Planning
Ongoing
Project
Planning
Intelligence
Startup Deployment
Solution
close
Solution
outline
Macro
design
Micro
design
BuildSolution
requirements
Industry and client benchmarking
21 3
4

ERUC 2009
Example: Macro Planning - Create a
Baseline
Baseline
 Derive Function Points for High / Medium and Low
TPOW Baseline
 Definition of Test Activities
 Definition of Triggers Distribution
 Allocation of Test Effort
– Per Test Activity
– Per Trigger
 Definition of Global Resource
Model
 Benefit of using “Car Wash”
 Assume 1 Function Point for 10 KLOC
Macro
Planning
Macro
Planning

ERUC 2009
Example: Macro Planning - Assess Process
Initiatives
Baseline + DAS Initiative
 Complete for medium case
Baseline + DAS
 Assume 1 FP for 10 KLOC
Baseline
Outcome
Baseline Das initiative %
Test Cost 4,263,066.00$ 4,430,601.50$ -4%
Defect Fix cost 19,861,196.50$ 10,667,479.75$ 46%
Business cost 5,272,015.00$
Escaped defects 375.00 61 84%
 Definition of Test Activities
 Definition of Triggers Distribution
 Allocation of Test Effort
– Per Test Activity
– Per Trigger
 Definition of Global Resource
Model
 Benefit of using “Car Wash”
Macro
Planning
Macro
Planning

ERUC 2009
Agenda
 Defect Reduction Methord
 Test Planning and Optmization Workbench (TPOW)
 Summary
 Client Examples
 Introduction to Advanced Defect Management

ERUC 2009
Reduction of Client PLM Incidents
PLM Incidents Trend
0
20
40
60
80
100
120
140
160
180
June
July
A
ugustS
eptem
ber
O
ctoberN
ovem
ber
No.ofIncidents
Sev2
Sev3
Sev4
Total
Achieved through application of Defect Reduction Methods (DRM)
*Trough in August attributed to vacation

ERUC 2009
Client Example: Selection of Accelerators based
on TPOW Analysis
Scenario Escaped
defects 2)
Production
Defect rate
Cost to fix
Production
Defect 1) DKK
Potential
annual saving
Baseline 43.226 23% 562.000.000 0
TPOW 42.031 21% 546.000.000 16.000.000
+Carwash,+ DRM Phase1 +
Test Data Mgt phase 1&2
33.785 17% 439.000.000 123.000.000
+ DRM phase 2+3, + Test
data management phase 3
26.928 14% 350.000.000 212.000.000
TCOE 14.022 7% 182.000.000 380.000.000

ERUC 2009
Team of testers spent 3 1/2 months developing
detailed estimates for a major development
project.
Since the project was in the design phase, they
were trying to develop estimates that would
support a fixed budget for testing.
The client wanted a guarantee that no more than
5% of the defects associated with the application
lifecycle would arise during production.
If weight was a measure of accuracy, the 10,378
person days of effort looked like a solid estimate.
The team asked for an independent estimate to
validate their numbers.
In just 1 1/2 days using advanced defect
methods, the assessment of the teams' estimate
yielded …

ERUC 2009
Increase test effort to meet decreased defect rate ..
PD: Person Day
ST SIT PT UAT Total
XYZ 21.68%
2250
45.37%
4708
10.79%
1120
22.16%
2300
100%
10,378
Improvement
suggestion
26.85%
3812
51%
7242
6.41%
910
15.75%
2236
100%
14,200
DecreaseIncrease
Scenario Output
Projection Defect discovery and escape to production
Defect production rate = 368 / 1921 = 19%
Validation
and Improvement
Validation result
Given the quality objective of production defect rate 5%, the existing estimated test
effort (10,378PD) is not enough
Improvement suggestion
1. Add the total test effort to 14,200PD
2. Adjust the test effort distribution on test activities (see table below)

ERUC 2009
Summary
 Applying predictive analytics to reduce
defects and to move defects earlier in
the lifecycle delivers substantial gains
 Taking a total view of testing generates
dramatic benefits
 IBM’s capabilities in advanced defect
management are deemed unique by
technology analysts

ERUC 2009
Contact:
Sabarinath Venugopalan
Practise Leader
IBM Test Services, Nordic
+46-70-793 2665
sabari@se.ibm.com

Advanced Defect Management

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Advanced Defect Management

Similar to Advanced Defect Management (20)

Advanced Defect Management