2. OVERVIEW
• Describes an on-going engagement to expand a measurement process within a
commercial organization
• Large international company involved in commercial software-intensive systems
development
• Explore how to provide measurement guidance in environments where there is no
realistic/practical executive commitment to quantitative understanding of
business performance
3. ABOUT DISTRIBUTIVE
• Involved in Measurement since 1996, primarily servicing high maturity
customers (with 95% at CMMI level 3 and higher)
• Market, sell and support the DataDrill measurement tool for collection,
storage, analysis and reporting
• DataDrill currently in use at over 300 sites in US, Europe and Asia
• Distributive has privilege of working to implement or expand measurement
processes in environments using advanced practices
4. GOALS
At the Program Level
• Provide a defect discovery estimate that can be used as a target/goal for
monitoring testing activities after design
• Replace the current trending calculation that is overly complex, trying to
compensate for not being accurate during most of the phases
At the Feature Level
• Develop a method to estimate defect discovery by activity
In the Company Culture
Demonstrate that measurement can provide useful information which encourages:
1) Executive to commit to quantitative management
2) first and second line managers to participate in minor process changes
5. CULTURE
There is a great deal of project resistance to changing the planning and management of
testing activities, which leads to …
If measurement is so great, then set it up and show me, which is difficult because …
There is no historical data, further compounded by …
The project won’t change to provide much required data , which leads to …
My approach was to demonstrate that defect discovery prediction (themselves) doesn’t
take much time and is more effective than a 3 week trend line.
6. AVAILABLE DATA
About six and half years worth of data
Spans 3 builds (A, B and C)
Defects found during unit test and later activities
Roughly 5500 total defects
Classified by:
Build
Feature
Discovery date
Priority
Activity Where Discovered
7. PROGRAM LEVEL DEFECT DISCOVERY
Converted the defect discovery numbers to a percent of total and plotted by activity
Defect discovery by activity exhibited nice similar shape
8. PROGRAM LEVEL DEFECT DISCOVERY
• Grouped the 9 activities into four phases:
1. Design and development testing
2. Unit testing
3. Integration and system testing
4. Acceptance testing
• Plotted the defects per phase as a percent of total.
• Preliminary fit of Weibull shown at right
9. PROGRAM LEVEL DEFECT DISCOVERY
• Plot discovery per phase by build
• Fit to a Weibull distribution (with good results)
• Compared four phase data to a published commercial study (Infosys published a
CMMI Level 5 company
10. PROGRAM LEVEL
Next Steps
• Move backwards into development and design activities
• Start collecting and reviewing (the very sparse) software size and
feature/program characteristics to continue with defect prediction … not sure how
many “X” are going to be available
• As of today, have collected about 18 months of detailed staff, cost and schedule
data so we can start tuning parametric estimation model
• Planning to use Schedule Risk Analysis for presenting schedule delivery risk to
augment existing schedule measures
11. FEATURE LEVEL
Feature represents a lower level than program, where there are a small number of
features (typically less than 25) in a build.
Across the three builds, there were roughly 45 features developed which followed 9
testing activities in the 4 phases
Current feature level planning is not formal. Primary drivers are:
• All features development and unit testing be completed before program starts
integration test activities
• Program can track to a “pencils down” event at end of acceptance test
12. FEATURE LEVEL
My suggestion was to use a belief network to accommodate the drivers of defects and
the effectiveness of the testing activities trying to remove them
For example, if an early activity finds “fewer” defects, then instead of assuming the
feature has high quality, it assumes that testing quality was low and the latent
defects will be higher.
Design process
quality
Feature
complexity
Testing quality
Operational
Usage
Defects inserted
Defects Found
& Fixed
Latent
Defects
Defects
In Use
Belief network is based on approach published by Fenton, Neil and Marquez in 2007.
13. FEATURE LEVEL
For better precision, a single node can be decomposed into a sub-network which
provides more granularity when needed.
Belief network is based on approach published by Fenton, Neil and Marquez in 2007.
Design process
quality
Feature
complexity
Testing quality
Operational
Usage
Defects inserted
Defects Found
& Fixed
Latent
Defects
Defects
In Use
Design
process quality
Feature
complexity
Testing quality
Operational
Usage
Defects
inserted
Defects Found
& Fixed
Latent
Defects
Defects
In Use
14. FEATURE LEVEL
• Started building distributions for three nodes in top part of the network based on analysis of
defects by feature.
• Validate the network by comparing the forward direction with feature defect counts
• Only have the top-half data to validate
• Distribution values are defects per 1 KSLOC
Design Process Quality
Low Medium High
4 2 1
Feature Complexity **
Low Medium High
0.5 1 3
** Preliminary.
Testing Quality
Low Medium High
0.3 0.8 1.7
Example:
For Build 1 -Feature 1,
Predicts 252 defects
Found 277 defects
15. FEATURE LEVEL
Not yet finished analyzing all feature data for the distributions, validating or adjusting
the network
One benefit is that the network model is constructed quickly for sparse sets of data
and as accurate as available data
One side effect of the belief network is to get managers to start discussing the
effectiveness of their sub-processes without the feeling that “Hey this is process
improvement!”
Also, the network provides a flexible method for accommodating unique process or
work products through the use of new nodes and edges without changing the
model for existing, validated models
