1. Supplemental Data To Linked In
Post On 6/29/2015 On Capture-
Recapture For Software
Inspections With A Single Defect
James K. Orr
6/30/2015
2. Motivation For This Presentation
• I have been reviewing information that was still available from my time with
the Space Shuttle Primary Avionics Software (PASS) system.
• In August / September, 2011, I emailed a number of files to colleagues
continuing to work at NASA. This material was on PASS quality data and
discrepancy reports/analysis, with copies sent to my home email address.
• Recently, I was reviewing what information I still had access to, and moving
data from email attachments to more permanent backup.
• In doing this, I realized that I did not have any of the data on the study that I
did on using the Capture-Recapture concepts applied to inspections with a
single defect.
• I created the post on 6/29/2015 by dumping what I remembers in a single
composition activity.
• In hindsight, the absence on any data probably means that most readers will
miss key insights that I gained from the original data.
• This presentation is an attempt to quantify the concepts, although by
definition I am limited to artificial / simulated data only.
3. Warning On Potential For Misuse
• This presentation attempts to show significant
advantages in use of data as collected on the
Space Shuttle Primary Avionics Software (PASS)
project for quality and process management.
• However, this data must never be used in
personnel decisions such as appraisals.
• In the hands of software process professionals, it
can provide accurate assessment of the state of
the project and information for team training.
4. Reference, Background Information
• Reference:
– http://www.sei.cmu.edu/library/assets/capture-recapture1.pdf
– Getting More Out of Your Inspection Data: Using Capture-Recapture
Models for the Reinspection Decision; Julie Barnard, Khaled El Emam, and
Dave Zubrow; European SEPG Conference – Amsterdam - April 12, 2002
• In the sample of data that I was using, there were 861 inspections, of which
553 had no major errors. Of the 308 with major errors, only 89 had meet the
criteria for the above referenced study. My memory is that there were 71
inspections that had one and only one major error present.
5. Source Of Data
• I have spent about two hours recreating a simulation of software inspections
for the following narrow scope:
– Inspection randomly (equally probably) have 3 to 6 inspectors
– Inspection material contains one and only one major error
– Probably that a single inspector will detect the one major error crudely
model on my memory of Space Shuttle Primary Avionics Software (PASS)
project consistent with 2001 study in collaboration with Software
Engineering Institute
• Probability of a single inspector detecting on major error was
between 0 % and 40 % (randomly distributed).
6. Comment On Inspector Effectiveness
• For the entire set of 308 inspections, I analyzed each inspector for their individual
effectiveness in finding major Errors
– The data used in the SEI collaboration was provided to me with all indication of
inspector’s name converted to aliases. The maintainer of the inspection data could
relate an alias to an individual inspector. I never desired nor never knew the identity of
the individual inspectors; I only worked with the aliases.
– I identified data for all aliases (all inspectors) in the 308 inspections with major errors.
For each alias, I computed the present of errors found by the individual alias (individual
inspector) in all inspections by the total number of errors and later defects found in all
inspections that the individual participated. This represented the probability that this
individual would discover an error present in an individual inspection.
• Summary
– A relatively large percentage of inspectors had never participated in a single inspection
with a major error present (553 of 861 inspections had no major errors).
– Several inspectors had zero major error detected when in inspections with one or more
major errors.
– For those inspectors who had been in inspections with a large number of major errors,
most had detected less than 40 % of the major errors present.
7. Results
• Data is show for three separate simulations
• Data presented is
– Probability that single major error is found by one or more inspectors
– Probability that single major error is found by two or more inspectors
• Statistical simulation for each inspection (number of inspectors,
fixed probability of finding error for each inspector) was done 1000
times
• “Actual” results was simulated by randomly selecting the outcome
(results) on only one of the 1000 simulations
• This was done for 80 different inspections
• 80 inspections were grouped into 8 groups of 10 each where the
lowest group contained the 10 inspections with the lowest mean
probability of finding the error and the highest group contained the
10 inspection with the highest mean probability of finding the error.
8. Data For Simulation # 1
• Data will be shown for three simulations
• Inputs for all three are identical
• The only differences are the random statistical
differences (e.g., a different random number
selected to simulate “actual”, different
inspection team randomly assigned
characteristics)
12. Data For Simulation # 2
• The only differences are the random statistical
differences (e.g., a different random number
selected to simulate “actual”, different
inspection team randomly assigned
characteristics)
16. Data For Simulation # 3
• The only differences are the random statistical
differences (e.g., a different random number
selected to simulate “actual”, different
inspection team randomly assigned
characteristics)
