In my recent Asset Management work on a buildings register with around 1 million components, I have become interested in statistical methods for estimating the quality of the information. This reminded me of a technique I discovered years ago in my work in software project management, which I hope you will enjoy.

1. Estimating Total Defects WA Integrated Asset Management, 28 November 2016
TotalDefectsSoftwareProject.docx 1
Estimating the Total Defects in a Software
Project before Completion
Hein Aucamp
WA Integrated Asset Management
WA Integrated Asset Management is an
Infrastructure Asset Management company
serving Local Government and the mining
sector based in Perth, Western Australia.
www.waiam.com.au
In my recent Asset Management work on a buildings register with around 1 million components, I
have become interested in statistical methods for estimating the quality of the information. This
reminded me of a technique I discovered years ago in my work in software project management,
which I hope you will enjoy.
In 2003 I attended a session on software testing. A question arose about estimating the total defects in
a project before project completion. The answer given was that there was no way of doing this. But in
fact there are 2 ways. I want to mention one briefly and discuss the other in more detail.
A technique to estimate the total defects tell a project team at least 2 important things:
1. Their present progress in removing defects.
2. The remaining defects they still have to discover.
Here is the brief mention: the method of seeding. Seeding come from biological investigations. A
specific number of tagged fish are released into a population, and investigators monitor the proportion
of non-tagged to tagged fish in catches. So if a catch of 1,000 fish contains 5 tagged fish, and if 100
tagged fish were released, the estimated total population is 100 / 5 * 1,000: a total of 20,000.
Seeding is used in software projects by deliberately introducing defects (they keep a record of them!),
and then measuring the proportion of the introduced defects of the actual defects found.
But I am not talking about seeding in detail here. Rather, my focus is a formula mentioned by Steve
McConnell in his Software Project Survival Guide (Microsoft Press 1998).
McConnell mentions a technique where 2 independent testers collect defects on a project. If Tester 1
finds A defects, and Tester 2 finds B defects, and if there are C common defects in their findings, then
the estimate of the total defects is T = A * B / C.
McConnell mentions this formula without elaboration, so let’s investigate it in some detail.
If there are T defects in a project, the probability of any particular defect being found is 1/T. The
probability of a particular defect occurring in a set of size A is A/T. The probability of a particular
defect occurring in a set of size B is B/T.

2. Estimating Total Defects WA Integrated Asset Management, 28 November 2016
TotalDefectsSoftwareProject.docx 2
Now the assumption is that the probability of a particular defect occurring in both Set A and Set A is
(A/T) * (B/T). (We shall accept the assumption at this stage, and investigate it below.) We have thus
the probability of a point occurring in Set C: the set of common defects.
But we also have an independent probability of a point occurring in Set C, which is simply C/T. By
equating the 2 expressions and solving for T, we get McConnell’s formula.
But how valid is this assumption: that the probability of a particular defect occurring in both Set A
and Set B is (A/T) * (B/T)? This calculation of probability applies strictly only to independent events
(like rolling a die twice in succession).
In testing there is the real (but remote) possibility that the 2 testers will each discover exactly half of
the defects, with none in common. In that case, McConnell’s formula will suggest infinite defects,
when in fact all are known.
Let’s take an example of a total of 12 defects, where Tester 1 discovers 6 defects and Tester 2
discovers 6. The number of unique sets that each can discover is a well-known mathematical
combination: (12!) / (6! * 6!). The result is 924.
The possibility of Tester 1 discovering 6 defects which entirely exclude the 6 of Tester B’s set is
1/924, or 0.11%, in which case they will conclude from the formula that there are infinite defects
(when in reality all are known).
On the other hand, the possibility of their both discovering the same 6 is the also 0.11%, in which case
they will conclude from the formula that all defects are known (when in reality only half are known).
McConnell’s formula in fact relies on a common amount of 3 defects: T = 6 * 6 / 3 = 12.
The chart below shows the probabilities of the two 6-point samples having a number of common
points:
The probability of having 0 or 6 points in common is so remote that it does not appear on the chart.
These are the 0.11% cases which led to widely wrong conclusions.
The probability of having 3 points in common (on which McConnell’s formula depends) is 43.29%.
The chart shows that the probabilities favour the formula’s being a reasonable estimate. The 3 middle
columns account for 91.99% of the probability, making extreme estimation errors increasingly
unlikely.1
1
If you are interested in the maths, please email me at hein.aucamp@waiam.com.au, and I will send you the
detail. It involves combining groups of different length from partitions in the sample space.