This report discusses the time constraints of testing, its impact on testing stakeholders and possible ways to mitigate this problem. The report is available for download here: http://www.softwarevalue.com/insights/publications/ta-archives/why-do-we-never-have-time-to-test/ To access more Trusted Advisor articles, visit: www.softwarevalue.com/insights/publications/#trustedadvisor

1. ©2015 David Consulting Group Page 1 of 5
“Why do we never have time to test?”
June 2015
Scope of this Report
This paper discusses the time constraints of testing, its impact on several testing stakeholders, and
possible ways to mitigate this problem. It includes:
 Statistics on testing length.
 Who are some of the stakeholders for software testing?
 What kinds of delays do testers frequently face?
 Making more time to test.
Testing Length
The following estimate of an average testing length is drawn from The Economics of Software Quality, by
Capers Jones and Olivier Bonsignour, which is based on the authors’ clients average test cases per
function point and time per case. The calculation is based on the types of tests used by 70% or more
projects, and provides the following average for a 100 function point project. This assumes a thorough
test of the system using an appropriate number of test cases for the size of the project.
Therefore, between one or two months might be spent for the testing of the sample project. Note that
a 100 FP project is relatively small. Projects of ten thousand function points or more, especially for new
software, are not uncommon. Testing these larger projects to the same degree could take a year or
more, assuming testing time increased in a linear fashion. Testing time actually increases faster as the
project grows larger. To completely test all combinations of possible processes for very large projects
will take far more time than is available. For practical purposes, exhaustive testing is impossible for such
projects.
Minimum Average Maximum
Subroutine testing 0.2 0.3 0.4
Unit testing 7.0 10.0 13.0
Automatic regression testing 9.3 13.3 17.3
Automatic new function testing 17.6 25.2 32.8
TOTAL TEST TIME 34.2 48.8 63.4
EFFORT MONTHS 1.1 1.6 2.1
TESTING TIMES FOR A 100 FP PROJECT IN EFFORT DAYS

2. ©2015 David Consulting Group Page 2 of 5
The above calculation uses only a tenth of the total number of testing types. For example, integration
and system testing were not included in the list of four most common testing types but are still quite
common (listed as used in 60% and 40% of projects, respectively). The more testing methods applied,
the better the final software quality but they will require still more time. Again, the potential testing
time starts to exceed practical reasonableness. As a result, risk analysis and other measures must be
applied to determine how much testing is enough.
Stakeholders
Any software project will have a number of stakeholders, each with potentially very different needs that
can actively conflict with one another. Several stakeholders, with example goals, are presented here.
Testers
A goal of a software tester is to ensure that the application being tested is as free of defects as possible.
As noted above, for any large project this is at best a lengthy and expensive goal, at worst an
unobtainable one. At a certain point, further testing will not be practical. Some testers may naturally
want to be more thorough than is necessary while others may fear reprisal if a defect is missed. Either
way, the testing can easily become overlong because of these goals.
Developers
Developers want the project to be done and out the door so they can move on to the next one, or in
Agile, to finish the current sprint. Testing, or having to return to the program to remove defects, delays
this goal. As for the testers, this can become an even greater problem for developers penalties are
imposed for defects in the code.
Customers
Customers want the application to be perfect out of the box for a low cost. The software must work
without defect on all hardware combinations (especially for home users) and fulfill all requirements.
Most customers realize that this is not going to happen but that is the goal. Software that does not
meet the ideal will be well known in the community very quickly. This can put pressure on the business,
which puts pressure on the manager, and finally on the development and testing teams.
Managers
Like the customer, the manager wants the program to be good quality and low cost, and most
likely also wants a short development time. Improving any one of these goals (reducing cost, increasing
quality, reducing time) requires a sacrifice in one or both of the other two. To decrease time and cost,
testing may be cut or reduced. The process of solving a problem (development) is often, from the
manager’s point of view, more clearly important than the process of preventing one (testing).
Ultimately, management must make the decisions on how much time can be expended on any part of a
project, and testing is often sacrificed in favor of the more visible development.

3. ©2015 David Consulting Group Page 3 of 5
Delays, Delays
There is always the potential for any work to run into delays. Unforeseen problems with the
software, changing personnel, and damaged equipment are all potential problems. There are too many
possibilities to list here, but two will be presented: human factors and changing requirements.
Human Factors
However well-planned the software testing portion of a project might be, there is always the
possibility that attitudes and habits of the development team can get in the way of completion.
Distractions, attitudes towards testing, and politics can all cause delays.
Software teams, clearly, must work on computers much of their time, and computers are rife
with potential distractions: social media, games, e-mail, and so on. These pursuits can sometimes
improve efficiency in various ways, but they are still a lure to waste more time than is appropriate.
A number of testing types, including Subroutine and Unit testing (included in the time estimate
above), are often most appropriately performed by the developers. Additionally, and pre-test defect
removal will also involve the developer. Sometimes, developers do not believe that their time is
properly spent on such activities. Further, even if the developers do relatively little testing themselves,
a separate group of testers sending back the work due to defects, especially if this happens multiple
times, can cause friction and further delays.
Changing Requirements
Most projects are going to have requirements evolve over time. Except with very small
applications, as the work progresses, new features will be desired, a better view of current features will
be available, and some requirements may actually be removed as unnecessary. Priorities will also shift,
even if the requirements remain relatively stable. This increases both the development time and the
testing time, as adaptions are made, but in this case testing is more likely to be sacrificed than
development.
Making More Time for Testing
Defect Prevention
Traditionally, testing has been done after all development tasks are finished but before deployment.
This means that for a small project an extra two months (according to the earlier testing time estimate)
would be added to the end of the project, increasing the likelihood that the testing will be cut short.
Finding defects throughout the development process (as in Figure 1) may increase the efficiency of
removing defects, making the testing needed after the coding phase shorter.

4. ©2015 David Consulting Group Page 4 of 5
Figure 1: Finding Defects throughout the Development Process
Test Driven Development
Test driven development (TDD) comes from Agile practices. It is a discipline in which development does
not start with code, but with the development of a test case, which in turn is based on a requirement.
The code is then written to pass the test case. If the code does not initially pass, it is returned to the
backlog and attempted once again until it succeeds. This means that the testing is spread throughout
the development process and the tests are ready at hand. However, studies of the technique show
inconsistent benefits to costs: the process often costs more than other testing methods.
Test Maturity Model integration
TMMi is similar to Capability Maturity Model Integration (CMMI). It is a set of criteria that are used to
determine the fitness of an organization’s testing processes. It does not dictate how the testing is done,
rather it gives guidance in making improvements. It has five levels: Initial, Managed, Defined,
Measured, and Optimizations. Each level except Initial has a set of requirements for an organization to
be certified to it. The model is growing in popularity, as it gives a map to continuous improvement of
testing systems. While not all organizations will need to obtain the highest level of TMMi, even the
lower levels can lend insight to testing. Indeed, under TMMi, it is perfectly acceptable to be operating at
several different levels simultaneously if those levels reflect the goals of the organization.
Conclusions
So, why is there never enough time for testing? Part of this is perception. All stakeholders want, to
greater or lesser extents, an error-free application. Unfortunately, the exhaustive testing that this
would require takes too much time to be possible in all but the smallest projects. As long as the goal is
finding and fixing all defects, there can never be enough time to test. Proper risk assessment and
prioritization is necessary before testing to reduce this problem.

5. ©2015 David Consulting Group Page 5 of 5
Another factor is human behavior and habits. For example, testing can cause friction on a team,
whether done by an internal or external group of testers. Management, with its differing needs and
viewpoints, can put inappropriate pressures on the testers, as well as the developers. If the developer
does the testing, it may be considered a lower priority, and a dedicated testing team may be viewed as
outsiders or meddlers. These can all delay testing or make it less effective, so proper buy in to the
process becomes necessary.
Finally, the traditional testing process used in waterfall is suspect. With all other steps being performed
first, creating a fully developed but relatively untested application, if the testing stage finds excessive
defects, the process may need to be largely restarted, and each time this happens there will be less time
for testing. Finding the defects throughout the project reduces this possibility, though the best ways to
do this are still debated.
Sources
Jones, Capers; Bonsignour, Olivier (2011-07-19). The Economics of Software Quality. Pearson Education
Van Veenendaal, Erik; Wells, Brian (2012), Test Maturity Model integration: Guidelines for Test Process
Improvement, UTN Publishers
Morrison, John (2010-02), Software Testing – for whom do we test?,
http://www.techmanageronline.com/2010/02/software-testing-for-whom-do-we-test.html
Hower, Rick (2015-02-08) Software QA and Testing Less-Frequently-Asked-Questions,
http://www.softwareqatest.com/qat_lfaq1.html#LFAQ1_1
Cagley, Tom (2014), TMMi Best Practices (presentation)