This document discusses the practical application of software testing techniques in industry. While testing techniques are widely taught, some evidence suggests testers struggle to apply them in practice. The document examines why this is through case studies. In the studies, applying techniques like risk-based testing and automation helped reduce costs and defects. The document concludes that further research is needed to better understand how to promote adoption of techniques in industry testing.

1. Testing Techniques – Are they of any Practical Use?
Stephen Allott Bsc (Hons) MBCS CITP
ElectroMind Ltd
stephen.allott@electromind.com
Abstract solving or analytical skills are missing from the tester’s
repertoire. Or is it the case that the techniques are out
Testing techniques are taught in Universities and of date and no longer apply to testing of a modern
on many industrial training courses yet there is software development project? These issues are
evidence to suggest that, in the corporate world, explored and illustrated using case studies from the
testers often find it difficult to apply the techniques and financial services, software and travel industries.
instead rely on domain knowledge and experience
when designing tests. This is a shame as testing 2. Background
techniques are very effective at reducing the cost of
testing by providing better coverage with less effort. Many testers are swamped by the day to day pressures
When applying techniques it is important to be of having to get through so much testing in what
pragmatic and understand where they can add value to appears to be so little time. The problem is that
your test process. A business focused approach, based ineffective approaches to testing tend to create too
on risk, is essential. Commercial strength tools must many of the wrong sorts of test which leads to
be considered to support the test process although duplicated effort and consumes extra time and
home grown solutions should not be ruled out in some resources. With a large number of tests it is more
situations. An appropriate investment in effective difficult to set up, control and manage the test project.
testing will yield dividends in the longer term and so Introducing test automation is also more challenging
further research is required to understand why testing and testers generally do not have any slack time to
techniques are not much more widely adopted think about making improvements to the test process.
throughout industry.
At the unit testing level, Reid [1] has shown the
1. Introduction effectiveness of test case design techniques such as
equivalence class partitioning and boundary value
This paper provides some practical insights into the analysis. These formal approaches, and related test
application of testing techniques in industry and is case design techniques such as Pairwise, should
based on the author’s experience and observations as a provide better coverage, reduce the number of physical
senior consultant working on software testing projects test cases and help facilitate test automation. Practice
in the financial services and travel industry sectors. based models, for example TPI® – test process
This paper examines the relationship between testing improvement [2], can offer stepwise improvements and
theory and testing practice with particular reference to generate real benefits over time. The testing theory
the techniques used for test case design, automation of also suggests, for example, adopting a risk-based
test execution and test process improvement. In approach will allow testers to prioritize their tests so
general, the theory is well understood yet it appears that they run the most important tests in the time
that some testing groups often shy away from using the available. In practice it is sometimes difficult to get all
very techniques that will help them to solve their most the stakeholders to agree on the real risks and provide
immediate and pressing problems. Is it possible that, sufficient information for testers to make an informed
although the theory of test case design is well taught, choice about what to test.
there is something missing from the skill set of the
tester when he or she tries to apply the technique to a
real life problem? It may be that some problem

2. 3. Testing Techniques may feel are important. However, the beauty of using
these techniques is that having created a reduced test
According to Beizer [3] there are perhaps as many as set the team can then engage with other stakeholders to
57 different varieties of test case design techniques and explore any additional testing that may be required,
Copeland [4] has produced an excellent practical due to specific risks being identified with a particular
description of some of the most popular and useful business function for example.
techniques. Others prefer to use the term ‘techniques’
in the higher level sense of an approach or 4.2 Management information system
methodology, for example the ‘techniques’ of risk
based testing [5], or ‘this is our technique for This project aimed to consolidate financial data from
producing a test strategy’. over 15 independent corporate repositories for a major
software organisation. The technology architecture
The more common usage of the term test technique comprised an SQL data warehouse with MS Excel
relates to either black box or white box techniques. support for XML. The application characteristics
Both types require the designer to specify inputs and included over 30 data feeds, 100 locations, multiple
expected outputs in advance of execution. Black box data warehouses three profit & loss forecasts, 30
techniques [6] help to design tests based on the million rows, and several terabytes of storage.
functionality of a software component without the test
analyst having to know about the details of how the The original testers were struggling to test this high
component works. White box techniques are based on profile application. There was no overall planning,
the internal structure of a component and are mainly poor communication, a scattergun approach to the
used by developers or technical testers. testing and no evidence of risk-based testing. They
were attempting to test everything, logging defects that
Formal descriptions of test case design techniques, were extraneous, and did not keep any coverage
examples and their related coverage measures can be measures to monitor progress.
found in BS7925 [7]. The advantages of using
techniques are many and include consistency, Average Change Request/Month=56, Average Churn of Change Requests/Month =
148, Average Churn/Change Request =3.5
improved coverage, auditability and repeatability of 300 300
tests. 250 250
Churn of Change Requests
200 200
Change Request
4. Case Studies 150 150
100 100
4.1 Front end trading application 50 50
0 0
June July August September October November December January
A European bank reduced their test set for a GUI Months
based trading application from several thousand to just Change Requests Change Request Churn Bugs
a few hundred key tests using test case design
techniques. This enabled automated regression testing Figure 1. Requirements churn
for an important migration from one software release
to another. There are two popular approaches to Between June and October the application was
Pairwise testing techniques which reduce the number unstable. The number of change requests continued to
of physical test cases required: Orthogonal Arrays [8] rise and the churn, the number of times a change was
and the “all pairs” algorithm [9]. Instead of testing all made after the testers received the change request, was
the combinations for all the values for all the variables, also on the increase (figure 1).
Pairwise techniques require that you be more selective
and test just all pairs of variables. This saves time as An analysis of the defects (figure 2) by severity
you have significantly fewer tests to run. Also, 100% (impact of the defect) and priority (when will it be
all pairs coverage gives the team more confidence in fixed) concludes that 62% of severity 1 and severity 2
the testing achieved. If you later decide to automate defects, the key release criteria, had no priority set
your test execution, there are fewer scripts to develop, (356/574) leading to overload of both the developers
maintain and manage thus enabling the automation and the original test team.
process. It’s not a magic solution and may miss some
important combinations that you or other stakeholders

3. of $300,000 from the introduction of the new team. If
Severity they had taken this approach from the start the savings
would have been almost $825,000 over the year. There
1 2 3 4 Total were no defects found from January onwards, despite
Priority 1 57 81 14 152 further changes, and there were no defects reported in
Priority 2 19 52 12 1 84 production during release 1.
Priority 3 6 14 10 30 The business team was content and the test group’s
Priority 4 2 1 3 reputation was enhanced by their success. There were
No priority 68 288 114 10 480 additional savings in the sales & marketing function
due to the overall increased stability of the application.
Total 146 428 154 21 749
The test lead in this project offered to help with unit
Figure 2. Snapshot of defects by severity/priority testing and was able to get an unofficial early release
to execute all test cases in advance of the official
Additional investigation of defects by resolution code release into test. Curiously, after a while, the
(figure 3) suggests that almost a third (32%) of the developers started to produce better code, which was
total defects reported are questionable (*241/749) an interesting and useful side benefit that had not been
which is not what you want from a testing perspective. expected at the outset.
Severity 4.3 Investment banking middleware
Resolution 1 2 3 4 Total
By Design* 10 37 29 4 80 A global leader in corporate financial services
found that, despite having a good track record of
Duplicate* 6 33 7 1 47
formal acceptance testing, there was still an
External 2 16 4 0 22 unacceptable level of defects found during production.
Fixed 86 254 95 12 447 This was not confined to one or two projects but an
Not Repro* 29 56 4 0 89 across the board problem for the group which is
Postponed 12 17 6 4 39 responsible for a variety of “middle office” systems
Will Not Fix* 1 15 9 0 25 that process the bank’s business transactions after the
front end capture has taken place.
Total 146 428 154 21 749
An initiative to downsize a variety of key systems
Figure 3. Snapshot of defects by resolution code throughout the company meant that various transaction
flows had to be re-routed as systems through which
The test lead was a very experienced manager who they originally traveled were taken out of service.
selected from the original test team those that had the Having moved the flows, functionality which had been
most appropriate technical skills. They had the SQL working correctly suddenly stopped working. This is a
experience and technical ability to build their own classic regression testing problem faced by many
automation tool specifically tailored for this project. companies and not easy to solve without automation.
The tool was well documented, reusable, and
automated 80% of the existing manual test suite. The Employing more people was not an option as the
test lead did not do any test execution. What he did overhead of manually maintaining the regression test
was to involve the new test team from the beginning pack was too high. Therefore a decision was quickly
and impose control on the project to reduce the churn taken to look into automation options. At the time
in requirements and change requests. This had the there did not seem to be an automated regression
effect of stabilizing the application and the defect testing tool on the market that would help. Primarily
count dropped dramatically in November (figure 1). this was because most tools assumed that there was
some sort of GUI interface which could be used to
The original one-year project cost was projected at drive the tests (either through capture replay or
$2.6m and after 6 months approx. 50% of this cost had scripting). The majority of data was received from
been incurred with little to show in terms of a stable other systems within the company; none of them had a
application. For the period November to April, the user interface where the same data could be entered.
cost dropped to approx. $1m yielding an overall saving

4. For the people developing on these other data 4.4 Travel industry web application
capture systems they had already avoided using
commercially available tools and developed their own. Table 1 shows a section of a test maturity matrix as
Their UI was effectively a ‘dumb’ interface and easy to a result of a quick scan assessment, using the TPI®
test manually. All the serious processing was done model [10], of a testing process for a web application
‘server side’ which led to the decision being taken to in the travel industry. The TPI® model proposes that
develop their own automated regression testing tool. progress should be made (from level A, to B, to C) in
each of the 20 key areas at a steady rate and so the area
After a short study the group decided that they under the scales 1 – 5, called controlled, should all be
could not quite quantify the potential savings and so a highlighted before any attempt is made to make the
decision was taken to “just do it” based on the feeling process more efficient (scales 6 to 10) or optimized
shared by all stakeholders that this was “the right thing (scales 11 to 13 – not shown). The scales, levels, key
to do” for the organization. A small amount of seed areas and dependencies between key areas are based
money was allocated to the project to help develop the on practical experience and are fully described in the
initial concepts on a trial basis. If it worked out, which model. In this organization an attempt was made to
of course it did, additional funding would be made develop a risk-based test strategy and introduce
available to roll out the tool throughout the automation. However, the model requires that any
organization. improvement is dependent upon testing techniques.
Improvement of key area 1, test strategy, is dependent
A development resource from a NY team who had on techniques (5B). For test automation (8) the model
test tool experience was assigned to the development also shows a dependency on techniques (5A & 5B).
activities to work alongside the VP in charge of the Progress on key area 2, testing life-cycle model
project. The underlying test automation framework beyond level A is dependent on static test techniques
used was based on IBM’s STAF and STAX software (6A).
which is freely available from the Internet. The VP
put together a vision document (30 functions) which Table 1. Test Maturity Matrix (extract)
was walked through with various senior managers.
The original tool was developed in Java which is a Scale 1 2 3 4 5 6 7
popular platform in the US (there is a possibility that it Test strategy (1) A B
will be moved to C#). Life-cycle model (2) A B
Moment of involvement (3) A B
The design of the tests and the creation of test data
Estimating and planning (4) A
evolved over time; some issues remain. Large
Test techniques (5) A B
volumes of data are not necessarily complex although
further work needs to be done in this area; the Static test techniques (6) A B
applications and message flows are very complex due Metrics (7) A
to the large number of switches and parameters within Test automation (8) A B
the applications. Test environment (9) A B
Office environment (10) A
There were many benefits. Regression tests are Commitment & motivation A B
now executed unattended (overnight) and so regression
Test functions and training A B
testing is now done in days instead of weeks. It is
repeatable and tests can be executed in any Scope of methodology (13) A
environment. Errors were detected earlier thus saving Communication (14) A B
dollars. There is a dedicated test team that owns the Reporting (15) A B C
test product, scripts and regression test data. There is Defect management (16) A B C
an improved focus of key business and technical Testware management (17) A B
resources on more critical items (this allows the Test process management A B
business to focus on Acceptance Testing rather than
Evaluation (19) A
Regression Testing). Finally, maintenance costs
reduced due to improved quality of code. Low-level testing (20) A B

5. 5. Summary & Conclusions 6. References
Many testers try to use techniques such as equivalence [1] Reid S, Module Testing Techniques – which are the most
classes, boundary value analysis and decision tables effective? 1997
yet Pairwise techniques appear to be difficult to apply
without tool support. Several commercial and freeware [2] Koomen & Pol, Test Process Improvement, Addison
Wesley, 1999, ISBN 0-201-59624-5.
tools are available on the market to support Pairwise
but considerable skills are required to use them [3] Beizer, Software Testing Techniques (Second Edition),
effectively. Many companies see test automation as a Van Nostrand Reinhold, 1990, ISBN 0-442-20672-0.
“silver bullet” to solve all their testing problems.
However, if they do not have the basics in place these [4] Copeland, A Practitioner’s Guide to Software Test
automation attempts often fail. The TPI® model shows Design, Artech House, 2004, ISBN 1-58053-791-X.
the dependencies required for introducing test
automation tools and this can be a useful guide when [5] Gerrard & Thompson, Risk-based E-Business Testing,
considering automation. You must at least have some Artech House, 2002, ISBN 1-58053-314-0.
basic test process in place and an understanding of test
[6] Beizer, Black-Box Testing: Techniques for Functional
case design. Risk-based approaches are of course Testing of Software and Systems, John Wiley & Sons, 1995,
useful yet they are often difficult to implement in ISBN 0-471-12094-4.
practice and need strong leadership and consultancy
skills within the test group if they are to be [7] Reid et al, BS7925, A Standard for Software Component
successfully implemented. Process improvement Testing, British Standards Institute, 1998
techniques require a strong business case and buy-in
from everyone, including senior management, to kick [8] N. J. A. Slone maintains a library of 200 Orthogonal
start these activities. In the corporate world, Arrays www.research.att.com/~njas/oadir/index.html
ineffective approaches to testing tend to create too
[9] Pairwise techniques – The original allpairs algorithm
many of the wrong sorts of test which leads to devised by James Bach can be found at www.satisfice.com ;
duplicated effort and consumes extra time and for a list of either commercial or freeware test case design
resources. Effective testing using formal techniques and test data generation tools using several techniques
can save you time and money in the long run by including Equivalence Classes, Boundary Value Analysis,
creating fewer tests with increased coverage. and Pairwise please visit www.pairwise.org
Obviously there is some initial investment required in
training the test analysts to use the techniques and [10] TPI® is a registered trademark of Sogeti and refers
showing them how to apply them in practice in real life to a model for test process improvement which is now
situations. The reluctance of companies to adopt test in the public domain www.sogeti.nl/tpi
techniques probably stems partly from a lack of
knowledge, but perhaps also from a lack of confidence
as people seem to be too busy these days to take time
out to try new ideas.