February 8, 2008
This document discusses integration testing which is an important
stage in the overall testing processes. Various diﬀerent approaches to
integration testing will be discussed including their individual advan-
tages and disadvantages plus there is a brief overview of two diﬀerent
software testing tools that can be used help automate testing tasks.
“testing is a vital component of a quality software process, and
is one of the most challenging and costly activities carried out
during software development and maintenance”
Software testing as a whole is an essential part of the software development
processes (often accounting for over 50% of the total cost of a project) it
is however an area that is often underestimated during planning. If a project
is under signiﬁcant time pressure it is the testing stage of the development
process that is aﬀected most signiﬁcantly. It is therefore very important
that software developers throughly understand the diﬀerent stages of testing
and the diﬀerent techniques that are used.
Integration Testing is the testing of software components after they have
been integrated and this form of testing will be the focus of this dissertation.
Integration testing assumes that the individual software components have
been successfully unit tested1 and is an important form of testing because
“Some bugs that can be discovered as the units are integrated are impossible
to ﬁnd when testing isolated units”. It is also worth noting that often
diﬀerent software companies collaborate on projects and it is critical that
once the individual companies software are integrated together it is tested
There is one well known example that is often used to explain why testing
is important and it is perfect for illustrating what happens when integration
testing is not performed correctly.
The Mars Climate Orbiter was a spacecraft developed by NASA designed
to be launched from Earth, travel to Mars, move into a stable orbit with
a periapse2 altitude of 160km and once there record data regarding the
atmospheric conditions of the planet. The Orbiter was launched success-
fully on 11 December 1998 however while moving into orbit around Mars (on
September 23,1999) NASA lost contact with the spacecraft.
Testing of individual software components.
The position in an orbital path that is closest to the primary body.
Figure 1: Mars Climate Orbiter
The investigation board found that the cause of the accident was a sim-
ple software error. Speciﬁcally one module outputted thruster performance
data in imperial measurements however in the software documentation it was
stated that this information should be in metric units.
During the nine month journey to Mars, NASA made several propulsion ma-
neuvers based upon data which was not in the correct format. This caused
small errors to be introduced in the trajectory during the journey meaning
that by the time the craft reached Mars, it was 170 km lower than was orig-
inally planned. As a result of this the craft was quickly destroyed due to
hitting the atmosphere of the planet.
This should never have happened, although the individual software units
would appear to be working correctly, appropriate integration testing should
have been able to discover the error between the units. This small error
which was not picked up due to insuﬃcient testing, cost NASA over $300
2 Decomposition-Based Integration Testing
Decomposition-based integration testing is the most popular form of integra-
tion testing, it is widely publicised and in general when integration testing is
mentioned in an article or book they are referring to this form of testing. This
approach to integration testing is “based on the functional decomposition3 of
the system tested”. There are four major techniques that can be used in
decomposition-based integration testing, speciﬁcally: top-down, bottom-up,
sandwich and big bang. The major diﬀerences in all these approaches is the
order that the individual units of the system are integrated and tested. As
stated previously, integration testing assumes that the individual units have
been successfully unit tested and therefore the goal of this testing is to dis-
cover any errors in the integration of the individual units.
This section will discuss these four diﬀerent approaches to decomposition-
based integration testing with use of the example backup program which is
detailed in table 1 and shown as a decomposition tree in ﬁgure 2
Figure 2: Decomposition Tree Example
Resolving a functional relationship into its constituent parts.
Module Name Module Description
Main This is the ﬁrst module that will be run,
it provides the initial interface.
Backup Provides the interface for the various
types of backup that the user can perform.
USB Backup Called if the user requests a USB Backup,
it starts the backup process.
Mount USB Drive Mounts the USB Drive, once the mount-
ing process has completed it calls the
backup to USB module.
Backup to USB Uploads all the necessary ﬁles to the USB
Update Last USB Backup Updates the lastUSB ﬁle with the current
date and time.
Server Backup Called if the user requests a Server
Backup, it starts the backup process.
Remote Connect Connects to the remote server, once suc-
cessfully connected it calls the Upload
Upload Backup Uploads all the necessary ﬁles to the server
Update Last Server Backup Updates the lastServer ﬁle with the cur-
rent date and time.
Manual Backup This module is used if the user backups
Update Manual Backup Date Updates the lastManual ﬁle with the cur-
rent date and time.
Recent Backups Displays the date and time of all the last
backups (by calling the appropriate mod-
Last USB Backup Returns the date and time stored in the
Last Manual Backup Returns the date and time stored in the
Last Server Backup Returns the date and time stored in the
Table 1: Backup Program Example
The Top-down method begins by testing the main program unit (the root4 of
the tree) with one lower level node. Any other lower level units/nodes that
may be connected should created as a stub5 . In the example used here (see
ﬁgure 3 for the colour key and ﬁgure 4) the ‘Main’ unit would ﬁrst be tested
with ‘Backup’ with stubs created for ‘Recent Backups’, ‘Manual Backup’,
‘USB Backup” and ‘Server Backup’. The stub ﬁles would be created with
the absolute minimum amount of information needed. for example the ‘Re-
cent Backup’ stub could be created to simply return a speciﬁc set of dates
with no dynamic lookups at all.
Figure 3: Colour Code Key
Once the appropriate stub ﬁles have been created and everything has been
integrated, the tester simply tests the program as if it were a stand-alone
program therefore any appropriate testing techniques could be used to look
for faults. If the tests are successful then the project can continue by pro-
ducing real code to use in place of one of the stubs that were used in the
previous test. There is no set way to traverse through the decomposition
tree. Breadth-ﬁrst or Depth-ﬁrst are both perfectly acceptable methods.
The topmost node in a tree.
A piece of ‘throw-away’ code that emulates the unit in question.
Figure 4: Top Down Integration Testing: Stage 1
It is up to the tester to decide how many stubs should be replaced before
the next test is performed (see ﬁgure 5). The more stubs that are replaced
before a test, the harder it would be to localise a fault; however if a test was
performed after every new unit was produced then the number of tests that
have to be performed rises quickly. This rise in test numbers could cause
problems due to scheduling or associated costs.
The major advantage of this top-down method is the fact that a system
prototype can be developed early on in the project process. This is a very
attractive property of this integration testing technique as usually the client
of a software company will have little or no software engineering knowledge
and therefore if the client cannot see something tangible then it would be
easy for the client to assume that little or no work is being completed. In
addition to this, having an early prototype will give the client a better feel
for how the ﬁnal product will look which is useful as any changes that the
client might think of will appear early in the development process (where it
would be easier to implement).
One disadvantage of this method is the fact that the programmers will have
to produce a large number of stubs, this means extra work is needed on top
of producing the actual system by producing units that will later be thrown
Figure 5: Top Down Integration Testing: Stage 2
Assuming the developer is using the decomposition-tree model the formula
below can be used to calculate the number of stubs that must be produced.
number of stubs = (number of nodes − 1)
In our backup example, that means that the software engineers will have to
spend time developing an additional 15 units that will be simply thrown-
away as the project progress. In large scale projects the amount of time and
therefore money spent developing huge numbers of stubs could easily become
The bottom-up method begins by testing one of the leaves6 of the program
with it’s parent7 node ﬁrst. Any higher nodes that are directly connected to
the nodes being tested are constructed as drivers.8 In our example (ﬁgure 6)
the ﬁrst nodes that we would test would be ‘Update Last USB Backup’ and
‘Backup to USB’ with a ‘Mount USB Drive’ created as a driver. The driver
would be created with the minimum amount of code needed.
A type of node that has no child (directly connected) nodes.
A node that has a child is called the child’s parent node.
a piece of ‘throw-away’ code that emulates the unit in question.
Figure 6: Bottom Up Integration Testing: Stage 1
In exactly the same way as the top-down method, it is up to the user how
to continue testing and how many nodes to create before performing another
test (see ﬁgure 7). This has the exact same issue of number of tests vs ease
of fault isolation.
The major advantage of this method of integration testing is that the pro-
gram itself is fully functional9 at every stage. This is in contrast from the
top-down method, where as a a prototype can be created at an early stage,
but it will have little or no functionality e.g. it may have a button for ‘USB
Backup’ but that button will not do anything. However the obvious disad-
vantage to this approach is the inability to create an early prototype for the
client. This could easily lead to problems down the line, such as the client
may question the progress of the project or when the prototype is ﬁnally
produced the client might decide on changes that are diﬃcult to implement
at that late stage of the project.
Although drivers are diﬀerent from stubs (e.g. there are less drivers, but
they are more complicated to produce) there is still the same problem of
producing large quantities of ‘throw-away’ code. This problem will grow
quickly in larger projects.
the units will be fully functional, however the entire program will not immediately
have all the functionality in the speciﬁcation.
Figure 7: Bottom Up Integration Testing: Stage 2
Assuming the developer is using the decomposition-tree model the formula
below can be used to calculate the number of stubs that must be produced.
number of drivers = (number of nodes − number of leaves)
In our backup example this will mean that the software engineers will have
to spend time developing an additional ten units of code that will simply be
The sandwich method attempts to solve some of the problems associated
with the previous two methods by combining them into a single method.
The initial leaf node and corresponding parent is created (along with any
necessary drivers) plus the root node along with a corresponding child (and
any stubs) are created. This way the program is tested from both the top of
the tree moving down and the bottom of the tree moving up.
In our example (ﬁgure 8), this would mean that initially the ‘Main’, ‘Backup’,
‘Update Last USB Backup’ and ‘Backup to USB’ would be created, with ‘Re-
cent Backups’, ‘Manual Backup’, ‘USB Backup’ and ‘Server Backup’ created
as stubs plus ‘Backup to USB’ created as a driver. These nodes would then
be tested and if successful the tester will the start traversing the tree in both
directions (from the top and the bottom, see ﬁgure 9).
Figure 8: Sandwich Integration Testing: Stage 1
This method combines the two major advantages of the previous two meth-
ods (being able to produce an early prototype and producing a functional
program at each stage of development). In addition to this, by using the
sandwich method the number of stubs and drivers that have to be produced
are reduced. However The signiﬁcant disadvantage of the sandwich method
is that fault isolation can be diﬃcult, especially in larger projects.
2.4 Big bang
“In its purest (and vilest) form, big-bang testing is no method at
all - ‘Let’s ﬁre it up and see if it works!’ It doesn’t of course”
The big bang method is the least systematic method of integration testing,
the idea for this method is to simply go ahead and produce all of the indi-
vidual nodes/units of the program and only once they have all been created
do you perform the integration tests. This removes the problem of having
to waste time producing ‘throw-away’ code, however quite obviously even
moderately sized projects fault isolation is very diﬃcult. This method has
many problems associated with it and only one minor advantage such that
it is considered to be “such transparently bad thinking”. Despite all of
this it is still “probably is the most commonly used method for the integration
Figure 9: Sandwich Integration Testing: Stage 2
purpose”, this popularity is likely to be because many project managers
“just want to be able to report to the management that ‘Coding is complete’
as soon as possible, even if nothing works”.
3 Call Graph Based
The methods that have been discussed so far are based upon the functional
decomposition trees rather than the actual structure of the program, also
the major disadvantage in the previous10 testing techniques is the fact that it
is necessary to produce large quantities of ‘throw-away’ code in the form of
stubs and drivers. call graph based integration is a type of testing designed
to solve these two problems.
By using a call graph the “internal structure of the program is clearly rep-
resented” and similar incremental methods to the approaches discussed
above (to-down, bottom-up etc) can be used. However the big diﬀerence be-
tween this and the previous techniques is that no stubs or drivers are needed.
All the testing will be performed on actual units of fully functional code i.e
the program is constructed/produced before integration testing takes place.
This sounds similar to the big bang approach of integration testing however
by ensuring that the units that are tested are split up into smaller groups
excluding the ‘big bang’ method.
we can reduce the diﬃculty of fault isolation to a manageable level. The
two main approaches to this form of testing will be discussed with use of an
example door lock program which is detailed in table 2 and shown as a call
graph in ﬁgure 10.
Figure 10: Call Graph Example
Module Name Module Description
Main This is the ﬁrst module that will be run.
Display Displays a list of users that have recently
opened the door plus displays a message
if a user’s card is valid or invalid.
Override This is the emergency door override mod-
ule, it can be activated by an admin to
open the door without needing a card.
Card Reader This module reads the card and checks to
see if the user is valid or not by checking
with the database, if it is valid it calls the
Database This module contains the list of all users
on the system.
Lock Controller When this method is called it unlocks the
door (by calling the unlock module) then
after a set amount of time it re-locks the
door (by calling the lock module).
Unlock Unlocks the door and calls the light mod-
Lock Simply locks the door
Lights This module turns on the lights (if they
are not already on).
Table 2:Door Lock Program Example
Pair-Wise integration testing restricts the testing session to a pair of units
in the call graph (see ﬁgure 11). This ensures that faults can be easily
isolated while still working without stubs or drivers. A disadvantage to this
method is that there needs to be as many test sessions are there are edges.11
However, because no time is wasted in producing ‘throw-away code’ (in the
form of drivers or stubs) this method can be signiﬁcantly quicker than the
decomposition based methods. There is no speciﬁc traversal strategy that
has to be followed when using this technique so long as in the end each edge
has been successfully tested.
A representation of a relationship rendered as a straight or curved line.
Figure 11: Call Graph: Pair-Wise Example
Neighbourhood Integration Testing attempts to reduce the number of testing
sessions necessary in order to fully test the software system while still trying
to keep the diﬃculty of fault isolation low. In order to do this the tech-
nique restricts the testing session to the neighbourhood of the source node12
where the neighbourhood of a source node includes every other node that is
directly connected to the that source node (see ﬁgure 12). Each source node
(with it’s corresponding neighbourhood) is tested ensuring that every node
(including sink nodes13 ) is tested. This greatly reduces the number of testing
sessions there are, as the number of testing sessions is equal to the number of
neighbourhoods that exists which can be calculated using the formula below.
N umber of N eighbourhoods = (number of nodes − number of sink nodes)
A type of node that has outgoing relations to other nodes (similar to a parent node
in a decomposition tree).
A type of node that has no outgoing relations (similar to a leaf in a decomposition
In our example this will mean 5 tests sessions will have to be run which is a
big improvement on the pair-based method which would have required 9 test
sessions (one for each edge). This method attempts to balance the number of
test sessions required (and therefore the amount of time required to test the
system) and the diﬃculty of fault isolation (and therefore the amount of time
wasted trying to ﬁnd the fault). However in smaller software projects the
size of the neighbourhood will be relatively low and therefore fault isolation
would not be an issue, in larger projects the size of a neighbourhood could
easily become large enough so that it would be diﬃcult to isolate a single (or
collection of) faults.
Figure 12: Call Graph: Neighbourhood Example
An additional problem that arises due to the fact that nodes overlap neigh-
bourhoods. If fault is found and corrected in a node that is in several neigh-
bourhoods, due to the fact that the node’s code will have been changed it
will be necessary to retest every neighbourhood that the node in question
can be found in. This form of testing is called “Regression Testing” and
is performed in every testing technique once a fault has been corrected to
ensure that the corrected code has not caused faults elsewhere.
4 Software Testing Tools
In general, Software testing tools exist to automate the testing process. Es-
sentially testing tools create many “little programs that automatically check
to ensure that your program behaves as expected”. This greatly reduces
the amount of time needed to test a system (be it a unit test, integration test
or even system test). In general these tools have the ability to run many tests
automatically and summerise the results into a readable format. There are
two small disadvantages that are worth mentioning, ﬁrst of all it is necessary
to write more code that will not be used in the ﬁnal program (similar to
the problem with stubs and drivers) however the time saved by automating
software tests greatly outweighs the time spent writing the necessary code.14
Secondly testing tools themselves like any other program are never totally
bug free and while using one to test a system, there is the assumption that
the results of the tests are accurate which may not be the case.
Junit is a generic unit testing framework for the Java programing language it
was created by Kent Beck and Erich Gamma in 1998 and is used for regres-
sion, unit and integration testing. As it is a generic framework it is supported
by all the major java IDEs (for example BlueJ, see ﬁgure 13). This allows
the tester to automate the testing process, performing any number of tests
with a click of a button.
Below is an example JUnit test case15 , it is testing a simple item basket
program to check to see if the totalCost method is working correctly.
public class BasketTest extends TestCase
Obviously for very small programs this might not be true, but for any realistically
sized project this statement holds.
The test case is to demonstrate the code used with JUnit, not to demonstrate inte-
Figure 13: BlueJ: JUnit screenshot
private Basket myBasket;
private Item dvd1
private Item dvd2
* This method runs before any test case method
protected void setUp()
myBasket = new Basket();
dvd1 = new Item("Hot Fuzz", 9.99);
dvd2 = new Item("Sean of the Dead", 3.99);
public void testTotalCost()
double totalCost = myBasket.totalCost();
double confirm = dvd1.price() + dvd2.price();
/* assertEquals checks to see if the two
* variables are equal, if they are it does nothing
* if however they are not it throws an unhandled exception
Test Cases can be much larger and more complicated than the example above,
also it is possible to create ‘Test Suites’ which are a collection on individual
test cases which can all be run simply by running the test suite, this is very
useful as you can create collections of test cases in diﬀerent suites and run
the group that is needed. Below is a small example of a test suite
public class ShoppingTestSuite
public static Test suite()
JUnit is a testing tool designed to be easy to use, so that the average program-
mer (who may not have much experience in testing) can write and perform
tests while still allowing the user to create complicated test cases and suites.
“Software integration testing; quicker, easier and more reliable
Eggplant is software testing tool produced by Redstone. Rather than in-
teracting with the underlying software code it attempts to recreate the the
user experience and is designed to work complimentary with other code level
In essence this testing tool simulates a real user and can be set up to perform
automated tests on the user interface of the program because it is based on
performing functions on the user interface. It is not dependent on any one
programing language or even operating system. Eggplant uses a scripting
Figure 14: Eggplant screenshot
language called Sensetalk which is “a very high level, object oriented, inter-
preted scripting language, based on a modular, extensible architecture.”
The language is designed so that it is very easy to use (hence the “very
high level”) and Eggplant provides an interface for the script creation that
identiﬁes the elements in the interface that can be tested (for example buttons
and text ﬁelds) which makes it a straight forward process to script actions
for testing the elements. Below is the example16 sample script for testing a
WaitFor 5, "SafariMenu"
TypeText "Very good post! I aggree totally with what you were sayingt"
TypeCommand "Q" -- Quit Safari
The test case is to demonstrate the code used with Eggplant, not to demonstrate
In summary, Software testing is a vital part of the software development pro-
cess even though it is the most likely part that will be cut out (or at least
signiﬁcantly reduced) due to time and cost problems. Integration testing is a
signiﬁcant part of the overall testing activity and is very useful in discovering
faults with the program that may otherwise go overlooked.
The main goal of integration testing is to test how previously tested units
(be they small modules or entire systems) interact with each other and un-
cover problems that would not be apparent during the unit testing stages.
As we have seen with the Mars Climate Orbiter example, not performing full
integration testing on all systems can have disastrous consequences (in the
example case, the consequences costed NASA over $300 million)
The two major types of integration testing that have been discussed were ‘de-
composition tree based’ and ‘call graph based’. These two types of testing
have their own advantages and disadvantages and both types have a number
of diﬀerent approaches to tackling the problem of integration testing.
Decomposition Tree based integration testing primarily focused on the order
that the individual units were developed and tested, the major disadvantage
of this form of testing was the creation of throw-away code in the form of
stubs and drivers.
Call Graph based integration testing has the advantage of not requiring any
additional throw-away code to be produced, however that does make fault-
isolation a problem especially in larger software projects.
There are a vast number of software testing tools available for use (both
free and commercial tools) that are designed to help automate the testing
process. The two tools that were discussed in this dissertation were JUnit,
which is a generic unit testing framework for Java (therefore implemented in
many diﬀerent IDEs) and Eggplant which is a commercial tool produced by
Redstone which is designed to recreate the user experience. These two tools
are both very diﬀerent from each other, JUnit aims speciﬁcally at testing
Java code where as Eggplant aims at testing via the programs user interface.
 BURNSTEIN, I. 2002. Practical Software Testing: A Process-Oriented
 MYERS, G. 2004. The Art of Software Testing
 KAN, S. 2002. Metrics and Models in Software Quality Engineering
 CRAIG, R; JACKIEL, C. 2002. Systematic Software Testing
 Mars Climate Orbiter Mission Overview. Accessed 22/12/07
http: // mars. jpl. nasa. gov/ msp98/ orbiter/ mission. html
 1999. Mars Climate Orbiter Mishap Investigation Board Phase I Report.
 Mars Climate Orbiter Fact Sheet. Accessed 22/12/07
http: // mars. jpl. nasa. gov/ msp98/ orbiter/ fact. html
 JORGENSEN, P. 2002. Software Testing: A Craftman’s Approach
 BEIZER, B. 1984. Software System Testing and Quality Assurance
 KANER, C; FALK, J; NGUYEN H. 1993 Testing Computer Software
 DHILLON, B. 1987. Reliability in Computer System Design
 TSAO, H; Wu, Y. 2003 Testing and Quality Assurance for Component-
 BECK, K. 2004. JUnit Pocket Guide
 Redstone Software. Accessed 07/01/07,
 SenseTalk Reference Manual. Accessed 07/01/07,
1 Mars Climate Orbiter. Accessed 07.01.07,
2 Decomposition Tree Example, created by Thomas Bradley with Gliﬀy
2 Colour Code Key, created by Thomas Bradley with:
4 Top Down Integration Testing: Stage 1, created by Thomas Bradley with:
5 Top Down Integration Testing: Stage 2, created by Thomas Bradley with:
6 Bottom Up Integration Testing: Stage 1, created by Thomas Bradley with:
7 Bottom Up Integration Testing: Stage 2, created by Thomas Bradley with:
8 Sandwich Integration Testing: Stage 1, created by Thomas Bradley with:
9 Sandwich Integration Testing: Stage 2, created by Thomas Bradley with:
10 Call Graph Example, created by Thomas Bradley with:
11 Call Graph: Pair-Wise Example, created by Thomas Bradley with:
12 Call Graph: Neighbourhood Example, created by Thomas Bradley with:
13 BlueJ: Junit screenshot, taken of BlueJ version 2.2.1 with example code
written by Thomas Bradley
14 Eggplant screenshot, taken of Eggplant Version 4.01 (1101) with example
code written by Thomas Bradley