The document discusses the importance of systems testing in software development, highlighting the criteria for correctness and the challenges of exhaustive testing. It details two main testing strategies: black box and white box testing, along with methods like unit and integration testing to ensure software robustness. The document emphasizes that a structured approach to testing can improve the effectiveness of debugging despite the time and effort it requires.

1. Testing
Michael Heron

2. Introduction
• Getting a program running is only the first step.
• Getting it working is the next… and this is often more costly.
• Having a structured approach to systems testing is one of the
key requirements for building robust and correct software.

3. Testing
• Testing helps to ensure that a program is correct.
• But what do we mean by correct?
• There are a range of criteria that we can apply to any given
program to judge its correctness.

4. Testing
• Correctness:
• It must handle all the requested functionality for all
valid data.
• It must not generate unexpected behaviour.
• It must respond appropriately to all anticipated data.
• It must have sufficient error checking to ensure the
program will not produce unexpected results in the
case of being given unexpected input.

5. Testing
• This is a very tall order for most software
projects.
• Sometimes it’s even a tall order for very simple
projects.
• It is impossible to exhaustively test a program of
any significant complexity.
• There are just far too many variables.
• Not all of the variables are found in the program itself.

6. Testing
• The goal of testing is to reveal inadequacies in a
piece of software.
• This is an important point.
• Programmers are often not the best people to
write the test cases for their own programs.
• Many large companies have dedicated testing
departments to ensure a minimalisation of bias.

7. Testing
• As a developer, it is tempting to write biased test cases that
code is very likely to pass.
• A good test however is one that reveals a problem.
• A structured approach to testing involves creating a number
of strong test cases for a program and comparing the actual
results against the expected results.

8. Test Sets
• For each test case in a test set, documentation
should include:
• The exact input data and the conditions under which
the data were inputted.
• The purpose of the test.
• The exact output expected from the program with the
given test data.
• Once the test is completed, the actual output is
compared to the expected output.

9. Test Strategies
• The kind of test cases used in a given test set depends on the
kind of testing strategy adopted.
• There are two main families of testing:
• Black box testing
• White box testing
• We shall look at these in turn.

10. Black Box Testing
• Black box (or functional) testing is centred around creating a
set of test cases based on sample data that is representative
of all data.
• Black box testing requires no knowledge of the program
architecture… the program is a ‘black box’ that we cannot see
into.
• Think of Searle’s Chinese room.

11. Black Box Testing
• In Black Box testing, all we consider are the inputs and the
outputs.
• Because it is impossible to exhaustively test a program, we
must choose data that is representative of other data.
• We must also consider special cases where errors are
particularly likely.

12. Black Box Testing
• Consider a program designed to multiply two numbers
together.
• If it works for 1 * 1, and 2 * 2, you may think it is safe to think
it will work for any positive integer.
• But this is not always true.
• Consider the following table

13. Black Box Testing
Number 1 Number 2 Result
1 1 1
2 2 4
3 3 27

14. Black Box Testing
• It is important that you choose a large enough sample from
any representative set.
• It is also important to check for specific kinds of input:
• Boundaries
• Maximum and minimums
• Invalid input

15. Black Box Testing
• Boundaries are those areas where errors are very likely to
creep in because of the transition between sets.
• Between positive and negative
• Between valid and invalid
• These areas are often fruitful tests because programmers
often make errors in boundary checking:

16. Examples
• If (a = a) {
do_something();
}
vs
if (a == a) {
do_something();
}

17. Examples
• If (a > b) {
do_something();
}
vs
if (a >= b) {
do_something();
}

18. Examples
• For (int I = 0; I < 100; i++) {
do_something_with (i);
}
vs
for (int I = 0; I <= 100; i++) {
do_something_with(i);
}

19. Black Box Testing
• It is very easy for these kind of errors to slip into a program.
• The ramifications can often be very subtle.
• A good test case will take numerous samples from a
representative set and also check at the boundaries of each
set.
• Remember – good test cases are those that reveal errors.

20. White Box Testing
• A counterpoint to black box testing is White Box
(or structural) testing.
• In White Box testing we do not care about the
matching of expected output to actual output.
• We do care that every statement in a program
gets executed at some point.
• White Box testing requires significant knowledge
of a program’s architecture.

21. White Box Testing
• In White Box testing, test cases are developed to ensure every
path of execution through a program is touched.
• Where there are loops and conditionals, test cases should be
devised to ensure that these are executed at some point with
valid test data.

22. White Box Testing
• Consider the following piece of code:
public static void main (String args[]) {
int a, b;
a = Integer.parseInt (args[0]);
b = Integer.parseInt (args[1]);
if (a < b); {
System.out.println (“A is less than B”);
}
if (a > b) {
System.out.println (“A is greater than B”);
}
else {
System.out.println (“A is equal to B”);
}
}

23. White Box Testing
Value of
A
Value of
B
Expected Actual
1 2 A is less
than B
A is less
than B
A is equal
to B
2 1 A is greater
than B
A is less
than B
A is greater
than B

24. White Box Testing
• White box testing allows us to test every conditional to ensure
that it gets called with the right sets of data.
• White box testing is not a replacement for black box testing.
• The two approaches work best as complements to each other.

25. Complementary Approaches
• Why use White Box Testing in addition to Black Box Testing?
• Logic errors are usually made when coding for ‘special cases’.
The correct execution of these logic paths should be tested.
• Assumptions about execution paths can be incorrect, and white
box testing shows this.
• Errors are random, and just as likely to be on an obscure path as
a mainstream path.

26. Complementary Approaches
• Why use Black Box Testing in addition to White Box Testing?
• We can check to see how tightly a method conforms to the
software specification
• We can ensure reliability of user input.
• We cannot assume that just because an execution path is
followed that the logic will be correct. Black Box Testing
highlights this.

27. Unit Testing
• Most programs are more complex than the ones shown in this
lecture.
• How is it possible to test these using proper test cases?
• It is not possible to test most complex programs as a whole.
• It’s just too complicated.

28. Unit Testing
• Instead, we use a technique called unit testing.
• We take each of the methods or classes in turn and test them
with all valid sets of data in a harness.
• A harness is just a small program that calls the method with
various appropriate sets of data and outputs the returned values.

29. Unit Testing
• This takes advantage of the modularity of modern software
development.
• Once a unit has been tested with white and black box testing
and has deemed to be ‘correct’, then it is usually safe to
assume the unit will work correctly when called by the
program itself.
• However, this is not always true!

30. Integration Testing
• However, we can apply a more structured approach to this
also with Integration Testing.
• We start off with a unit that has no dependencies on other
units and test that to ensure that it works.
• Then we test any units that make use of that unit.

31. Integration Testing
• Once we have tested these, then we use Integration Testing to
test them together to ensure that the result is correct.
• This is a time consuming, but very valuable process.
• In this way, a whole program can be broken up into chunks
and then reintegrated with a good degree of confidence.

32. Summary
• Testing is vital in ensuring correctness in programming.
• It is however very time consuming and often frustrating.
• It is possible to develop a structured approach to testing that
makes debugging an easier process.