1. Software Testing Strategies
based on
Chapter 13
1

2. Software Testing
Testing is the process of exercising a program with the
specific intent of finding errors prior to delivery
o the end user.
errors
requirements conformance
performance
an indication
of quality
2

3. Who Tests the Software?
developer
Understands the system
but, will test "gently"
and, is driven by "delivery "
3
independent tester
Must learn about the system,
but, will attempt to break it
and, is driven by quality

4. Software Testing
Goals
Types of tests
Levels of tests
Test measures
Test plan

5. Goals
Verification: Have we built the software right?
Bug-free, meets specs
Validation: Have we built the right software?
Meets customers’ needs
Are Verification and Validation different or variation of the
same idea
My argument: “meeting specs” should equal “meeting
customer needs”...
not generally true (my kitchen, satellite systems)

6. Software Testing
Goals
Types of tests
Levels of tests
Test measures
Test plan
most of this discussion focuses
on verification
(more specifically bug testing)

7. Types of tests
Black box
White box

8. Black box tests
input
output
interfa
ce
1. Does it perform the specified
functions?
2.Does it handle obvious errors in
input?
3.Ariane5 – lousy error handling
4.Classic ints vs floats, yards vs
meters
Black box should catch these if
there is adequate “test coverage”

9. Example: ordered list of ints
L=create()
L.insert(5)
L.insert(-1)
L.insert(-1)
p=L.getFirst()
print (p)
L.delete(p)
p=L.getFirst()
print(p)
p=L.getNext(p)
print(p)
p=L.getNext(p)
class OrdInts
create
getFirst
getNext
insert
delete
print
-1
-1
5
error

10. Black box tests
Advantage: black box tester≠developer is unbiased by
implementation details. e.g., Use Case testing, just work through
all the Use Cases
Disadvantage: black box tester is uninformed about
implementation details
unnecessary tests – test same thing in different way
insufficient tests – can miss the extremes, especially if actual use
follows a different pattern

11. black box tests
Code
Input
x
x
x
x
choose good distribution of input – hope good distribution of code tested

12. unnecessary tests
Input
Code
large range of input may exercise a small part of code
e.g., operator test of satellite control stations, run through
each
input and output light/key option. Testing same functions,

13. insufficient tests
Input
Code
a small range of input may exercise a large range of code
but can you ‘know’ this without knowing the code? Did we miss
the 20%

14. sufficient tests
Input
Code
complex
code
a small range of input may exercise a small but important/error-prone region

15. White box tests
Based on code
test 1
test 2

16. Example: ordered list of ints
class ordInts {
public: …
private:
int vals[1000];
int maxElements=1000;
…
}

17. Example: ordered list of ints
bool testMax()
{
L=create();
num=maxElements;
for (int i=0; i<=num; i++)
print i
L.insert(i)
print maxElements;
}

18. White box tests
Advantage:
design tests to achieve good code coverage and avoid duplication
can stress complicated, error-prone code
can stress boundary values (fault injection)
Disadvantage:
tester=developer may have bias
if code changes, tests may have to be redesigned (is this bad?)

19. Example: ordered list of ints
L=create()
L.insert(1)
L.insert(2)
L.insert(3)
class OrdInts
create
getFirst
1
2
3
insert
delete
print
…
L.insert(1001)
p=L.getFirst()
print(p)
p=L.getNext(p)
print p
…
…
getNext

20. Types of tests
Black box: test based on interface, through interface
White box: test based on code, through code
Testing strategy can/should include all
approaches!
My experience:
Black Box = non developer, outside testing idiots
in your case who?
White Box = developer, part of development process
in your case who?
Gray Box = hated, non developer within developer
organization
in your case who?

21. Software Testing
White-Box Testing
Black-Box Testing
requirements
output
.
.. our goal is to ensure that all
statements and conditions have
een executed at least once ...
21
input
events

22. White-Box Testing
Basis Path
Testingwe compute the cyclomatic
First,
complexity:
number of simple decisions + 1
or
number of enclosed areas + 1
In this case, V(G) = 4
22

23. White-Box Testing
Basis Path
Next, we derive the
Testing independent paths:
1
Since V(G) = 4,
there are four paths
2
3
4
5
7
8
6
Path 1:
Path 2:
Path 3:
Path 4:
1,2,4,7,8
1,2,3,5,7,8
1,2,3,6,7,8
1,2,4,7,2,4,...7,8
Finally, we derive test
cases to exercise these
paths.
A number of industry studies have indicated
that the higher V(G), the higher the probability
23
or errors.

24. White-Box Testing
Loop Testing
Simple
loop
Nested
Loops
Concatenated
Unstructured
Loops
Loops
24
Why is loop testing important?

25. White-Box Testing
Equivalence Partitioning
Black-Box
& Boundary Testing
Value Analysis


25
If x = 5 then …
If x > -5 and x < 5 then …
What would be the equivalence classes?

26. Black-Box Testing
Comparison Testing
Used only in situations in which the reliability of software is
absolutely critical (e.g., human-rated systems)
Separate software engineering teams develop independent
versions of an application using the same specification
 Each version can be tested with the same test data to ensure
that all provide identical output
Then all versions are executed in parallel with real-time
comparison of results to ensure consistency
26

27. Levels of tests
Unit
white
Integration
System
System integration
black

28. Testing measures (white box)
Code coverage – individual modules
Path coverage – sequence diagrams
Code coverage based on complexity – test of the risks, tricky
part of code (e.g., Unix “you are not expected to understand
this” code)

29. Levels of Testing
 Unit testing
 Integration testing
 Validation testing
 Focus is on software requirements
 System testing
 Focus is on system integration
 Alpha/Beta testing
 Focus is on customer usage
 Recovery testing
 forces the software to fail in a variety of ways and verifies that recovery is properly performed
 Security testing
 verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration
 Stress testing
 executes a system in a manner that demands resources in abnormal quantity, frequency, or volume
 Performance Testing
 test the run-time performance of software within the context of an integrated system
29

30. Unit Testing
Tests the smallest individually executable code units.
Usually done by programmers. Test cases might be
selected based on code, specification, intuition, etc.
Tools:
Test driver/harness
Code coverage analyzer
Automatic test case generator

31. Integration Testing
Tests interactions between two or more units or
components. Usually done by programmers.
Emphasizes interfaces.
Issues:
In what order are units combined?
How do you assure the compatibility and correctness of
externally-supplied components?

32. Integration Testing
How are units integrated? What are the implications of
this order?
Top-down => need stubs; top-level tested repeatedly.
Bottom-up => need drivers; bottom-levels tested repeatedly.
Critical units first => stubs & drivers needed; critical units tested
repeatedly.

33. Integration Testing
Potential Problems:
Inadequate unit testing.
Inadequate planning & organization for integration testing.
Inadequate documentation and testing of externally-supplied
components.

34. Stages of Testing
Testing in the Large
 System Testing
 End-to-End Testing
 Operations Readiness Testing
 Beta Testing
 Load Testing
 Stress Testing
 Performance Testing
 Reliability Testing
 Regression Testing

35. System Testing
Test the functionality of the entire system.
Usually done by professional testers.

36. Realities of System Testing
Not all problems will be found no matter how thorough or
systematic the testing.
Testing resources (staff, time, tools, labs) are limited.
Specifications are frequently unclear/ambiguous and changing
(and not necessarily complete and up-to-date).
Systems are almost always too large to permit test cases to be
selected based on code characteristics.