Software Testing Definitions and Approaches

Chanderprabhu Jain College of Higher Studies & School of Law
Plot No. OCF, Sector A-8, Narela, New Delhi – 110040
(Affiliated to Guru Gobind Singh Indraprastha University and Approved by Govt of NCT of Delhi & Bar Council of India)
Software Testing
BCA – V
Paper Code 307

Definition of Software Testing
Software testing is the process of
executing a software system to determine
whether it matches its specification and
executes in its intended environment.

“Program testing can be a very effective
way to show the presence of bugs, but it
is hopelessly inadequate for showing
their absence” [Dijkstra, 1972]

Why Test?
Q: If all software is released to
customers with faults, why should we
spend so much time, effort, and money
on testing?

Black-Box / White-Box Testing
• Black-box tests are driven by the program’s
specification
• White-box tests are driven by the program’s
implementation
Specification ProgramBlack-
box tests
White-box
tests

Black Box Testing
• Checks that the product conforms to
specifications
• Cannot determine how much code has
been tested
Program
Omissions detected
by black-box tests

White Box Testing
• Allows tester to be sure every statement
has been tested.
• Difficult to discover missing functionality.
Commissions detected
by White-box testsProgram

Basic Definitions
• Test case: specifies
– Inputs + pre-test state of the software
– Expected results (outputs an state)
• Black-box testing: ignores the internal logic of the
software, and looks at what happens at the interface
(e.g., given this inputs, was the produced output
correct?)
• White-box testing: uses knowledge of the internal
structure of the software
– E.g., write tests to “cover” internal paths

Testing Approaches
• Will look at a sample of approaches for
testing
• White-box testing
– Control-flow-based testing
– Data-flow-based testing
• Black-box testing
– Equivalence partitioning

Control-flow-based Testing
• A traditional form of white-box testing
• Step 1: From the source, extract a CFG
• Step 2: Design test cases to cover certain elements of
this graph
– Nodes, edges, paths
• Basic idea: given the CFG, define a coverage
target and write test cases to achieve it

Statement Coverage
• Traditional target: statement coverage
– Need to write test cases that cover all nodes in
the control flow graph
• Intuition: code that has never been executed
during testing may contain errors
– Often this is the “low-probability” code

Branch Coverage
• Target: write test cases that cover all
branches of predicate nodes
– True and false branches of each IF
– The two branches corresponding to the
condition of a loop
– All alternatives in a SWITCH statement

Branch Coverage
• Statement coverage does not imply branch
coverage
• Can you think of an example?
• Motivation for branch coverage: experience
shows that many errors occur in “decision
making” (i.e., branching)
– Plus, it subsumes statement coverage.

Example
• Same example as before
• Test case #1: follows path 1-
2-exit
• Test case #2: 1-2-3-4-5-7-8-
2-3-4-5-7-8-2-exit
• Problem?
1
2
3
4
5 6
7
8
T F

Achieving Branch Coverage
• For decades, branch coverage has been
considered a necessary testing minimum
• To achieve it: pick a set of start-to-end paths in
the CFG, that cover all branches
– Consider the current set of chosen paths
– Try to add a new path that covers at least one
edge that is not covered by the current paths
• Then write test cases to execute these paths

Some Observations
• It may be impossible to execute some of the
chosen paths from start-to-end
– Why? Can you think of an example?
– Thus, branches should be executed as part of
other chosen paths
• There are many possible sets of paths that
achieve branch coverage

Example
if x ≤ y
x=
y
T
F
if x == y
z=
1
z=
0
FT
Candidate start-to-end paths:
(1) green path
(2) red path
% branch coverage?
Problem?

Data-flow-based Testing
• Basic idea: test the connections between variable
definitions (“write”) and variable uses (“read”)
• Starting point: variation of the control flow graph
– Statement nodes represent one statement
• Set Def(n) contains variables that are defined at
node n (i.e., they are written)
– The definitions at node nSet Use(n): variables that
are read
– The uses at node nStudies & School of Law

Remember Reaching Definitions
• Definition A statement that may change the
value of a variable (e.g., x = i+5)
• A definition of a variable x at node k reaches
node n if there is a path from k to n, clear of a
definition of x.
k
n
x = …
… = x
x = …

Def-use Pairs
• A def-use pair (DU pair) for variable x is a pair of
nodes (n1,n2) such that
– x is in Def(n1)
– The definition of x at n1 reaches n2
– x is in Use(n2)
• In other words, the value that is assigned to x at n1
is used at n2
– Since the definition reaches n2, the value is not “killed”
along some path n1...n2.

Examples of Reaching Definitions
Assume y is an input variable
1 s:= 0;
2 x:= 0;
3 while (x<y) {
4 x:=x+3;
5 y:=y+2;
6 if (x+y<10)
7 s:=s+x+y;
else
8 s:=s+x-y;
1
2
3
4
5
6
7 8
9
1
0
What are the def-use pairs for s?
What are the def-use pairs for x?

Data-flow-based Testing
• Data-flow-based coverage target: DU pair coverage
– Compute all DU pairs, and construct test cases that cover
these pairs. HOW DO WE COMPUTE DU PAIRS?
• Several coverage targets (criteria), with different
relative strength
• Motivation for data-flow-based testing coverage:
see the effects of using the values produced by
computations Focuses on the data, while control-flow-
based testing focuses on the control

Finally, the targets (criteria):
All-defs criterion
• If variable x is in Def(n1), the all-defs criterion requires the
test data to exercise at least one path free of definition of x
which goes from n1 to some node n2 such that (n1,n2) is a DU
pair for x.
– Remember, x is defined at n1,
– The definition of x at n1 reaches n2, and
– x is used at n2

All-uses criterion
• If variable x is in Def(n1), the all-uses criterion
requires the test data to exercise at least one
path free of definition of x which goes from
n1 to each node n2 such that (n1,n2) is a DU
pair for x.

All-DU-paths criterion
• If variable x is in Def(n1), the all-DU-paths
criterion requires the test data to exercise
each path free of definition of x which goes
from n1 to each node n2 such that (n1,n2) is a
DU pair for x.
• So what is the relative strength of the three
criteria: All-defs, All-uses, All-DU-paths?

All-defs, all-uses, all-du-paths
Assume y is input
1 s:= 0;
2 x:= 0;
3 while (x<y) {
4 x:=x+3;
5 y:=y+2;
6 if (x+y<10)
7 s:=s+x+y;
else
8 s:=s+x-y;
}
1. Design test cases that cover all-uses

Black-box Testing
• Unlike white-box testing, no knowledge
about the internals of the code
• Test cases are designed based on
specifications

Equivalence Partitioning
• Basic idea: consider input/output domains and partition them into
equiv. classes
– For different values from the same class, the software should
behave equivalently
• Use test values from each class
– Example: if the range for input x is 2..5, there are three classes:
“<2”, “between 2..5”, “5<”
– Testing with values from different classes is more likely to uncover
errors than testing with values from the same class

Equivalence Classes
• Examples of equivalence classes
– Input x in a certain range [a..b]: this defines three classes “x<a”,
“a<=x<=b”, “b<x”
– Input x is boolean: classes “true” and “false”
– Some classes may represent invalid input
• Choosing test values
– Choose a typical value in the middle of the class(es) that represent
valid input
– Also choose values at the boundaries of all classes: e.g., if the range is
[a..b], use a-1,a, a+1, b-1,b,b+1

Example
• Suppose our spec says that the code accepts
between 4 and 24 inputs, and each one is a 3-digit
positive integer
• One dimension: partition the number of inputs
– Classes are “x<4”, “4<=x<=24”, “24<x”
– Chosen values: 3,4,5, 14, 23,24,25
• Another dimension: partition the integer values
– Classes are “x<100”, “100<=x<=999”, “999<x”
– Chosen values: 99,100,101, 500, 998,999,1000

Another Example
• Similar approach can be used for the output: exercise
boundary values
• Suppose that the spec says “the output is between 3
and 6 integers, each one in the range 1000-2500
• Try to design input that produces
– 3 outputs with value 1000

Example: Searching
• Search for a value in an array
– Return value is the index of some occurrence
of the value, or -1 if the value does not occur in
the array
• One partition: size of the array
– Since people often make errors for arrays of
size 1, we decide to create a separate
equivalence class

Example: Searching
• Another partition: location of the value
– Four classes: “first element”, “last element”, “middle element”, “not
found”
Array Value Output
Empty 5 -1
[7] 7 0
[7] 2 -1
[1,6,4,7,2] 1 0
[1,6,4,7,2] 4 2
[1,6,4,7,2] 2 4
[1,6,4,7,2] 3 -1

Levels of Testing
• Unit Testing
• Integration Testing
• Validation Testing
– Regression Testing
– Alpha Testing
– Beta Testing
• Acceptance Testing

Unit Testing
• Algorithms and logic
• Data structures (global and local)
• Interfaces
• Independent paths
• Boundary conditions
• Error handling

Why Integration Testing Is Necessary
• One module can have an adverse effect on
another
• Subfunctions, when combined, may not
produce the desired major function
• Individually acceptable imprecision in
calculations may be magnified to
unacceptable levels

Why Integration Testing Is Necessary
(cont’d)
• Interfacing errors not detected in unit testing
may appear
• Timing problems (in real-time systems) are not
detectable by unit testing
• Resource contention problems are not
detectable by unit testing

Top-Down Integration
1. The main control module is used as a driver,
and stubs are substituted for all modules
directly subordinate to the main module.
2. Depending on the integration approach
selected (depth or breadth first),
subordinate stubs are replaced by modules
one at a time.

Top-Down Integration (cont’d)
3. Tests are run as each individual module is
integrated.
4. On the successful completion of a set of
tests, another stub is replaced with a real
module
5. Regression testing is performed to ensure
that errors have not developed as result of
integrating new modules

Problems with Top-Down Integration
• Many times, calculations are performed in the
modules at the bottom of the hierarchy
• Stubs typically do not pass data up to the higher
modules
• Delaying testing until lower-level modules are ready
usually results in integrating many modules at the
same time rather than one at a time
• Developing stubs that can pass data up is almost as
much work as developing the actual module

Bottom-Up Integration
• Integration begins with the lowest-level modules, which are
combined into clusters, or builds, that perform a specific
software subfunction
• Drivers (control programs developed as stubs) are written to
coordinate test case input and output
• The cluster is tested
• Drivers are removed and clusters are combined moving
upward in the program structure

Problems with Bottom-Up Integration
• The whole program does not exist until
the last module is integrated
• Timing and resource contention
problems are not found until late in
the process

Validation Testing
• Determine if the software meets all of the requirements
defined in the SRS
• Having written requirements is essential
• Regression testing is performed to determine if the software
still meets all of its requirements in light of changes and
modifications to the software
• Regression testing involves selectively repeating existing
validation tests, not developing new tests

Alpha and Beta Testing
• It’s best to provide customers with an outline
of the things that you would like them to focus
on and specific test scenarios for them to
execute.
• Provide with customers who are actively
involved with a commitment to fix defects
that they discover.

Acceptance Testing
• Similar to validation testing except that
customers are present or directly
involved.
• Usually the tests are developed by the
customer

Test Methods
• White box or glass box testing
• Black box testing
• Top-down and bottom-up for performing
incremental integration
• ALAC (Act-like-a-customer)

Validation Readiness Review
• During informal validation developers can
make any changes needed in order to comply
with the SRS.
• During informal validation QA runs tests and
makes changes as necessary in order for tests
to comply with the SRS.

Formal Validation
• The same tests that were run during informal
validation are executed again and the results
recorded.
• Software Problem Reports (SPRs) are submitted
for each test that fails.
• SPR tracking is performed and includes the status
of all SPRs ( i.e., open, fixed, verified, deferred,
not a bug)

Test Planning
• The Test Plan – defines the scope of the work
to be performed
• The Test Procedure – a container document
that holds all of the individual tests (test
scripts) that are to be executed
• The Test Report – documents what occurred
when the test scripts were run

Test Plan
• Questions to be answered:
– How many tests are needed?
– How long will it take to develop those tests?
– How long will it take to execute those tests?
• Topics to be addressed:
– Test estimation
– Test development and informal validation
– Validation readiness review and formal validation
– Test completion criteria

Test Estimation
• Number of test cases required is based on:
– Testing all functions and features in the SRS
– Including an appropriate number of ALAC (Act Like A
Customer) tests including:
• Do it wrong
• Use wrong or illegal combination of inputs
• Don’t do enough
• Do nothing
• Do too much
– Achieving some test coverage goal
– Achieving a software reliability goal

Considerations in
Test Estimation
• Test Complexity – It is better to have many small
tests that a few large ones.
• Different Platforms – Does testing need to be
modified for different platforms, operating systems,
etc.
• Automated or Manual Tests – Will automated tests
be developed? Automated tests take more time to
create but do not require human intervention to
run.

Estimating Tests Required
SRS
Reference
Estimated
Number of
Tests
Required
Notes
4.1.1 3 2 positive and 1 negative test
4.1.2 2 2 automated tests
4.1.3 4 4 manual tests
4.1.4 5 1 boundary condition, 2 error
conditions, 2 usability tests
…
Total 165

Test Report
• Completed copy of each test script with evidence
that it was executed (i.e., dated with the signature
of the person who ran the test)
• Copy of each SPR showing resolution
• List of open or unresolved SPRs
• Identification of SPRs found in each baseline along
with total number of SPRs in each baseline
• Regression tests executed for each software
baseline

OBJECT ORIENTED TESTING
SYSTEM TESTING
UNIT TESTING
INTEGRATION
TESTING
INHERITANCE
POLYMORPHISM
ENCAPSULATION

TRADITIONAL DEVELOPMENT & TESTING
(WATERFALL LIFE CYCLE)
• REQUIREMENTS SPEC SYSTEM TESTING
• PRELIMINARY DESIGN INTEGRATION
TESTING
– FUNCTIONAL
– DECOMPOSITION
• DETAILED DESIGN UNIT TESTING

TRADITIONAL TESTING
• SYSTEM
– VERIFY SW SATISFIES ALL SW REQRS
• INTEGRATION
– BASED ON STRUCTURE OF DESIGN
– TOP DOWN OR BOTTOM UP APPROACH
• UNIT
– ENCAPSULATES FUNCTIONALITY

OO DEVELOPMENT & TESTING
• DEVELOPMENT BASED ON BEHAVIOUR
• COMPOSITION
• TYPICALLY RAPID PROTOTYPING
• INCREMENTAL APPROACH
• 3 TRADITIONAL TESTING LEVELS ARE NOT AS
CLEARLY DEFINED

OBJECT ORIENTED TESTING
• SYSTEM
– SAME AS TRADITIONAL
– STILL BASED ON REQRS SPEC
• UNIT
– TWO COMMON STRUCTURES USED
• METHOD*
• CLASS
– SAME AS TRADITIONAL(DRIVERS & STUBS)

METHOD 2
METHOD 1
METHOD METHOD METHOD
METHOD
METHOD
OBJECT CLASS A
B C D
E
F

METHOD 2
METHOD 1
METHOD METHOD METHOD
METHOD
METHOD
OBJECT CLASS A
B C
D
E
F

OO INTEGRATION TESTING
• MAIN PROGRAM IS MINIMIZED
• MOST COMPLICATED PART OF OO TESTING
• TESTING BASED ON COMPOSITION IN BOTTOM UP APPROACH
• USE OF CLUSTERS
• ORD - CLASS DEPENDENCIES
• BBD OR DIRECTED GRAPHS - SHOWS METHOD DEPENDENCIES

meth1
meth3meth2
Class 1
A
B
OUTPUT PORT EVENT
A
meth1
meth3
meth2
meth2
meth1
B
OUTPUT PORT EVENT
Class 2
Class 3
MM-Path
1
2
3

OO CONCEPTS/EFFECTS ON TESTING
• ENCAPSULATION
• POLYMORPHISM
• INHERITANCE

ENCAPSULATION
• CLASS STRUCTURE
• INTERFACE DEFINED BY PUBLIC METHODS
• BEHAVIOR DEFINED BY METHODS THAT
OPERATE ON ITS INSTANCE DATA (IN
CONVENTIONAL SEPARATE)
• HELPS ENFORCE INFO HIDING

ENCAPSULATION TESTING ISSUES
• MINIMIZES RIPPLE EFFECT (AT THE UNIT
LEVEL) OF MAKING A CHANGE
• HIGHLY DELOCALIZED
– CHANGE COULD RESULT IN SIGNIFICANT
REGRESSION TESTING
• ORDER OF TESTING IS IMPORTANT (CAN
REDUCE TESTING EFFORT)

METHOD
METHOD
CLASS A
CLASS B
CLASS C
METHOD
USES USES

POLYMORPHISM
• AN ATTRIBUTE MAY HAVE MORE THAN ONE
SET OF VALUES
• AN OPERATION MAY BE IMPLEMENTED BY
MORE THAN ONE METHOD ( e.g GRAPHICS )
• OVERLOADING (type or number of variables)
• DYNAMIC BINDING

OO TESTING ISSUES
• POLYMORPHISM
– DO YOU TEST ONE VARIANT ?
– DO YOU TEST ALL VARIATIONS ?
– IF ALL, DO YOU TEST ALL VARIANTS AT ALL LEVELS
• UNIT
• “INTEGRATION” OR SYSTEM LEVEL
– REUSE DRIVERS AND STUBS

INHERITANCE STRUCTURES
BASE
SUBCLASS
SINGLE
BASEBASE
SUBCLASS
SUBCLASS
BASE
MULTIPLE MULTIPLE LEVELS

INHERITANCE
PARENT CLASS
MODIFIER
+
RESULT CLASS

INHERITANCE MODIFIERS
• NONE (ONLY INHERITED ATTRIBUTE)
• ADD NEW ATTRIBUTE(S)
• REDEFINE PARENT’S ATTRIBUTE(S)
• VIRTUAL ATTRIBUTE (THREADS IN JAVA)

OO TESTING ISSUES
• INHERITANCE
– DO YOU COMPLETELY TEST ALL BASE CLASSES AND
THEIR SUB-CLASSES ?
– DO YOU COMPLETELY TEST ALL BASE CLASSES AND
ONLY TEST THE CHANGES OR MODIFICATIONS IN
THEIR SUB-CLASSES ?
– AT WHAT LEVELS DO YOU TEST?
– IN WHICH ORDER DO YOU TEST?

INHERITED TESTING
SCENARIO UNIT INTEGRATION
NONE X?
NEW X X?
REDEFINED X X
VIRTUAL (COMPLETED
BY SUBCLASS)
X X?
VIRTUAL ( NOT
COMPLETED)

OO TESTING METHODOLOGY
• JORGENSEN AND ERICKSEN PROPOSE 5 LEVELS
• A METHOD - UNIT TESTING
• MESSAGE QUIESCENCE - INTEGRATION
• EVENT QUIESCENCE - INTEGRATION
• THREAD TESTING -SYSTEM
• THREAD INTERACTION -SYSTEM

Software Testing Definitions and Approaches

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Software Testing Definitions and Approaches

Similar to Software Testing Definitions and Approaches (20)

More from cpjcollege

More from cpjcollege (20)

Recently uploaded

Recently uploaded (20)

Software Testing Definitions and Approaches