Software Testing

1. SJB INSTITUTE OF TECHNOLOGY
By
ASHWINI K
Asst. Prof.
6th ‘A’
Department of Information Science and Engineering
Module 2
“Functional testing”

2. Agenda
2
 BOUNDARY VALUE ANALYSIS
 EQUVALEANCE CLASS TESTING
 DECESION TABLE
 FAULT BASED TESTING
 MUTATION TESTING
Dept of ISE SJBIT

3. Generalizing Boundary Value Analysis
3
 Boundary value analysis can be applied on the 1)input variables
and on the 2) Kinds of ranges (output)
In the case of basing the analysis on the input, we hold all but
one of the variables at the nominal value and let the remaining
variable assume min, min+, nom, max-& max. (5 cases for
each var.)
In this case all variables will be held at their nominal values n
times, where n=number of variables being tested
It is useless to repeat the test n times when we could only do it
once.
Therefore we have 5 cases –nominal = 4 cases for each
variable = 4n test cases for all variables
We then add the test case where all variables are nominal
Thus for a function of n variables, boundary value analysis
yields 4n+1 test cases
Dept of ISE SJBIT

4. Generalizing Boundary Value Analysis
4
 In the case of applying boundary value analysis on the
kinds of ranges (output), this depends on the type of the
variables themselves
 Variables have discrete, bounded values
 Months can be coded as integers 1..12 Or user-defined like in
Pascal {Jan, Feb,..Dec}
 Variables have no explicit bounds that will be called as
artificial bounds
 Triangle problem
Boolean variables
 Decision table-based testing
Dept of ISE SJBIT

5. Limitations of Boundary Value Analysis
5
 Boundary value analysis works well when the
program to be tested is a function of several
independent variables that represent bounded
physical quantities
e.g. NextDate test cases are inadequate (little stress on
February, dependencies among month, day, and year)
When a variable refer to physical quantities, such as
temperature, air speed, load etc can be extremely
important.
 Boundary value analysis does not work well for
Boolean and logical variables (withdrawal, deposit,
inquiry)
Dept of ISE SJBIT

6. Boundary Value Analysis
6
 When the function f is implemented as a program of 2
input variables, the input variables x1 & x2 will have some
boundaries F(x1, x2), a ≤ x1 ≤ b, c ≤ x2 ≤ d
 The intervals [a,b] and [c,d] are referred to as the ranges of x1 & x2
so we have overloaded the term.
a
c
b
d
x1
x2 Input Space (domain)
Dept of ISE SJBIT

7. Boundary Value Analysis
7
F(x1, x2)
a
c
b
d
x1
x2
Test Cases (function
of two variables)
Dept of ISE SJBIT

8. Robustness Testing
8
 Simple extension of boundary value analysis
 In addition to the five boundary value analysis
values of a variable, see what happens when the
extrema are exceeded with a value slightly greater
than the maximum (max+) and a value slightly
less than the minimum (min-)
 If we adopt our function f to apply to Robustness
testing we find the following equation: f=6n+1
 Focuses on the expected outputs
e.g. exceeding load capacity of a public elevator
 Forces attention on exception handling
Dept of ISE SJBIT

9. Robustness Testing
9
a
c
b
d
x1
x2
Robustness Test Cases
(function of two variables)
<x1nom, x2min>
<x1nom, x2min+>
<x1nom, x2nom>
<x1nom, x2max->
<x1nom, x2max>
<x1min, x2nom>
<x1min+, x2nom>
<x1nom, x2nom>
<x1max-, x2nom>
<x1max, x2nom>
<x1max+, x2nom>
<x1min-, x2nom>
<x1nom, x2max+>
<x1nom, x2min->
Dept of ISE SJBIT

10. Equivalence Class Testing
10
 Boundary Value Testing derives test cases with
 Massive redundancy
 Serious gaps
 Equivalence class testing is the partitioning of the
input variables value set into classes(subsets) using
some equivalence relation
 The use of Equivalence classes as the basis for
functional testing has two motivations:
1. To have a senseof “complete” testing
2. Avoid redundancy (not having serious gaps)
Dept of ISE SJBIT

11. Equivalence Class Testing
11
 The idea of equivalence class testing is to identify test
cases by using one element from each equivalence
class
 The fact that the entire set is represented provides a
form of completeness
 The key of equivalence class testing is the choice of
the equivalence relation that determines the classes
 For a function of 2 variables x1and x2, where:
a ≤x1≤d with intervals [a,b),[b,c),[c,d]
e ≤x2≤g with intervals [e,f),[f,g]
Dept of ISE SJBIT

12. Equivalence Class Testing
12
 There are four types of equivalence class testing
1.Weak Normal equivalence class testing
2.Strong Normal equivalence class testing
3.Weak Robust equivalence class testing
4.Strong Robust equivalence class testing
 Normal ECT : only valid entries are considered while
determining the test cases
 We consider invalid values in Robust ECT.
 Weak ECT adopts the single fault assumption
 Strong adopts the multiple fault assumption
Dept of ISE SJBIT

13. Weak Normal Equivalence Class
Testing
13
 It is accomplished by using one variable from each
equivalence class in a test case
 It follows single fault assumption stating that errors
are only rarely the result of simultaneous occurrence
of 2 or more faults
 Assume the equivalence partitioning of input x1 is
[a,b),[b,c),[c,d]
 Assume the equivalence partitioning of input x2 is
[e,f),[f,g]
Dept of ISE SJBIT

14. Example of : Weak Normal Equivalence
testing
14 Dept of ISE SJBIT

15. 15 Dept of ISE SJBIT

16. Strong Normal Equivalence Class Testing
16
 It is based on multiple fault assumption which states that
errors will result in a combination of faults
 It tests every combination of elements formed as aresult of
the cartesian product of the equivalence classes
 Cartesian product guarentees notion of completeness in 2
sense
1. We cover all the equivalence classes
2. We have one of the each possible combination of inputs
 This is same as weak normal ECT except for “multiple fault
assumption”
 Assume the equivalence partitioning of input x1 is
[a,b),[b,c),[c,d]
 Assume the equivalence partitioning of input x2 is [e,f),[f,g]
Dept of ISE SJBIT

17. 17 Dept of ISE SJBIT

18. 18 Dept of ISE SJBIT

19. Weak Robust Equivalence Class Testing
19
 There are 2 parts here one is weak and one more is
Robust
 The robust part comes from consideration of invalid values
 The weak part refers to the single fault assumption
• For valid inputs, use one value from each valid class
 For invalid inputs, a test case will have one invalid value
and remaining values will be all valid
 Weak robust is similar to weak normal ECT except that the
invalid input variables are now considered
 Assume the equivalence partitioning of input x1 is
[a,b),[b,c),[c,d]
 Assume the equivalence partitioning of input x2 is [e,f),[f,g]
Dept of ISE,SJBIT

20. 20 Dept of ISE,SJBIT

21. 21 Dept of ISE,SJBIT

22. Strong Robust Equivalence Class Testing
22
 Doesn’t assume single fault but it will assume multiple fault
 Strong robust is similar to strong normal ECT except that the
invalid input variables are now considered
 There are 2 parts here one is strong and one more is Robust
 The robust part comes from consideration of invalid values
 The strong part refers to the multiple fault assumption
• For valid inputs, use one value from each valid class
 For invalid inputs, a test case will have one invalid value and
remaining values will be all valid
 Assume the equivalence partitioning of input x1 is
[a,b),[b,c),[c,d]
 Assume the equivalence partitioning of input x2 is [e,f),[f,g]
Dept of ISE SJBIT

23. 23 Dept of ISE SJBIT

24. Strong Robust Equivalence Class Testing
TestCase X1 X2
SR1 ab ef
SR2 ab fg
SR3 ab <e
SR4 ab >g
SR5 bc ef
SR6 bc fg
SR7 bc <e
SR8 bc >g
SR9 cd ef
SR10 cd fg
TestCase X1 X2
SR11 cd <e
SR12 cd >g
SR13 <a ef
SR14 <a fg
SR15 <a <e
SR16 <a >g
SR17 >d ef
SR18 >d fg
SR19 >d <e
SR20 >d >g
24 Dept of ISE SJBIT

25. Guidelines & Observations
 If the implementation language is strongly typed then there is
no need for robust form
 ECT is strengthened when combined with boundary value
testing
 ECT is appropriate when input data is defined in terms of
intervals & sets of discrete values
 If error conditions are a high priority the robust forms are
appropriate(commission problem)
 Obviously the weak forms of ECT are not as comprehensive
as the corresponding strong forms
25 Dept of ISE SJBIT

26. Weak Normal ET for Triangle problem
Four weak normal equivalence class testcases chosen
arbitrarily from each class are
TestCase Id Inputs Expected Output
a b c
WN1 5 5 5 Equilateral
WN2 2 2 3 Isosceles
WN3 3 4 5 Scalene
WN4 4 1 2 Not a Triangle
26 Dept of ISE SJBIT

27. Strong Normal ET for Triangle problem
Since there is no sub-intervals inside the valid inputs for the 3
sides a, b & c are Strong Normal Equivalence is same as the
Weak Normal equivalence
TestCase Id Inputs Expected Output
a b c
SN1 5 5 5 Equilateral
SN2 2 2 3 Isosceles
SN3 3 4 5 Scalene
SN4 4 1 2 Not a Triangle
27 Dept of ISE SJBIT

28. Weak Robust ET for Triangle problem
TestCase Id Inputs Expected Output
a b c
WR1 -1 5 5 Value of a not in the range 1…10
WR2 5 -1 5 Value of b not in the range 1…10
WR3 5 5 -1 Value of c not in the range 1…10
WR4 11 5 5 Value of a not in the range 1…10
WR5 5 11 5 Value of b not in the range 1…10
WR6 5 5 11 Value of c not in the range 1…10
28 Dept of ISE SJBIT

29. Strong Robust ET for Triangle problem
 Similar to weak robust but all combinations of invalid inputs must be
included to the strong normal
 Look at the cube figure & consider the corners (2 diagonal ones)
a) Consider one of the corners <10,10,10> : there should be (23 – 1) = 7 cases of “invalids”
< 11, 5 , 5 > < 5 , 11, 5 > < 5, 5 , 11 >
< 11, 11, 5 > < 5 , 11, 11 > < 11, 5 , 11 >
< 11, 11, 11 >
b) There will be 7 more “invalids” when we consider the other corner , <1,1,1 >:
< -1, 5, 5 > < 5, -1, 5 > < 5, 5, -1 >
< -1, -1, 5 > < -1, 5, -1 > < 5, -1, -1 >
< -1, -1, -1 >
29 Dept of ISE SJBIT

30. Decision Table Based Testing Outline
• Decision table vocabulary
– Limited Entry Decision Tables (LEDT)
– Extended Entry Decision Tables (EEDT)
• Techniques
– Redundant LEDTs
– Inconsistent LEDTs
• Examples
• Guidelines
• Cause and Effect Graphs
30 Dept of ISE SJBIT

31. Decision Table Based Testing
 Decision table is based on logical relationships just as the truth table.
 Decision Table is a tool that helps us look at the combination of
conditions
 Completeness of conditions
 Inconsistency of conditions
31 Dept of ISE SJBIT

32. Decision Table Based Testing
 Originally known as Cause and Effect Graphing
 Done with a graphical technique that expressed
 AND-OR-NOT logic.
 Causes and Effects were graphed like circuit components
 Inputs to a circuit “caused” outputs (effects)
 Equivalent to forming a decision table in which:
 inputs are conditions
 outputs are actions
 Test every (possible) rule in the decision table.
 Recommended for logically complex situations.
 Excellent example of Model-Based Testing (MBT)
32 Dept of ISE SJBIT

33. Content of a Decision Table
• Conditions
– binary in a Limited Entry Decision Table
– finite set in an Extended Entry Decision Table
– condition stub
– condition entries
• Actions
– also binary, either do or skip
– the “impossible” action
• Rules
– a rule consists of condition entries and action entries
– a complete, non-redundant LEDT with n conditions has 2n
rules
– logically impossible combinations of conditions are “impossible rules”, denoted by an
entry in the impossible action
33 Dept of ISE SJBIT

34. Decision Table Based -EXAMPLE
34 Dept of ISE SJBIT

35. Fault based testing
35 Dept of ISE SJBIT

36. Assumption in fault Based Testing
 The main idea behind fault based testing is to
select test cases that would distinguish the
program under test from alternative program that
contains hypothetical faults.
36 Dept of ISE SJBIT

37. FBT
 The competent programmer hypothesis
assumes that competent programmers tend to
write nearly "correct" programs .
 That is programs written by experienced
programmers may not be correct, but they will
differ from the corrected version by some
relatively simple faults such as off-by-one fault.
37 Dept of ISE SJBIT

38. FBT
 The coupling effect stated that a set of test data
that can uncover all simple faults in a program is
also capable of detecting more complex faults.
38 Dept of ISE SJBIT

39. FBT
 Competent programmer hypothesis:
 Programs are nearly correct
 Real faults are small variations from the correct program
 => Mutants are reasonable models of real buggy programs
 Coupling effect hypothesis:
 Tests that find simple faults also find more complex
faults
 Even if mutants are not perfect representatives of real
faults, a test suite that kills mutants is good at finding real
faults too
39 Dept of ISE SJBIT

40. Mutation Analysis
Dept of ISE,SJBIT
40
 Mutation Testing is a powerful error-based testing technique for
unit testing. It provides high test coverage and detects many
simple syntactic faults.
 A mutant is a copy of a program with a mutation
 A mutation is a syntactic change (a seeded bug)
 Example: change (i < 0) to (i <= 0)
 Run test suite on all the mutant programs
 A mutant is killed if it fails on at least one test case
 If many mutants are killed, infer that the test suite is also
effective at finding real bugs

41. Mutation Testing
Dept of ISE,SJBIT
41
 The goal of Mutation Testing is to assess the
quality of the test cases which should be robust
enough to fail mutant code.
 To maintain the effectiveness of test sets.
 Mutation was originally proposed in 1971 but lost
fervor due to high costs involved. Now, again it
has picked steam and is widely used for
languages such as Java and XML.

42. Different types of Mutants
Dept of ISE,SJBIT
42
 Stillborn mutants: Syntactically incorrect, killed by
compiler, e.g., x = a ++ b
 Trivial mutants: Killed by almost any test case
 Equivalent mutant: Always acts in the same behavior
as the original program, e.g., x = a + b and x = a – (-b)
 None of the above are interesting from a mutation
testing perspective
 Those mutants are interesting which behave
differently than the original program, and we do not
have test cases to identify them (to cover those
specific changes)

43. Mutation Testing
Dept of ISE,SJBIT
43

44. Automation of Mutation
Testing:
Dept of ISE,SJBIT
44
 Mutation testing is extremely time consuming
and complicated to execute manually. To
speed up the process, it is advisable to go for
automation tools. Automation tools reduce
cost of testing as well.
 List of tools available -
 Ninja Turtles- .net mutation testing tool
 Mutagenesis– PHP mutation testing
framework
 Heckle– Ruby Mutation Testing Tool
 Jester– Mutation Testing Tool for Java

45. Mutation Score:
Dept of ISE,SJBIT
45
 The mutation score is defined as the percentage
of killed mutants with the total number of mutants.
 Mutation Score = (Killed Mutants / Total number
of Mutants) * 100
 The effectiveness of the test data set is measured
by the percentage of mutants killed.

46. Advantages of Mutation Testing
Dept of ISE,SJBIT
46
 It is a powerful approach to attain high coverage
of the source program.
 This testing is capable comprehensively testing
the mutant program.
 Mutation testing brings a good level of error
detection to the software developer.
 This method uncovers ambiguities in the source
code, and has the capacity to detect all the faults
in the program.
 Customers are benefited from this testing by
getting most reliable and stable system.

47. Disadvantages of Mutant testing
Dept of ISE,SJBIT
47
 Mutation testing is extremely costly and time
consuming since there are many mutant programs
that need to be generated.
 Since its time consuming, it’s fair to say that this
testing cannot be done without an automation tool.
 Each mutation will have the same number of test
cases than that of the original program. So, a large
number of mutant programs may need to be tested
against the original test suite.
 As this method involves source code changes, it is
not at all applicable for black box testing.

48. Mutation Operators
Dept of ISE,SJBIT
48
 Syntactic change from legal program to legal
program
 So: Specific to each programming language. C++
mutations don’t work for Java, Java mutations don’t work for
Python
 Examples:
 crp: constant for constant replacement
 for instance: from (x < 5) to (x < 12)
 select from constants found somewhere in program text
 ror: relational operator replacement
 for instance: from (x <= 5) to (x < 5)
 vie: variable initialization elimination
 change int x =5; to int x;

49. Live Mutants
Dept of ISE,SJBIT
49
 Scenario:
 We create 100 mutants from our program
 We run our test suite on all 100 mutants, plus the
original program
 The original program passes all tests
 94 mutant programs are killed (fail at least one test)
 6 mutants remain alive
 What can we learn from the living mutants?

50. Variation in mutation Testing
Dept of ISE,SJBIT
50
 Since the number of mutants that can be
generated is large (the number is usually on the
order of N2, where N is the number of variable
references in the program), methods have been
suggested to reduce the computational expenses
of this testing technique.
 Methods proposed over the years to combat the
expensive computation problem include weak
mutation, Strong mutation, Statistical mutation.

51. Weak mutation
Dept of ISE,SJBIT
51
 Weak mutation only requires the test data to cause a mutated
component to take on a different value in at least one execution,
instead of outputting a different value from the expected result.
 Problem: There are lots of mutants. Running each test case to
completion on every mutant is expensive
 Number of mutants grows with the square of program size
 Approach:
 Execute meta-mutant (with many seeded faults) together with
original program
 Mark a seeded fault as “killed” as soon as a difference in
intermediate state is found
 Without waiting for program completion
 Restart with new mutant selection after each “kill”

52. References/Bibliography
Dept. of ISE, SJBIT
52
[1] https://www.testim.io/blog/software-testing-
basics
[2] https://www.guru99.com/software-testing-
introduction-importance.html
[3] https://www.geeksforgeeks.org/software-testing-
basics/