Boosting Search Based Testing using Constraint Based Testing

1. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Boosting Search Based Testing by using
Constraint Based Testing
Abdelilah Sakti Yann-Ga¨el Gu´eh´eneuc Gilles Pesant
Department of Computer and Software Engineering
´Ecole Polytechnique de Montr´eal, Qu´ebec, Canada
September 28, 2012
SSBSE, Riva del Garda
Pattern Trace Identification, Detection, and Enhancement in Java
SOftware Cost-effective Change and Evolution Research Lab

2. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
What is the cost of a
programming error ?
Software testing approaches
for finding errors
CBT versus SBT
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
What is the cost of a programming error ?
Every year inadequate infrastructure software costs the U.S.
economy around 60 billion $.
1996 : Flight 501 caused the destruction of the
Ariane 5 rocket just 40 seconds after launching.
Between 1985 and 1987 : the radiotherapy
machine Therac-25 sent to patients a X-ray dose
100 times greater than expected.
The test is a very important phase in the life cycle of
software. It may cost more than 50 % of the budget of a
critical software.
2 / 19

3. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
What is the cost of a
programming error ?
Software testing approaches
for finding errors
CBT versus SBT
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Software testing approaches for finding errors
Search based testing (SBT)
Was introduced in 1976 by Miller and Spooner ;
Dynamic analysis of the program ;
Consists of translating the test problem into an
optimization problem.
Constraint based testing (CBT)
Was introduced in 1976 by J.C. KING ;
Static analysis of the program ;
Consist of translating a test problem into a constraint
satisfaction problem (CSP).
3 / 19

4. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
What is the cost of a
programming error ?
Software testing approaches
for finding errors
CBT versus SBT
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
CBT versus SBT
Search based testing (SBT)
+ Handles any sort of programs ;
- Incomplete
- Depends on many parameters.
Constraint based testing (CBT)
+ Precision in test data generation ;
+ Ability to prove that some paths are unreachable ;
- Not Scalable ;
- The incompleteness of decision procedures.
4 / 19

5. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Example : foo function
1 int foo(int X, int Y){
2 if(X<=0 || Y<=0)
3 return 0;
4 int Z;
5 if ((X < Y/2)|| (Y==0))
6 Z= 1 ; //Target 1
7 else if (Y>3∗X)
8 Z=2 ; //Target 2 : (2 ∗ X >= Y ) ∧ (Y > 3 ∗ X) is unsatisfied
9 else{
10 Z = fun(X,Y); //native function that returns X2/Y
11 if ((Z >8) && (Y==10))
12 if(Z==Y)
13 Z=3 ;//Target 3 is a nested branch predicates (10,10)
14 }
5 / 19

6. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Example : foo function
1 int foo(int X, int Y){
2 if(X<=0 || Y<=0)
3 return 0;
4 int Z;
5 if ((X < Y/2)|| (Y==0))
6 Z= 1 ; //Target 1
7 else if (Y>3∗X)
8 Z=2 ; //Target 2 : (2 ∗ X >= Y ) ∧ (Y > 3 ∗ X) is unsatisfied
9 else{
10 Z = fun(X,Y); //native function that returns X2/Y
11 if ((Z >8) && (Y==10))
12 if(Z==Y)
13 Z=3 ;//Target 3 is a nested branch predicates (10,10)
14 }
5 / 19

7. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Example : foo function
1 int foo(int X, int Y){
2 if(X<=0 || Y<=0)
3 return 0;
4 int Z;
5 if ((X < Y/2)|| (Y==0))
6 Z= 1 ; //Target 1
7 else if (Y>3∗X)
8 Z=2 ; //Target 2 : (2 ∗ X >= Y ) ∧ (Y > 3 ∗ X) is unsatisfied
9 else{
10 Z = fun(X,Y); //native function that returns X2/Y
11 if ((Z >8) && (Y==10))
12 if(Z==Y)
13 Z=3 ;//Target 3 is a nested branch predicates (10,10)
14 }
5 / 19

8. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Example : foo function
1 int foo(int X, int Y){
2 if(X<=0 || Y<=0)
3 return 0;
4 int Z;
5 if ((X < Y/2)|| (Y==0))
6 Z= 1 ; //Target 1
7 else if (Y>3∗X)
8 Z=2 ; //Target 2 : (2 ∗ X >= Y ) ∧ (Y > 3 ∗ X) is unsatisfied
9 else{
10 Z = fun(X,Y); //native function that returns X2/Y
11 if ((Z >8) && (Y==10))
12 if(Z==Y)
13 Z=3 ;//Target 3 is a nested branch predicates (10,10)
14 }
5 / 19

9. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Example : foo function
1 int foo(int X, int Y){
2 if(X<=0 || Y<=0)
3 return 0;
4 int Z;
5 if ((X < Y/2)|| (Y==0))
6 Z= 1 ; //Target 1
7 else if (Y>3∗X)
8 Z=2 ; //Target 2 : (2 ∗ X >= Y ) ∧ (Y > 3 ∗ X) is unsatisfied
9 else{
10 Z = fun(X,Y); //native function that returns X2/Y
11 if ((Z >8) && (Y==10))
12 if(Z==Y)
13 Z=3 ;//Target 3 is a nested branch predicates (10,10)
14 }
Challenge
Combine CBT and SBT to overcome the limitations
associated with each of them.
5 / 19

10. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Constraint Based Testing
Example 1 : Generated constraints for Target 3
Original version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
6 / 19

11. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Constraint Based Testing
Example 1 : Generated constraints for Target 3
Original version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Example 1 : Constraints solutions in the domain
[−2 × 104
+ 1, 2 × 104
]
Original version :
1 Unsolvable
2
3
4
6 / 19

12. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Constraint Based Testing
Example 1 : Generated constraints for Target 3
Original version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Example 1 : Constraints solutions in the domain
[−2 × 104
+ 1, 2 × 104
]
Original version :
1 Unsolvable
2
3
4
6 / 19

13. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Constraint Based Testing
Example 1 : Generated constraints for Target 3
Original version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Relaxed version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Example 1 : Constraints solutions in the domain
[−2 × 104
+ 1, 2 × 104
]
Original version :
1 Unsolvable
2
3
4
6 / 19

14. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Constraint Based Testing
Example 1 : Generated constraints for Target 3
Original version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Relaxed version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Example 1 : Constraints solutions in the domain
[−2 × 104
+ 1, 2 × 104
]
Original version :
1 Unsolvable
2
3
4
Relaxed version :
1 x0 ∈ [5, 2 × 104]
2 y0 ∈ [10, 10]
3
4
6 / 19

15. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Motivating Example
Constraint Based Testing
Example 1 : Generated constraints for Target 3
Original version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Relaxed version :
1 not((x0 ≤ 0)||(y0 ≤ 0))∧
2 not((x0 < y0/2)||(y0 = 0))∧
3 not(y0 > 3 × x0)∧
4 z3 = fun(x0, y0)∧
5 ((z3 > 8) ∧ (y0 = 10))∧
6 (z3 = y0)
Example 1 : Constraints solutions in the domain
[−2 × 104
+ 1, 2 × 104
]
Original version :
1 Unsolvable
2
3
4 Search space size is (16 × 108)
Relaxed version :
1 x0 ∈ [5, 2 × 104]
2 y0 ∈ [10, 10]
3
4 Search space size is (2 × 104 − 5)
6 / 19

16. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Search based testing
7 / 19

17. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Search based testing
7 / 19

18. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Search based testing
7 / 19

19. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Search based testing
Random aspect used to generate the initial population and
also, less often, during the evolution process ;
7 / 19

20. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
Constraint Based Testing
Search based testing
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Search based testing
Random aspect used to generate the initial population and
also, less often, during the evolution process ;
Start SBT with some test candidates drawn from CBT is
better than a random initial population.
7 / 19

21. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Overview
Program input data
8 / 19

22. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Overview
Program input data
T1
T2
T3
8 / 19

23. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Overview
Program input data
T1
T2
T3
R1
R2
Unit Under Test
Relaxation.
8 / 19

24. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Overview
Program input data
T1
T2
T3
R1
R2
×
×
×
×
×
Unit Under Test
Relaxation.
8 / 19

25. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Overview
Program input data
T1
T2
T3
R1
R2
×
×
×
×
×
∗
∗ ∗∗
∗
Unit Under Test
Relaxation.
Constrained Population
Generator (CPG).
8 / 19

26. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Overview
Program input data
T1
T2
T3
R1
R2
×
×
×
×
×
∗ ∗∗
∗
∗
Unit Under Test
Relaxation.
Constrained Population
Generator (CPG).
8 / 19

27. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Overview
Program input data
T1
T2
T3
R1
R2
×
×
×
×
×
∗ ∗∗
∗
∗
∗
Unit Under Test
Relaxation.
Constrained Population
Generator (CPG).
Constrained Evolution
Operator (CEO).
8 / 19

28. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Unit Under Test Relaxation
1. For each data type that needs a different solver a new
relaxed version is created.
2. Any statement over unsupported data type is ignored ;
3. Any unsupported expression (function call, operator) is
relaxed ;
4. Any variable assigned inside a loop is relaxed just after
this loop ;
9 / 19

29. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Example : Unit Under Test Relaxation
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
10 / 19

30. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Example : Unit Under Test Relaxation
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
10 / 19

31. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Example : Unit Under Test Relaxation
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
10 / 19

32. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Example : Unit Under Test Relaxation
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
10 / 19

33. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Example : Unit Under Test Relaxation
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
1 intStr(int x,int y,
) {
2 int y= x«y;
3 int x=y+x/y;
4 ;
5 if((
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
10 / 19

34. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Example : Unit Under Test Relaxation
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
1 intStr(int x,int y,
) {
2 int y= R1 ;
3 int x=y+x/y;
4 ;
5 if((
6 && (x>0)
7 && R2 >x)
8 return 1; // Target
9 return 0;
10 }
10 / 19

35. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Example : Unit Under Test Relaxation
1 intStr(int x,int y,String
S1,String S2) {
2 int y= x«y;
3 int x=y+x/y;
4 String s=S1+S2;
5 if((s.equals(”OK”)
6 && (x>0)
7 && s.length()>x)
8 return 1; // Target
9 return 0;
10 }
1 intStr(int x,int y,
) {
2 int y= R1 ;
3 int x=y+x/y;
4 ;
5 if((
6 && (x>0)
7 && R2 >x)
8 return 1; // Target
9 return 0;
10 }
1 intStr(
2 String S1,String S2) {
3
4
5 String s=S1+S2;
6 if(s.equals(”OK”))
7
8
9 return 1; // Target
10 return 0;
11 }
10 / 19

36. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Constrained Population Generator(CPG)
RV1 (integer)
s11
s12
s13
RV2 (float)
s21
s22
RV2 (string)
s31
s32
11 / 19

37. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Constrained Population Generator(CPG)
RV1 (integer)
s11
s12
s13
RV2 (float)
s21
s22
RV2 (string)
s31
s32
Initial Population
s11, s21, s31
11 / 19

38. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Constrained Population Generator(CPG)
RV1 (integer)
s11
s12
s13
RV2 (float)
s21
s22
RV2 (string)
s31
s32
Initial Population
s11, s21, s31 s11, s21, s32 s11, s22, s31
s11, s22, s32
s11, s21, s31 s11, s21, s32 s11, s22, s31
s11, s22, s32
s11, s21, s31 s11, s21, s32 s11, s22, s31
s11, s22, s32
11 / 19

39. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Overview
Unit Under Test Relaxation
Example : Unit Under Test
Relaxation
Constrained Population
Generator(CPG)
Constrained Evolution
Operator (CEO)
Implementation for
integer data type
Evaluation
Related work
Conclusions
CSBT : Constrained Search Based Testing
Constrained Evolution Operator (CEO)
int float
(val1, val2, ? , val4) smalltype ?
smallinteger Solver
smallfloat Solver
smallstring Solver
smallsolvable ?
(val1, val2, nval3 , val4)
(val1, val2,nval3, nval4)
12 / 19

40. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Implementation for integer data type
For SBT, CSBT uses eToc [P. Tonella, ISSTA 2004] ;
For CBT, CSBT uses Cp-sst [A. SAKTI, JFPC 2011] ;
13 / 19

41. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Subjects
Configurations
Results
Related work
Conclusions
Evaluation
Subjects
Publicly available classes from Java standard library and
Apache Commons project
Subject # LOC # Branches
Integer 1244 38
BitSet 755 145
ArithmticUtils 959 102
14 / 19

42. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Subjects
Configurations
Results
Related work
Conclusions
Evaluation
Configurations
analysed approaches
SBT alone : eToc ;
CBT alone : CP-SST ;
CSBT : SBT+CPG ; SBT+CEO ; SBT+CPG+CEO.
Parameters
All default eToc parameters are kept as is.
Integer domain : [−2 × 104, 2 × 104] ;
Solver time out : 500ms ;
Every approach has been executed 10 times on every
class for a runtime limited at 300 s.
15 / 19

43. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Subjects
Configurations
Results
Related work
Conclusions
Evaluation
Results
0 50 100 150 200 250 300
40
50
60
70
80
90
Comparing all techniques on all classes
Time (s)
BranchCoverage%
SBT
CBT
CPG
CEO
CPG+CEO
16 / 19

44. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Subjects
Configurations
Results
Related work
Conclusions
Evaluation
Results
0 50 100 150 200 250 300
40
50
60
70
80
90
Comparing all techniques on all classes
Time (s)
BranchCoverage%
SBT
CBT
CPG
CEO
CPG+CEO
16 / 19

45. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Subjects
Configurations
Results
Related work
Conclusions
Evaluation
Results
0 50 100 150 200 250 300
40
50
60
70
80
90
Comparing all techniques on all classes
Time (s)
BranchCoverage%
SBT
CBT
CPG
CEO
CPG+CEO
16 / 19

46. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Subjects
Configurations
Results
Related work
Conclusions
Evaluation
Results
0 50 100 150 200 250 300
40
50
60
70
80
90
Comparing all techniques on all classes
Time (s)
BranchCoverage%
SBT
CBT
CPG
CEO
CPG+CEO
16 / 19

47. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Subjects
Configurations
Results
Related work
Conclusions
Evaluation
Results
0 50 100 150 200 250 300
40
50
60
70
80
90
Comparing all techniques on all classes
Time (s)
BranchCoverage%
SBT
CBT
CPG
CEO
CPG+CEO
16 / 19

48. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Related work
Using SBT to overcame CBT limits
FLoPSy [K. Lakhotia et al, ICTSS 2010]
Using CBT to overcame SBT limits
Combining search based and constraint based testing.
[Malburg and Fraser, ISSTA 2011] ;
Symbolic search-based testing [M. Harman et al., ASE
2011].
Others
EVACON [T. Xie et al, ASE 2007].
17 / 19

49. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Conclusions
CSBT
CPG and CEO are two new techniques for combining
CBT and SBT.
The obtained results are promising but more
experiments must be performed.
Future work
Enhancing our technique CEO.
Extending results by using several solvers at the same
time.
Extending our approach by exploring new combination
techniques.
18 / 19

50. Boosting Search
Based Testing by
using Constraint
Based Testing
Abdelilah Sakti,
Yann-Ga¨el
Gu´eh´eneuc, Gilles
Pesant
Introduction
Motivating
Example
CSBT :
Constrained Search
Based Testing
Implementation for
integer data type
Evaluation
Related work
Conclusions
Thank you
Questions ?
19 / 19