JTExpert at the Fifth Unit Testing Tool Competition

1. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
JTExpert Results
Conclusions
JTExpert at the Fifth Unit Testing Tool
Competition
Abdelilah Sakti, Gilles Pesant, and Yann-Ga¨el Gu´eh´eneuc
Presented by: Rub´en Saborido Infantes
Department of Computer and Software Engineering
´Ecole Polytechnique de Montr´eal, Qu´ebec, Canada
SBST, May 22th, 2017, Buenos Aires, Argentina
Pattern Trace Identification, Detection, and Enhancement in Java
SOftware Cost-effective Change and Evolution Research Lab

2. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Automatic Test-Data Generation
Goal
Developing automated techniques to reveal hidden
errors.
Solutions
Static Approaches: Constraint Based and Model
checking; Complete; suffers from scalability issues
Dynamic Approaches: Random; scalable; widely used;
Incomplete; depend on many parameters.
JTExpert
Random with Static Analyses for automatic test-data
generation;
Generates test-data suites that reach high code
coverage;
Freely available at:
https://sites.google.com/site/saktiabdel/JTExpert.
2 / 13

3. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
3 / 13

4. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
3 / 13

5. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
©
3 / 13

6. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
3 / 13

7. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
The state of the classe under test
Data Member1
Data Member2
...
Data Memberm
3 / 13

8. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
The state of the classe under test
Data Member1
Data Member2
...
Data Memberm
Inaccessible §
3 / 13

9. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
The state of the classe under test
Data Member1
Data Member2
...
Data Memberm
Inaccessible §
Public methods to reach inaccessible parts.
3 / 13

10. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Challenge
1. Finding an instance of the class under test;
2. Finding a sequence of method calls;
3. Finding an instance of each required argument.
The number of sequence of method calls may be
unlimited;
The number of different instances of a given class may
be large.
4 / 13

11. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Challenge
1. Finding an instance of the class under test;
2. Finding a sequence of method calls;
3. Finding an instance of each required argument.
The number of sequence of method calls may be
unlimited;
The number of different instances of a given class may
be large.
A combinatorial search space
4 / 13

12. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
5 / 13

13. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn
5 / 13

14. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
5 / 13

15. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y
5 / 13

16. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
5 / 13

17. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z
5 / 13

18. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
5 / 13

19. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2
5 / 13

20. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2 setS2
5 / 13

21. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2 setS2
foo
5 / 13

22. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2 setS2
foo fooCaller
5 / 13

23. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert at the Fifth Unit Testing Tool
Competition: Results
6 / 13

24. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Overall
Average Coverage
34% line coverage;
28% branch coverage;
29% mutation coverage;
849 points and was ranked second.
Reasons
Generally, the results are in line with our expectations
except for classes where JTeXpert gets score 0:
0 point for 28 classes out of 69;
0 point in 40% of the competition score.
7 / 13

25. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Analysis
Problem: Compilation Error
15 uncompilable test-case files;
-2 points as penalty for each uncompilable file.
Root: Assertions generation
Constant string with a length greater that 4000;
8 / 13

26. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Analysis
Problem: Missed test-cases file
JTeXpert did not generate any test-cases file for 415
out of 1450 runs, which represents 27% of the
competition runs.
Root: Undetermined
We randomly selected 10 of the CUTs affected and ran
JTeXpert on them on the competition platform;
JTeXpert performed well on all the selected CUTs and
we have not observed in any run that JTeXpert failed to
generate test suite;
There are two other possible components may be
behind this problem:
1. The communication protocol;
2. The contest platform itself;
9 / 13

27. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
JTExpert Results
Conclusions
Conclusions
JTeXpert performed well compared to its results in the
two previous SBST Contests 2016 and 2015;
The SBST Contest 2017 showed us new bugs in
JTeXpert that should be tackled before the next SBST
Contest.
The SBST Contest 2017 provides a unique opportunity
to compare JTExpert to other tools and to identify its
limitations.
10 / 13

28. JTExpert at the
Fifth Unit Testing
Tool Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Rub´en Saborido
Infantes
Introduction
JTExpert
JTExpert Results
Conclusions
Thank you, Questions?
Thank you
Questions?
11 / 13