JTExpert at the Fourth Unity Testing Tool Competition

1. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Conclusions
JTExpert at the Fourth Unit Testing Tool
Competition
Abdelilah Sakti, Gilles Pesant, and Yann-Ga¨el Gu´eh´eneuc
Presented by: Mohammed Sayagh
Department of Computer and Software Engineering
´Ecole Polytechnique de Montr´eal, Qu´ebec, Canada
SBST, May 16th, 2016, Austin, TX, USA
Pattern Trace Identification, Detection, and Enhancement in Java
SOftware Cost-effective Change and Evolution Research Lab

2. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Automatic Test-Data Generation
Goal
Developing automated techniques to reveal hidden
errors.
Solutions
Static Approaches: Constraint Based and Model
checking; Complete; suffers from scalability issues
Dynamic Approaches: Random; scalable; widely used;
Incomplete; depend on many parameters.
JTExpert
Random with Static Analyses for automatic test-data
generation;
Generates test-data suites that reach high code
coverage;
Freely available at:
https://sites.google.com/site/saktiabdel/JTExpert.
2 / 15

3. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
3 / 15

4. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
3 / 15

5. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
©
3 / 15

6. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
3 / 15

7. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
The state of the classe under test
Data Member1
Data Member2
...
Data Memberm
3 / 15

8. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
The state of the classe under test
Data Member1
Data Member2
...
Data Memberm
Inaccessible §
3 / 15

9. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Test Target
Method Under Test
Argument1
...
Argumentn
smallPublic ?
© Inaccessible §
The state of the classe under test
Data Member1
Data Member2
...
Data Memberm
Inaccessible §
Public methods to reach inaccessible parts.
3 / 15

10. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Challenge
1. Finding an instance of the class under test;
2. Finding a sequence of method calls;
3. Finding an instance of each required argument.
The number of sequence of method calls may be
unlimited;
The number of different instances of a given class may
be large.
4 / 15

11. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
Automatic Test-Data
Generation
Test-Data Format of OOP
JTExpert
JTExpert Results
Conclusions
Test-Data Format of OOP
Challenge
1. Finding an instance of the class under test;
2. Finding a sequence of method calls;
3. Finding an instance of each required argument.
The number of sequence of method calls may be
unlimited;
The number of different instances of a given class may
be large.
A combinatorial search space
4 / 15

12. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
The key idea behind JTExpert
Exploring relevant sequences
Methods that may modify a data member or can reach
the test target;
A static analysis can determine the relevant methods.
5 / 15

13. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
The key idea behind JTExpert
. Requirements To analyze To extract
(I)
How to instantiate the class
under test and each required
argument ?
Constructors;
Factory methods;
Data Members;
Derived Classes
(SubClasses).
Set of means-of-instantiation.
(II)
A sequence of method calls to
make the class under test in a
relevant state.
Data Member1
Data Member2
.
.
.
Data Memberm
State Modifier1
State Modifier2
.
.
.
State Modifierm
(III) A method that can reach the
test target. Method Under Test Public Methods caller
6 / 15

14. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
7 / 15

15. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn
7 / 15

16. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
7 / 15

17. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y
7 / 15

18. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
7 / 15

19. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z
7 / 15

20. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
7 / 15

21. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2
7 / 15

22. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2 setS2
7 / 15

23. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2 setS2
foo
7 / 15

24. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
The key idea behind
JTExpert
Example: How does
JTExpert generate a test
candidate?
JTExpert Results
Conclusions
JTExpert
Example: How does JTExpert generate a test candidate?
Source Code
1 public class FooClass{
2 private int y;
3 private float z;
4 private float w;
5 private String s2;
6 public void setY(int y1) {y=y1};
7 public void setZ(float z1) {z=z1};
8 public void setS2(String s) {s2=s};
9 public int fooCaller(int a, String str){
10 ...
11 foo(a,str);
12 ...
13 }
14 private int foo(int x, String s1){
15 if(y==z)
16 if(y>0)
17 if(x==10)
18 return s1.length()+s2.length(); ←
//Test Target
19 return 0;
20 }
21 }
Static Analyses
MIn FooClass()
y setY
z setZ
s2 setS2
foo fooCaller
7 / 15

25. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert at the Third Unit Testing Tool
Competition: Results
8 / 15

26. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Overall
Average Coverage
56% line coverage;
49% branch coverage;
36% mutation coverage;
931 points and was ranked third.
Reasons
Uncompilable test-case suites (-2.15%);
Bad Time Management (-2% and -123 points as a
penalty);
Mutation Coverage Measure (Defect4j, -9%).
9 / 15

27. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Overall
Average Coverage
56% line coverage;
49% branch coverage;
36% mutation coverage;
931 points and was ranked third.
Reasons
Uncompilable test-case suites (-2.15%);
Bad Time Management (-2% and -123 points as a
penalty);
Mutation Coverage Measure (Defect4j, -9%).
If we fixed these issues before the competition, then the
score would be 1196;
9 / 15

28. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Analysis
Coverage versus Score
Metrics JTExpert Evosuite T3 Randoop
Line
Coverage 56.34 % 60.79 % 48.74 % 47.92 %
Score 166.43 181.54 128.74 129.85
Branch
Coverage 49.09 % 48.51 % 41.98 % 38.49 %
Score 284.31 309.37 224.81 201.65
Mutant
Coverage 35.51 % 33.50 % 37.56 % 25.99 %
Score 334.08 360.86 319.94 248.92
10 / 15

29. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Analysis
Coverage versus Score
Metrics JTExpert Evosuite T3 Randoop
Line
Coverage 56.34 % 60.79 % 48.74 % 47.92 %
Score 166.43 181.54 128.74 129.85
Branch
Coverage 49.09 % 48.51 % 41.98 % 38.49 %
Score 284.31 309.37 224.81 201.65
Mutant
Coverage 35.51 % 33.50 % 37.56 % 25.99 %
Score 334.08 360.86 319.94 248.92
The score does not reflect the coverage.
10 / 15

30. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Analysis
The number of test cases
Metrics JTExpert Evosuite T3 Randoop
Test Cases
# 76,407 88,531 194,193 14,013,583
Rank 1 2 3 4
JTExpert is the best in terms of the # of test cases;
With the smallest test-cases suite, JTExpert could
reach the best branch coverage and it is the second in
terms of mutant coverage and line coverage.
If we generated more test cases, then it would be
possible to increase our mutation coverage.
11 / 15

31. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Analysis
Penalties
Metrics JTExpert Evosuite T3 Randoop
Uncompilable
# 37 13 0 5728
Rank 3 2 1 4
Broken test
# 3,220 224 8,010 190,493
Rank 2 1 3 4
Time
# (h) 104.14 96.63 63.54 126.98
Rank 3 2 1 4
Penalty -123.15 -25.79 -17.26 -72.81
The list of metrics above significantly penalized our
final score.
12 / 15

32. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Overall
Analysis
Conclusions
JTExpert Results
Analysis
New Metric
Metrics JTExpert Evosuite T3 Randoop
Fail test
# 404 451 482 359
Rank 3 2 1 4
Score 269.33 300.67 321.33 239.33
The new metric, Fail, above decided the final score
because it allows 4 points for each fail detected.
a Bug in Defect4j
Defect4j considers a test case to reveal an actual error if
its execution fails on a buggy version of the CUT
regardless the reason why it failed.
Many empty test-case suites wrongly considered as
revealing actual errors.
A manual analysis of the log files revealed that some
test cases got 4 points for wrong alerts.
13 / 15

33. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Conclusions
Conclusions
Different internal and external factors negatively
affected our score and ranking
Some bugs in JTExpert;
The way the score is computed;
The bug detected in Defect4J.
The SBST Contest 2016 provides a unique opportunity
to compare JTExpert to other tools and to identify its
limitations.
14 / 15

34. JTExpert at the
Fourth Unit
Testing Tool
Competition
Abdelilah Sakti,
Gilles Pesant, and
Yann-Ga¨el
Gu´eh´eneuc
Presented by:
Mohammed
Sayagh
Introduction
JTExpert
JTExpert Results
Conclusions
Thank you, Questions?
Thank you
Questions?
15 / 15