The document describes MOSA, a new many-objective sorting algorithm for test case prioritization and generation. MOSA reformulates branch coverage as a multi-objective optimization problem by defining an objective function for each branch. It then uses a genetic algorithm and preference criterion to iteratively generate test cases that optimize coverage of multiple branches at once. MOSA was implemented in the Evosuite-MOSA tool and achieved competitive results in the Third Unit Testing Tool Competition, requiring less time than other high-scoring tools to achieve similar coverage levels. However, Evosuite-MOSA also encountered some crashes and limitations when handling certain classes in the competition benchmark.

1. Annibale Panichella Fitsum M. Kifetew Paolo Tonella
Results for EvoSuite-MOSA at
the Third Unit Testing Tool
Competition

2. What is MOSA?
A. Panichella, F. M. Kifetew, P. Tonella,
“Reformulating Branch Coverage as
Multi-Objective Optimization Problem”
8th IEEE International Conference on Software
Testing, Verification and Validation 2015
1
2 3
4 5
6
Control flow
graph

3. b7
Multi-Objective Optimization:
- Targeting all branches at once
- Chromosomes = Test Cases
- Running Many-Objective Genetic Algorithm
1
2 3
4 5
6
b1
b3
b2
b4
b5
b6
A. Panichella, F. M. Kifetew, P. Tonella,
“Reformulating Branch Coverage as
Multi-Objective Optimization Problem”
8th IEEE International Conference on Software
Testing, Verification and Validation 2015
What is MOSA?
Control flow
graph

4. Solving Multiple Branches at Once
Objectives
Objectives Functions =
1 2
2 5
2 4
5 6
1 3
3 6
f1 =
bn
b1
b3
b2
b4
b5
A. Panichella, F. M. Kifetew, P. Tonella,
“Reformulating Branch Coverage as
Multi-Objective Optimization Problem”
8th IEEE International Conference on Software
Testing, Verification and Validation 2015
f2 =
f3 =
f4 =
f5 =
fn =
…

5. Solving Multiple Branches at Once
Objectives
Objectives Functions =
1 2
2 5
2 4
5 6
1 3
3 6
f1 =
bn
b1
b3
b2
b4
b5
A. Panichella, F. M. Kifetew, P. Tonella,
“Reformulating Branch Coverage as
Multi-Objective Optimization Problem”
8th IEEE International Conference on Software
Testing, Verification and Validation 2015
f2 =
f3 =
f4 =
f5 =
fn =
= approach_level(b1) + branch_dist(b1)
= approach_level(b2) + branch_dist(b2)
= approach_level(b3) + branch_dist(b3)
= approach_level(b4) + branch_dist(b4)
= approach_level(b3) + branch_dist(b3)
= approach_level(bn) + branch_dist(bn)
… …

6. A. Panichella, F. M. Kifetew, P. Tonella,
“Reformulating Branch Coverage as
Multi-Objective Optimization Problem”
8th IEEE International Conference on Software
Testing, Verification and Validation 2015
b1
b2
What is MOSA?
Pareto Optimality: all solutions that are
not dominated by any other solutions
form the Pareto optimal set.

7. A. Panichella, F. M. Kifetew, P. Tonella,
“Reformulating Branch Coverage as
Multi-Objective Optimization Problem”
8th IEEE International Conference on Software
Testing, Verification and Validation 2015
b1
b2
What is MOSA?
Pareto Optimality: all solutions that are
not dominated by any other solutions
form the Pareto optimal set.
A
Test Cases Dominated by A
C
B

8. A
B
A. Panichella, F. M. Kifetew, P. Tonella,
“Reformulating Branch Coverage as
Multi-Objective Optimization Problem”
8th IEEE International Conference on Software
Testing, Verification and Validation 2015
b1
b2
What is MOSA?
Pareto Optimality: all solutions that are
not dominated by any other solutions
form the Pareto optimal set.
C

9. Many-objective Solvers
θ-Non-dominated Sorting
Genetic Algorithm-III
≤ 15 obj.
Grid-based Evolutionary
Algorithm
≤ 10 obj.
Hypervolume-based
Evolutionary Algorithm
≤ 25 obj.
ε-Multi-objective
Evolutionary Algorithm
≤ 3 obj.
Strength Pareto Evolutionary
Algorithm
≤ 3 obj.
Non-dominated Sorting
Genetic Algorithm-II
≤ 3 obj.

10. Many-objective Solvers
θ-Non-dominated Sorting
Genetic Algorithm-III
≤ 15 obj.
Grid-based Evolutionary
Algorithm
≤ 10 obj.
Hypervolume-based
Evolutionary Algorithm
≤ 25 obj.
ε-Multi-objective
Evolutionary Algorithm
≤ 3 obj.
Strength Pareto Evolutionary
Algorithm
≤ 3 obj.
Non-dominated Sorting
Genetic Algorithm-II
≤ 3 obj.
Many-Objective Solvers do not
scale beyond 25 Objectives
We cannot use Standard Many-
Objective Solvers

11. Branch Coverage
b2 = |b-c|
Example
b1 = |a-b|

12. Branch Coverage
b2 = |b-c|
Example
min
Point closest to cover b1
b1 = |a-b|

13. Branch Coverage
b2 = |b-c|
Example
Point closest to cover b1
b1 = |a-b|
Point closest to
cover b2
min

14. Branch Coverage
b2 = |b-c|
Example
Point closest to cover b1
b1 = |a-b|
Point closest to
cover b2
Not all non-dominated
Solutions are optimal

15. Branch Coverage
b2 = |b-c|
Example
Point closest to cover b1
b1 = |a-b|
Point closest to
cover b2
Not all non-dominated
Solutions are optimal
These points
are better than
the others

16. Preference Criterion:
MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
Point closest to cover b1
b1 = |a-b|
Point closest to
cover b2
Not all non-dominated
Solutions are optimal
These points
are better than
the others
Given a branch bi, a test case x is
preferred over another test case y if and
only if the values of the objective function
for bi satisfy the following condition:
bi(x) < bi(y)

17. MOSA: a New Many Objective Sorting Algorithm
Given a branch bi, a test case x is
preferred over another test case y if and
only if the values of the objective function
for bi satisfy the following condition:
bi(x) < bi(y)
or
bi(x) == bi(y) and #size(x) < #size(y)
Preference Criterion:
b2 = |b-c|
b1 = |a-b|
These points
are better than
the others

18. MOSA: a New Many Objective Sorting Algorithm
Preference Criterion:
b2 = |b-c|
b1 = |a-b|
First front
Given a branch bi, a test case x is
preferred over another test case y if and
only if the values of the objective function
for bi satisfy the following condition:
bi(x) < bi(y)
or
bi(x) == bi(y) and #size(x) < #size(y)

19. MOSA: a New Many Objective Sorting Algorithm
Preference Criterion:
b2 = |b-c|
b1 = |a-b|
Second front
Given a branch bi, a test case x is
preferred over another test case y if and
only if the values of the objective function
for bi satisfy the following condition:
bi(x) < bi(y)
or
bi(x) == bi(y) and #size(x) < #size(y)

20. MOSA: a New Many Objective Sorting Algorithm
Preference Criterion:
b2 = |b-c|
b1 = |a-b|
Third front
Given a branch bi, a test case x is
preferred over another test case y if and
only if the values of the objective function
for bi satisfy the following condition:
bi(x) < bi(y)
or
bi(x) == bi(y) and #size(x) < #size(y)

21. MOSA: a New Many Objective Sorting Algorithm
Preference Criterion:
b2 = |b-c|
b1 = |a-b|
Fourth front
Given a branch bi, a test case x is
preferred over another test case y if and
only if the values of the objective function
for bi satisfy the following condition:
bi(x) < bi(y)
or
bi(x) == bi(y) and #size(x) < #size(y)

22. MOSA: a New Many Objective Sorting Algorithm
Preference Criterion:
b2 = |b-c|
b1 = |a-b|
Fifth front
Given a branch bi, a test case x is
preferred over another test case y if and
only if the values of the objective function
for bi satisfy the following condition:
bi(x) < bi(y)
or
bi(x) == bi(y) and #size(x) < #size(y)

23. yes
Random Test Cases
Crossover
Mutation
Selection
End?
no
MOSA: a New Many Objective Sorting Algorithm

24. 1. Crossover: Single-point
2. Mutation: add, modify or
delete statements
3. Selection: preference
criterion + crowding
distance
5. Archive: keep track of the
best test cases covering
branches
yes
Random Test Cases
Crossover
Mutation
Selection
End?
no
Update Archive
MOSA: a New Many Objective Sorting Algorithm
Main Criterion = covered goals
Secondary Criterion = TC size

25. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|

26. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Archive
TC1

27. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Crossover: single-point
Archive
TC1

28. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Mutation: add, delete, modify statements
Archive
TC1

29. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Point closest to cover b1
Point closest to
cover b2
Update Archive:
1* Criterion = covered goals
2* Criterion = TC size
Selection: preference criterionArchive
TC1

30. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Second front
Update Archive:
1* Criterion = covered goals
2* Criterion = TC size
Selection: preference criterionArchive
TC1

31. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Third front
Update Archive:
1* Criterion = covered goals
2* Criterion = TC size
Selection: preference criterionArchive
TC1

32. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Archive
TC1

33. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Archive
TC1

34. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
b2 is covered
Archive
TC1

35. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
b2 is covered
Archive
TC1 TC2

36. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Covered goals are not
considered (optimized)
Archive
TC1 TC2

37. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Archive
TC1 TC2

38. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Archive
TC1 TC2

39. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
Archive
TC1 TC2

40. MOSA: a New Many Objective Sorting Algorithm
b2 = |b-c|
b1 = |a-b|
TC3 Final Test
Suite
Archive
TC1 TC2

41. Evosuite-MOSA: the tool
We have implemented MOSA in a
prototype tool by extending EvoSuite
Evosuite-MOSA uses the new Many-
Objective Genetic Algorithms (MOSA)
instead of the traditional single
objective GAs
Many-Objective approach instead of
Whole Suite approach

42. Third Unit Testing Tool Competition

43. Results
Total Score = 189.22
Statement Coverage = 56.41%
Branch Coverage = 46.13%
Mutation Coverage = 38.53%
Mean Number of Test Case = 20
Mean Running Time = 84s
Evosuite-MOSA uses branch
coverage as testing criterion

44. Results
Total Score = 189.22
Statement Coverage = 56.41%
Branch Coverage = 46.13%
Mutation Coverage = 38.53%
Mean Number of Test Case = 20
Mean Running Time = 84s
EvoSuite-MOSA requires
only 25% the time needed
other tools with highest
ranks in the contest
%BranchCoverage
0%
12,5%
25%
37,5%
50%
Running Time (sec)
0 100 200 300 400
Other Tools Evosuite-MOSA

45. Main Issues
Some classes of the
benchmark crashed our tool
due to bugs fixed in later
versions of EvoSuite
We used an older version of
EvoSuite when implementing
MOSA
LIST

46. Main Issues
Our tool was not able to handle
classes extending the Java class
Exception
For example
SearchException.java
TwitterException.java
Our tool crashed in few seconds.
Thus, it achieved 0% coverage
Java Exception✗
LIST

47. Main Issues
ARFFHandler.java take as input an
external file. Such file must respect the
standard format of ARFF files.
Page.java requires a valid instance of
Wikipedia.java and indirectly a valid
instance of the class
DatabaseConfiguration.java
Our tool achieved 0% coverage
Java Exception
Environmental dependencies
✗
LIST
✗

48. Main Issues
For example, for
Predicates.java
our tool crashed in some runs because
of an Out-of-memory exception
Java Exception
Environmental dependencies
Memory Consumption
✗
✗
✗
LIST
Some subjects lead to the generation of
test cases that may cause an excessive
memory consumption.

49. In other runs, for the same
subject, the tool didn’t
crash and generated test
cases providing the
following scores: 70% of
statement coverage, 62% of
branch coverage and 44%
of mutation score.
Main Issues
Our tool crashed in some runs for
CharMatcher.java, thus, terminating its
execution without generating any test case.
For CycleHandler.java and
WikipediaInfo.java classes, our tool
crashed in every run because of some bugs
related to our EvoSuite front-end.
Finally, for the class TwitterImpl.java
our tool had problems in finding all the
required dependences at preparation time.
Java Exception
Environmental dependencies
Memory Consumption
Other Issues
✗
✗
✗
✗
LIST

50. Remarks for the Contest
The analysis of the scores achieved by different tools in the contest
is challenging since multiple criteria are involved

51. Remarks for the Contest
The analysis of the scores achieved by different tools in the contest
is challenging since multiple criteria are involved
Total Score ≈ ωstat ⨉ covstat + ωbr ⨉ covbr+ωm ⨉ covm - ωt ⨉ Time/3600

52. Remarks for the Contest
The analysis of the scores achieved by different tools in the contest
is challenging since multiple criteria are involved
Execution times has marginal
relevance (weight)
Tools that were configured with few minutes of execution time are penalized
with respect to other tools that were configured with more time for the
generation process.
Total Score ≈ 1 ⨉ covstat + 2 ⨉ covbr+ 4 ⨉ covm- 1/3600 ⨉ Time

53. Remarks for the Contest
%BranchCoverage
0%
12,5%
25%
37,5%
50%
Running Time (sec)
0 100 200 300 400
Other Tools Evosuite-MOSA
Tools with the highest branch
coverage have been configured
with longer execution time

54. Remarks for the Contest
%BranchCoverage
0%
12,5%
25%
37,5%
50%
Running Time (sec)
0 100 200 300 400
Other Tools Evosuite-MOSA
Tools with the highest mutation
coverage have been configured
with longer execution time
Tools with the highest branch
coverage have been configured
with longer execution time

55. Remarks for the Contest
What about the size of the generated test cases?

56. Remarks for the Contest
What about the size of the generated test cases?
After our own experiment with some classes in the benchmark, at same level
of coverage, smaller test cases yielded lower mutation score.

57. Remarks for the Contest
Example
FlushLuceneWork.java
Average TC size = 26 statements
Branch coverage = 100%
Statement coverage = 89.74%
Mutation Score = 62%
Evosuite-MOSA maximizes branch
coverage while selecting the
shortest test cases at equal level of
branch coverage
What about the size of the generated test cases?
After our own experiment with some classes in the benchmark, at same level
of coverage, smaller test cases yielded lower mutation score.

58. Remarks for the Contest
Without minimization
Average TC size = 562 statements
Branch coverage = 100%
Statement coverage = 89.74%
Mutation Score = 86% (+26%)
Example
FlushLuceneWork.java
Average TC size = 26 statements
Branch coverage = 100%
Statement coverage = 89.74%
Mutation Score = 62%
What about the size of the generated test cases?
After our own experiment with some classes in the benchmark, at same level
of coverage, smaller test cases yielded lower mutation score.

59. Remarks for the Contest
Without minimization
Average TC size = 562 statements
Branch coverage = 100%
Statement coverage = 89.74%
Mutation Score = 86% (+26%)
Example
FlushLuceneWork.java
Average TC size = 26 statements
Branch coverage = 100%
Statement coverage = 89.74%
Mutation Score = 62%
What about the size of the generated test cases?
After our own experiment with some classes in the benchmark, at same level
of coverage, smaller test cases yielded lower mutation score.
Non-minimized test cases are able to kill more
mutants, but they are much longer
(more difficult to be analyzed manually)

60. Summary

61. Summary

62. Summary

63. Summary

64. Summary

65. Summary
Thank You!