Test case prioritization (TCP) is aimed at finding an ideal ordering for executing the available test cases to reveal faults earlier. To solve this problem greedy algorithms and meta-heuristics have been widely investigated, but in most cases there is no statistically significant difference between them in terms of effectiveness. The fitness function used to guide meta-heuristics condenses the cumulative coverage scores achieved by a test case ordering using the Area Under Curve (AUC) metric. In this paper we notice that the AUC metric represents a simplified version of the hypervolume metric used in many objective optimization and we propose HGA, a Hypervolume-based Genetic Algorithm, to solve the TCP problem when using multiple test criteria. The results shows that HGA is more cost-effective than the additional greedy algorithm on large systems and on average requires 36% of the execution time required by the additional greedy algorithm.

1. Hypervolume-based Search
forTest Case Prioritization
Dario Di Nucci Annibale Panichella Andy Zaidman Andrea De Lucia

2. fault
test case
Test Case Prioritization
Find an ideal sorting for executing test cases in order to reveal faults earlier

3. fault
test case
Test Case Prioritization
No a priori information
about faults
Find an ideal sorting for executing test cases in order to reveal faults earlier

4. fault
test case
Test Case Prioritization
No a priori information
about faults
Use of surrogates
Find an ideal sorting for executing test cases in order to reveal faults earlier

5. branch
test case
Test Case Prioritization
No a priori information
about faults
Use of surrogates
Find an ideal sorting for executing test cases in order to reveal faults earlier

6. statement
test case
Test Case Prioritization
No a priori information
about faults
Use of surrogates
Find an ideal sorting for executing test cases in order to reveal faults earlier

7. statement
test case
Test Case Prioritization
No a priori information
about faults
Use of surrogates
Find an ideal sorting for executing test cases in order to reveal faults earlier
NP-Complete problem

8. statement
test case
Test Case Prioritization
No a priori information
about faults
Use of surrogates
Find an ideal sorting for executing test cases in order to reveal faults earlier
NP-Complete problem
Approximation
algorithms

9. GreedyAlgorithms

10. GreedyAlgorithms

11. GreedyAlgorithms

12. GreedyAlgorithms
Any better solution?

13. Single Objective Metaheuristics

14. Single Objective Metaheuristics
Integer permutation for
representing a test suite
3 5 1 4 7 2 8 6 9
8 5 1 9 6 2 3 7 4

15. Single Objective Metaheuristics
Integer permutation for
representing a test suite
Maximize AUC
APBC – APDC - APSC
3 5 1 4 7 2 8 6 9
8 5 1 9 6 2 3 7 4

16. Single Objective Metaheuristics
Integer permutation for
representing a test suite
Maximize AUC
APBC – APDC - APSC
3 5 1 4 7 2 8 6 9
8 5 1 9 6 2 3 7 4
TCP problem has
a multi-objective nature

17. Multi Objective Metaheuristics
Pareto-ranking
Algorithms (NSGA-II)

18. Multi Objective Metaheuristics
Pareto-ranking
Algorithms (NSGA-II)

19. Multi Objective Metaheuristics
Pareto-ranking
Algorithms (NSGA-II)

20. Multi Objective Metaheuristics

21. Multi Objective Metaheuristics
Bad effectiveness as the
problem dimensionality
increases

22. Multi Objective Metaheuristics
Bad effectiveness as the
problem dimensionality
increases
For more than 3-objectives, all
individuals are non-dominated
 Poor Selective Pressure

23. Multi Objective Metaheuristics
Bad effectiveness as the
problem dimensionality
increases
No strong empirical evidence
of the cost-effectiveness with
respect to simpler heuristics
For more than 3-objectives, all
individuals are non-dominated
 Poor Selective Pressure

24. Hypervolume-based
Genetic Algorithm

25. Hypervolume
In many-objective optimization there is a growing trend to solve many-
objective problems using quality scalar indicators to condense multiple
objectives into a single objective.
Auger et al. - Theory of the hypervolume
indicator: optimal distributions and the
choice of the reference point - FOGA 2009

26. Hypervolume
In many-objective optimization there is a growing trend to solve many-
objective problems using quality scalar indicators to condense multiple
objectives into a single objective.
Auger et al. - Theory of the hypervolume
indicator: optimal distributions and the
choice of the reference point - FOGA 2009
The hypervolume measures the quality of a set of solutions as the total
size of the objective space that is dominated by one (or more) of such
solutions.

27. Hypervolume
F1
F2
min
min
A
B
A non-dominates B
C
B dominates C
B non-dominates A

28. Hypervolume
F1
F2
min
min
B
C
Area dominated by AA

29. Hypervolume
F1
F2
min
min
B
C
Area dominated by BA

30. Hypervolume
F1
F2
min
min
B
C
Area dominated by CA

31. Hypervolume
F1
F2
min
min
A
B
C
Area(B) > Area(A)
Area(B) > Area(C)
Stronger Selective
Pressure

32. Hypervolume Indicator forTCP
c2-objectives
min
max

33. Hypervolume Indicator forTCP
min
max
c3-objectives

34. Hypervolume Indicator forTCP
It can be used as fitness function in a Single Objective
Metaheuristic

35. Hypervolume Indicator forTCP
Computational Time 𝑶 𝒏 ∙ 𝒎
n test cases
m testing criteria
It can be used as fitness function in a Single Objective
Metaheuristic

36. Empirical Evaluation

37. Research Questions
RQ1 : What is the cost-effectiveness of HGA, compared to
cost-aware additional greedy algorithms?
Cost-cognizant Average Fault Detection Percentage (AFDPc)

38. Research Questions
RQ1 : What is the cost-effectiveness of HGA, compared to
cost-aware additional greedy algorithms?
Cost-cognizant Average Fault Detection Percentage (AFDPc)
RQ2 : What is the efficiency of HGA, compared to
cost-aware additional greedy algorithms?
Execution time (in seconds)

39. Design
Comparison with Additional Greedy algorithms
(2Obj and 3Obj)
6 programs from SIR 20 indipendent GA runs

40. Results RQ1
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
bash flex grep printtokens printtokens2 sed
Additional Greedy 2Obj HGA 2Obj
AFDPc
Average AFDPc
2-Objective formulation

41. Results RQ1
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
bash flex grep printtokens printtokens2 sed
Additional Greedy 3Obj HGA 3Obj
AFDPc
Average AFDPc
3-Objective formulation

42. Results RQ1
Program 2-Objective 3-Objective
p-value Â12 magnitude p-value Â12 magnitude
bash < 0.01 0.88 Large < 0.01 0.95 Large
flex < 0.01 0.70 Medium < 0.01 0.75 Large
grep < 0.01 0.85 Large < 0.01 0.85 Large
printtokens 1 0.10 Large 1 0.10 Large
printtokens 2 1 0.30 Large 0.73 0.40 Small
sed < 0.01 0.85 Large 0.01 0.80 Large

43. Average Execution Time
2-Objective formulation
Results RQ2
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
bash flex grep printtokens printtokens2 sed
Additional Greedy 2Obj HGA 2Obj
seconds

44. Average Execution Time
3-Objective formulation
Results RQ2
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
bash flex grep printtokens printtokens2 sed
Additional Greedy 3Obj HGA 3Obj
seconds

45. Summary

46. Summary

47. Summary

48. Summary

49. Summary

50. Summary

51. 3+
FutureWork
Investigate the scalability
for up than 3 testing criteria

52. FutureWork
Investigate the scalability
for up than 3 testing criteria
Incorporate diversity as a testing criteria

53. FutureWork
Incorporate diversity as a testing criteria
Apply HGA for other test case
optimization problems
Investigate the scalability
for up than 3 testing criteria

54. Thank You
for
Your Attention!
Questions?
Dario Di Nucci
University of Salerno
ddinucci@unisa.it
http://www.sesa.unisa.it/people/ddinucci/