1. 1
Multi-Objective Test Case Selection
Through Linkage Learning-Driven
Crossover
Mitchell Olsthoorn, Annibale Panichella

2. Context: Regression Testing
De
fi
nition:
Re-running tests to ensure that previously
developed and tested software still functions
the same after a change
Expensive process:
Running the full test suite can take up to days
for large software products
Methods:
• Test Case Minimization
• Test Case Prioritization
• Test Case Selection
2
Yoo and Harmon, STVR, 2012

3. Related Work: What Has Been Done So Far?
3
vs
{
}
test
{
}
test
{
}
test
{
}
test
{
}
test
{
}
test
B01 B02 B03 B04 B05 B06 B07 B08 B09 B10 B11 B12
0 1 1 0 1 0 1 0 0 1 0 1
1 0 1 1 0 0 0 1 0 1 0 0
. . . . . . . . . . . .
1 0 0 1 0 1 0 0 1 0 1 0
Maximize coverage
9477ms
7206ms
…
8768ms
Minimize cost
Multi-Objective Evolutionary Algorithms (MOEAs) in particular NSGA-II
Yoo, S., Harman, M.: Pareto e
ffi
cient multi-objective test case selection.
International Symposium on Software Testing and Analysis. (2007)

4. Problem: Classical Recombination (Crossover)
Randomized crossover performs
poorly on high numbers of problem
variables
Breaks up promising patterns:
4
0 1 1 0 1 0 1 0 0 1 0 1
1 0 1 1 0 1 0 1 0 0 1 1
0 0 1 1 0 1 0 0 0 0 1 1
x
=
x

5. Based on NSGA-II
Uses Linkage learning:
Infer groups of genes that should stay together
Our Solution
5
0 1 1 0 1 0 1 0 0 1 0 1
1 0 1 1 0 1 0 1 0 0 1 1
1 0 1 1 1 0 1 0 0 1 1 1
x

6. L2-NSGA
6
Pop. =
1 0 1 1 0 1 0 0 1 0 0 1
T01 T02 T03 T04 T05 T06 T07 T08 T09 T10 T11 T12
0 0 1 0 0 1 1 1 1 0 1 0
1 1 0 0 1 0 1 0 0 1 1 0
0 1 1 0 1 0 1 0 0 1 0 1
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every

7. L2-NSGA
7
Pop. =
1 0 1 1 0 1 0 0 1 0 0 1
T01 T02 T03 T04 T05 T06 T07 T08 T09 T10 T11 T12
0 0 1 0 0 1 1 1 1 0 1 0
1 1 0 0 1 0 1 0 0 1 1 0
0 1 1 0 1 0 1 0 0 1 0 1
dist. 2
Calculate distances between all test cases
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every

8. L2-NSGA
8
Combine closest clusters
Hierarchical Clustering (UPGMA)
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every
T1 … T12
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12
T1…T4
T2…T4
T3 T4
T5 T6
T5…T7
T8 T9
T10 T11
T10…T12
T5…T12

9. L2-NSGA
9
Parent 1 0 1 1 0 1 0 0 1 0 0 1
T01 T02 T03 T04 T05 T06 T07 T08 T09 T10 T11 T12
0 0 1 0 0 1 1 1 1 0 1 0
0 0 1 0 0 1 0 1 1 0 0 1
Donor
Child
[ {T08, T09}, {T10, T11, T12}, {T01, T02, T03, T04}, …]
FoS
We randomly sample subsets from the FoS
and apply the values from the donor
to the parent
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every

10. L2-NSGA
10
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every
Apply NSGA environmental selection through
non-dominated sorting and crowding distance

11. L2-NSGA
11
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every
Apply NSGA environmental selection through
non-dominated sorting and crowding distance

12. L2-NSGA
12
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every
Apply NSGA environmental selection through
non-dominated sorting and crowding distance

13. L2-NSGA
13
Test Case Selection Through Linkage Learning-based Crossover 5
Algorithm 1: L2-NSGA
1 begin
2 P Ω≠ INITIAL-POPULATION()
3 while not (end condition) do
4 FOS Ω≠ INFER-MODEL(P , 2)
5 P Õ
Ω≠ ÿ
6 forall i in 1..|P | do
7 Parent Ω≠ TOURNAMENT-SELECTION(P )
8 Donor Ω≠ TOURNAMENT-SELECTION(P )
9 Child Ω≠ L2-CROSSOVER(Parent, Donor, FOS)
10 Child Ω≠ MUTATE(Child)
11 P Õ
Ω≠ P Õ
t
{Child}
12 R Ω≠ P Õ
t
P
13 F Ω≠ FAST-NONDOMINATED-SORT(R)
14 P Ω≠ ÿ
15 d Ω≠ 1
16 while | P | + | Fd |Æ M do
17 CROWDING-DISTANCE-ASSIGNMENT(Fd)
18 P Ω≠ P
t
Fd
19 d Ω≠ d + 1
20 SORT-BY-CROWDING-DISTANCE(Fd)
21 P Ω≠ P
t
Fd[1 : (M≠ | P |)]
22 return F1
i.e., random subsets of test suites (line 2). The population then evolves through
subsequent generations to find nearby non-dominated solutions (loop in 3-24).
In line 4, the algorithm infers the linkage structures from the best individuals in
the population P using UPGMA. The structures (FOS) are only inferred every
Apply NSGA environmental selection through
non-dominated sorting and crowding distance

14. Evaluation
14

15. Benchmark: Software-Artifact Infrastructure Repository
(SIR)
15
Program Versions LOC #Tests
Bash v1, v2, v3 44 991 - 46 294 1 061
Flex v1, v2, v3 9 484 - 10 243 567
Grep v1, v2, v3 9 400 - 10 066 806
Sed v1, v2, v3 5 488 - 7 082 360
https://sir.csc.ncsu.edu/
Extensively used in literature
Largest programs in SIR
Contain seeded faults

16. Algorithms: NSGA-II vs L2-NSGA
Termination criterion: 200 generations
Number of repetitions: 20 runs
16
Algorithm Pop. size
Mutation
Operator
Mutation
Parameter
Crossover
Operator
Crossover
Parameters
NSGA-II 100 Bit-
fl
ip Pm = 1/n Scattered Pc = 0.8
L2-NSGA 100 Bit-
fl
ip Pm = 1/n L2
Pc = 0.
8
IntLL = 2

17. RQ1: To what extent does L2-NSGA
produce better Pareto efficient
solutions compared to NSGA-II?
17

18. RQ1: to What Extent Does L2-NSGA Produce Better Pareto
Efficient Solutions Compared to NSGA-II?
Performance Metrics:
• Inverted Generational Distance (IGD)
Measures proximity to reference front and diversity
• Hyper Volume (HV)
Measures dominated area
18
HV
IGD
cost
coverage

19. Statistical results:
22 out of 24 con
fi
gurations with
a p-value <= 0.01
1 out of 24 con
fi
gurations with a
0.01 < p-value <= 0.05
Flex v1: no signi
fi
cant di
ff
erence
E
ff
ect size:
21 Large, 1 Medium, and 2 Small
RQ1: to What Extent Does L2-NSGA Produce Better Pareto
Efficient Solutions Compared to NSGA-II?
19
IGD HV
System Version NSGA-II L2-NSGA NSGA-II L2-NSGA
Bash
v1 0.1987 0.1046 0.4165 0.6418
v2 0.2059 0.1136 0.6223 0.7710
v3 0.2839 0.1221 0.3638 0.6110
Flex
v1 0.0300 0.0265 0.9924 0.9937
v2 0.0324 0.0230 0.9810 0.9853
v3 0.0519 0.0350 0.9808 0.9857
Grep
v1 0.1872 0.0995 0.4623 0.6327
v2 0.1702 0.1301 0.5246 0.5991
v3 0.1920 0.1428 0.4310 0.5540
Sed
v1 0.1123 0.0544 0.8863 0.9580
v2 0.0546 0.0158 0.9508 0.9900
v3 0.0752 0.0253 0.8919 0.9761

20. Statistical results:
22 out of 24 con
fi
gurations with
a p-value <= 0.01
1 out of 24 con
fi
gurations with a
0.01 < p-value <= 0.05
Flex v1: no signi
fi
cant di
ff
erence
E
ff
ect size:
21 Large, 1 Medium, and 2 Small
RQ1: to What Extent Does L2-NSGA Produce Better Pareto
Efficient Solutions Compared to NSGA-II?
20
IGD HV
System Version NSGA-II L2-NSGA NSGA-II L2-NSGA
Bash
v1 0.1987 0.1046 0.4165 0.6418
v2 0.2059 0.1136 0.6223 0.7710
v3 0.2839 0.1221 0.3638 0.6110
Flex
v1 0.0300 0.0265 0.9924 0.9937
v2 0.0324 0.0230 0.9810 0.9853
v3 0.0519 0.0350 0.9808 0.9857
Grep
v1 0.1872 0.0995 0.4623 0.6327
v2 0.1702 0.1301 0.5246 0.5991
v3 0.1920 0.1428 0.4310 0.5540
Sed
v1 0.1123 0.0544 0.8863 0.9580
v2 0.0546 0.0158 0.9508 0.9900
v3 0.0752 0.0253 0.8919 0.9761

21. Statistical results:
22 out of 24 con
fi
gurations with
a p-value <= 0.01
1 out of 24 con
fi
gurations with a
0.01 < p-value <= 0.05
Flex v1: no signi
fi
cant di
ff
erence
E
ff
ect size:
21 Large, 1 Medium, and 2 Small
RQ1: to What Extent Does L2-NSGA Produce Better Pareto
Efficient Solutions Compared to NSGA-II?
21
IGD HV
System Version NSGA-II L2-NSGA NSGA-II L2-NSGA
Bash
v1 0.1987 0.1046 0.4165 0.6418
v2 0.2059 0.1136 0.6223 0.7710
v3 0.2839 0.1221 0.3638 0.6110
Flex
v1 0.0300 0.0265 0.9924 0.9937
v2 0.0324 0.0230 0.9810 0.9853
v3 0.0519 0.0350 0.9808 0.9857
Grep
v1 0.1872 0.0995 0.4623 0.6327
v2 0.1702 0.1301 0.5246 0.5991
v3 0.1920 0.1428 0.4310 0.5540
Sed
v1 0.1123 0.0544 0.8863 0.9580
v2 0.0546 0.0158 0.9508 0.9900
v3 0.0752 0.0253 0.8919 0.9761

22. RQ1: to What Extent Does L2-NSGA Produce Better Pareto
Efficient Solutions Compared to NSGA-II?
22
Sed v1
100 Generations 200 Generations

23. RQ1: to What Extent Does L2-NSGA Produce Better Pareto
Efficient Solutions Compared to NSGA-II?
23
Sed v1
100 Generations 200 Generations
L2-NSGA achieves better results independently of the size of the project
and the test suites

24. RQ2: What is the cost-effectiveness of
the solution produced by L2-NSGA vs.
NSGA-II ?
24

25. RQ2: What Is the Cost-Effectiveness of the Solution Produced by L2-NSGA vs.
NSGA-II?
25
ICE
System Version NSGA-II L2-NSGA
Bash
v1 0.6857 0.8566
v2 0.5711 0.7031
v3 0.6760 0.8559
Flex
v1 0.6718 0.6721
v2 0.5243 0.5244
v3 0.6809 0.6827
Grep
v1 0.3725 0.43031
v2 0.3474 0.4260
v3 0.1370 0.2052
Sed
v1 0.7552 0.7760
v2 0.9275 0.9414
v3 0.9476 0.9894
Statistical results:
10 out of 12 con
fi
gurations
with a p-value <= 0.01
Flex v1 and v2: no signi
fi
cant
di
ff
erence
E
ff
ect size:
10 Large and 2 Negl.

26. RQ2: What Is the Cost-Effectiveness of the Solution Produced by L2-NSGA vs.
NSGA-II?
26
0
10
20
30
40
bash−v1
bash−v2
bash−v3
flex−v1
flex−v2
flex−v3
grep−v1
grep−v2
grep−v3
sed−v1
sed−v2
sed−v3
System
Time
(s)
L2−NSGA NSGA−II

27. Concluding
27

28. Code Available
28
https://github.com/mitchellolsthoorn/SSBSE-Research-2021-regression-linkage-replication

29. Future Work
• Experiment with alternative clustering algorithms
• Apply this technique to other regression testing methods (TCP)
• Bigger dataset
• Dataset with other languages than C
29

30. 30
Multi-Objective Test Case Selection
Through Linkage Learning-Driven
Crossover
Mitchell Olsthoorn, Annibale Panichella