The presentation was given at the 10th International Symposium on Seach-Based Software Engineering (SSBSE 2018) Abstract: Several criteria have been proposed over the years for measuring test suite adequacy. Each criterion can be converted into a specific objective function to optimize with search-based techniques in an attempt to generate test suites achieving the highest possible coverage for that criterion. Recent work has tried to optimize for multiple-criteria at once by constructing a single objective function obtained as a weighted sum of the objective functions of the respective criteria. However, this solution suffers the problem of sum scalarization, i.e., differences along the various dimensions being optimized get lost when such dimensions are projected into a single value. Recent advances in SBST formulated coverage as a many-objective optimization problem rather than applying sum scalarization. Starting from this formulation, in this work, we apply many-objective test generation that handles multiple adequacy criteria simultaneously. To scale the approach to the big number of objectives to be optimized at the same time, we adopt an incremental strategy, where only coverage targets in the control dependency frontier are considered until the frontier is expanded by covering a previously uncovered target.

1. Incremental Control Dependency Frontier
Exploration for Many-Criteria
Test Case Generation
Annibale Panichella, Fitsum M. Kifetew, Paolo Tonella
September 11, 2017
!1

2. Background
Rojas et al. SSBSE 2015
"Optimising for several criteria at
the same time is feasible without
increasing computational costs"
!2

3. Classical Approach (WSA)
1
2 3
4 5
8
Class Under Test
7
9
6
Suite-level Fitness Function
3

4. Classical Approach (WSA)
1
2 3
4 5
8
7
9
6
Suite-level Fitness Function
Branch
Coverage
b2b1
b3 b4
b5
b6 b7
b8
b9
b10
b11 b12
F = f(b1) + f(b2) + … +f (b12)
Class Under Test
4

5. Classical Approach (WSA)
Suite-level Fitness Function
Statement
Coverage
F = f(b1) + f(b2) + … +f (b12)
+
f(s1) + f(s2) + … +f (s9)
1
2 3
5 4
8
7
9
6
Class Under Test
5

6. Classical Approach (WSA)
Suite-level Fitness Function
Weak Mutation
Coverage
F = f(b1) + f(b2) + … +f (b12)
+
f(s1) + f(s2) + … +f (s9)
f(m1) + f(m2) + … +f (m3)
+
1
2 m1
m2 m3
8
7
9
6
Class Under Test
5

7. Incremental Exploration
7

8. Enhanced Control Flow Graph
6 Annibale Panichella1
, Fitsum Meshesha Kifetew2
, and Paolo Tonella3
(a) Example program (b) Single criterion (c) Multiple criteria
Fig. 1. Code (left), CDG (middle), and ECDG (right) of an example program
Algorithm 1: MC-DynaMOSA
Input:
B = {⌧1, . . . , ⌧m} the set of coverage targets of a program.
CDG = hN, E, si: control dependency graph of the program
Result: A test suite T
L1
L2
L5
L3
b1
b4
b2
b3
L6
L7
L8
b5
b6
b0
Control Flow Graph
Branches + Statements
7

9. Enhanced Control Flow Graph
6 Annibale Panichella1
, Fitsum Meshesha Kifetew2
, and Paolo Tonella3
(a) Example program (b) Single criterion (c) Multiple criteria
Fig. 1. Code (left), CDG (middle), and ECDG (right) of an example program
Algorithm 1: MC-DynaMOSA
Input:
B = {⌧1, . . . , ⌧m} the set of coverage targets of a program.
CDG = hN, E, si: control dependency graph of the program
Result: A test suite T
L1
L2
L5
b1
b4
b2
L7
L8
b5
b6
b0
Enhanced CFG
L3, μ1
L6, μ2
Branches + Statements +
+ Weak Mutation
b3
8

10. Enhanced Control Flow Graph
6 Annibale Panichella1
, Fitsum Meshesha Kifetew2
, and Paolo Tonella3
(a) Example program (b) Single criterion (c) Multiple criteria
Fig. 1. Code (left), CDG (middle), and ECDG (right) of an example program
Algorithm 1: MC-DynaMOSA
Input:
B = {⌧1, . . . , ⌧m} the set of coverage targets of a program.
CDG = hN, E, si: control dependency graph of the program
Result: A test suite T
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
Enhanced CFG
L3, μ1
L6, μ2// o2: x<0, o3: x>0,…
L8, o1, o2
Branches + Statements
+ Weak Mutation
+ Output Diversity
b3
9

11. Dynamic Many-Objective Sorting Algorithm for
Multi-Criteria Coverage
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
L3, μ1
L6, μ2
L8, o1, o2
b3
Status Current Objectives Covered Targets
Init. {b0}
10

12. Dynamic Many-Objective Sorting Algorithm for
Multi-Criteria Coverage
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
L3, μ1
L6, μ2
L8, o1, o2
b3
Status Current Objectives Covered Targets
Init. {b0}
b0 covered {b1, b5} {b0, L1}
11

13. Dynamic Many-Objective Sorting Algorithm for
Multi-Criteria Coverage
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
L3, μ1
L6, μ2
L8, o1, o2
b3
11
f(b1)
f(b5)

14. Dynamic Many-Objective Sorting Algorithm for
Multi-Criteria Coverage
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
L3, μ1
L6, μ2
L8, o1, o2
b3
11
f(b1)
f(b5)
T1
T2
T3
T4

15. Dynamic Many-Objective Sorting Algorithm for
Multi-Criteria Coverage
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
L3, μ1
L6, μ2
L8, o1, o2
b3
Status Current Objectives Covered Targets
Init. {b0}
b0 covered {b1, b5} {b0, L1}
b1 covered {b2, b3, b5} {b0, L1, b1, L2}
12

16. Dynamic Many-Objective Sorting Algorithm for
Multi-Criteria Coverage
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
L3, μ1
L6, μ2
L8, o1, o2
b3
Status Current Objectives Covered Targets
Init. {b0}
b0 covered {b1, b5} {b0, L1}
b1 covered {b2, b3, b5} {b0, L1, b1, L2}
b5 covered {b2, b3, µ2, b4, b6}
{b0, L1, b1, L2, b5,
L6}
13

17. Dynamic Many-Objective Sorting Algorithm for
Multi-Criteria Coverage
L1
L2
L5
b1
b4
b2
L7
b5
b6
b0
L3, μ1
L6, μ2
L8, o1, o2
b3
Status Current Objectives Covered Targets
Init. {b0}
b0 covered {b1, b5} {b0, L1}
b1 covered {b2, b3, b5} {b0, L1, b1, L2}
b5 covered {b2, b3, µ2, b4, b6} {b0, L1, b1, L2, b5, L6}
b2 covered {b3, b4, b6}
{b0, L1, b1, L2, b5, L6, b2,
L3, µ1, µ2}
b3 covered {b4, b6}
{b0, L1, b1, L2, b5, L6, b2,
L3, µ1, µ2, b3, L5}
14

18. Empirical Study
18

19. Study Context
Benchmark:
180 non-trivial classes sampled from SF110
Selection Procedure:
• Computing the McCabe’s cyclomatic complexity (MCC)
• Filtering out all trivial classes (having only methods with MCC <5)
• Random sampling from the pruned projects
Search Budget: Three minutes
16

20. RQ1: DynaMOSA vs. WSAFrequency
0
35
70
105
140
Coverage Criterion
Branch Line WeakMutation Output
Equivalent Better (p-value<0.05) Worse (p-value<0.05)
17

21. DynaMOSA is better
than WSA
WSA is better
than DynaMOSA
RQ1: DynaMOSA vs. WSA
%DifferenceinBranchCoverage
-50
-38
-25
-13
0
13
25
38
50
Classes Under Test
18

22. RQ1: DynaMOSA vs. WSA
%DifferenceinBranchCoverage
-10
1
13
24
35
Classes Under Test
Larger difference
CUT = TableMeta
Project = schemaspy
N.Branches = 68
% Cov. Diff = +31.71%
Larger Class
CUT = JVCParserTokenMan.
Project = JVC
N.Branches = 4252
% Cov. Diff = +13.09%
19

23. RQ2: DynaMOSA vs. MOSAFrequency
0
35
70
105
140
Coverage Criterion
Branch Line WeakMutation Output
Equivalent Better (p-value<0.05) Worse (p-value<0.05)
20

24. RQ2: DynaMOSA vs. MOSA
%DifferenceinBranchCoverage
-10
-1
8
16
25
Classes Under Test
Larger difference
CUT = CoordSystemUtilities
Project = schemaspy
N.Branches = 882
% Cov. Diff = +23.68%
Larger Class
CUT = JVCParserTokenMan.
Project = JVC
N.Branches = 4252
% Cov. Diff = +13.79%
21

25. RQ3: DynaMOSA Multi-Criteria vs. Branch OnlyFrequency
0
30
60
90
120
Coverage Criterion
Branch Line WeakMutation Output
Equivalent Better (p-value<0.05) Worse (p-value<0.05)
22

26. RQ4: What Are the Benefits in Terms of
Fault Detection Capability?Frequency
0
25
50
75
100
DynaMOSA vs WSA DynaMOSA vs MOSA Multi-Criteria vs Branch Only
Equivalent Better (p-value<0.05) Worse (p-value<0.05)
23

27. RQ4: What Are the Benefits in Terms of
Fault Detection Capability?
DynaMOSA vs MOSA DynaMOSA vs WSA Multi-Criteria vs Branch Only
DifferenceinMutationScore
60%
40%
20%
0%
-20%
24

28. What EAs to Choose?
Even Larger

29. Incremental Control Dependency Frontier
Exploration for Many-Criteria
Test Case Generation
Annibale Panichella, Fitsum M. Kifetew, Paolo Tonella
September 11, 2017