ICSE'23 - Journal-First Track: "Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments" Testing with simulation environments helps to identify critical failing scenarios for self-driving cars (SDCs). Simulation-based tests are safer than in-field operational tests and allow detecting software defects before deployment. However, these tests are very expensive and are too many to be run frequently within limited time constraints. In this paper, we investigate test case prioritization techniques to increase the ability to detect SDC regression faults with virtual tests earlier. Our approach, called SDC-Prioritizer, prioritizes virtual tests for SDCs according to static features of the roads we designed to be used within the driving scenarios. These features can be collected without running the tests, which means that they do not require past execution results. We introduce two evolutionary approaches to prioritize the test cases using diversity metrics (black-box heuristics) computed on these static features. These two approaches, called SO-SDC-Prioritizer and MO-SDC-Prioritizer, use single-objective and multi-objective genetic algorithms, respectively, to find trade-offs between executing the less expensive tests and the most diverse test cases earlier. Our empirical study conducted in the SDC domain shows that MO-SDC-Prioritizer significantly (p-value <= 0.1e−10) improves the ability to detect safety-critical failures at the same level of execution time compared to baselines: random and greedy-based test case orderings. Besides, our study indicates that multi-objective meta-heuristics outperform single-objective approaches when prioritizing simulation-based tests for SDCs. MO-SDC-Prioritizer prioritizes test cases with a large improvement in fault detection while its overhead (up to 0.45% of the test execution cost) is negligible.

1. Single and Multi-objective Test Cases
Prioritization for Self-driving Cars in
Virtual Environments
Christian Birchler, Sajad Khatiri, Pouria Derakhshanfar,
Sebastiano Panichella, Annibale Panichella
ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 32, Issue 2

2. Testing Autonomous Driving Systems
2
The New York Times, April 2021
npr, January 2022 Reuters, September 2021
19.05.23 ICSE 2023 Journal-First

3. Testing Autonomous Driving Systems
3
World of Agile, 2018
Testing on-the-road
!
Simulation-based Testing
19.05.23 ICSE 2023 Journal-First

4. Simulator: BeamNG
4
19.05.23 ICSE 2023 Journal-First

5. 5

6. 6

7. Regression Testing
7
Prioritization
Minimization
Selection
19.05.23 ICSE 2023 Journal-First
Yoo and Harman, STVR 2013

8. Test Prioritization
8
19.05.23 ICSE 2023 Journal-First

9. Test Prioritization
9
19.05.23 ICSE 2023 Journal-First

10. Test Prioritization
10
19.05.23 ICSE 2023 Journal-First

11. Execution Time is a Problem!
19.05.23 ICSE 2023 Journal-First 11
è In average 49 seconds for a single test execution with
moderate driving style (BeamNG.AI.AF1.5)!

12. Our intuition
12
Birchler et al., TOSEM 2023
è Optimization Problem
19.05.23 ICSE 2023 Journal-First
Minimize time Maximize diversity

13. Black-box Test Case Prioritization
19.05.23 ICSE 2023 Journal-First 13
Input diversity:
T1 T2 T3 T4
T1 - 5.91 6.40 12.40
T2 5.91 - 1.41 8.54
T3 6.40 1.41 - 8.30
T4 12.40 8.54 8.30 -
Euclidean distance between
input vectors of T4 and T2

14. Road Features
14
19.05.23 ICSE 2023 Journal-First

15. Measuring the Road Similarity
15
Road Features
We extract these road features from
simulation-based tests
Roads with more diverse set of
features correspond to more diverse
test case inputs
Diversity is measured using the
Euclidean distance between two
feature vectors

16. Measuring the Road Similarity
16
Road Features
19.05.23 ICSE 2023 Journal-First

17. Genetic Algorithm for Test Prioritization
17
19.05.23 ICSE 2023 Journal-First

18. Single-objective and Multi-Objective Test
Prioritization
18
Multi-objective
Single-objective
19.05.23 ICSE 2023 Journal-First
NSGA-II + Knee Point
à 1 function à 2 functions
Roulette wheel selection
probability:

19. Optimal Solution Selection
19.05.23 ICSE 2023 Journal-First 19

20. Multi-objective Genetic Algorithms
20
We can use Genetic Algorithms to search for
optimal test case permutation/prioritization of
the test case
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T8 … Tn
We can encode a solution as a permutation of n test cases
In this example, T4 is executed before T2 and after T1
19.05.23 ICSE 2023 Journal-First

21. Multi-objective Genetic Algorithms
21
We need search objectives to measure the quality
of each solution. We have two objectives:
Initial Population
Selection
Crossover
Mutation
End?
YES NO
Objective 1: the sum of the
distances between each pair
of consecutive test cases ti
and ti-1 in the permutation
Objective 2: the sum of the
(past) execution cost for all
test cases. The cost of the test
ti is divided by its position i in
the permutation
19.05.23 ICSE 2023 Journal-First

22. Multi-objective Genetic Algorithms
22
New solutions (offspring) are generated by
recombining the genes of two parent solutions
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
Parent solutions
19.05.23 ICSE 2023 Journal-First

23. Multi-objective Genetic Algorithms
23
New solutions (offspring) are generated by
recombining the genes of two parent solutions
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
Parent solutions
T1 T4 T2
Offspring
19.05.23 ICSE 2023 Journal-First

24. Multi-objective Genetic Algorithms
24
New solutions (offspring) are generated by
recombining the genes of two parent solutions
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
T1 T4 T2 T3 T6 T5
19.05.23 ICSE 2023 Journal-First
Parent solutions Offspring

25. Multi-objective Genetic Algorithms
25
New solutions (offspring) are generated by
recombining the genes of two parent solutions
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
Parent solutions
T1 T4 T2 T3 T6 T5
Offspring 1
T3 T1 T2 T4 T5 T6
Offspring 2
19.05.23 ICSE 2023 Journal-First

26. Multi-objective Genetic Algorithms
26
The newly generated permutation will be further
changes by applying random small mutation
Offspring
T1 T4 T2 T3 T6 T5
Mutated Offspring
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T6 T2 T3 T4 T5
19.05.23 ICSE 2023 Journal-First
The swap mutation operator randomly swaps two genes in a
(offspring) solution

27. Baselines
27
We compare the final solution produced by
GA against two baselines:
• Random prioritization (average of 100K
randomly generated permutations)
• Greedy algorithm, which build a
permutation by greedily selects the test
cases that is the more diverse from the
previously selected one
Safe Driving Style
19.05.23 ICSE 2023 Journal-First

28. 19.05.23 ICSE 2023 Journal-First 28

29. 19.05.23 ICSE 2023 Journal-First 29

30. 19.05.23 ICSE 2023 Journal-First 30

31. 19.05.23 ICSE 2023 Journal-First 31
Safe Driving Style

32. Cost-effectiveness
32
Safe Driving Style Aggressive Driving Style Driving Style of a different AI
19.05.23 ICSE 2023 Journal-First

33. Cost-effectiveness
33
SO-SDC-Prioritizer
MO-SDC-Prioritizer
Greedy
19.05.23 ICSE 2023 Journal-First

34. Wrap up
34
19.05.23 ICSE 2023 Journal-First
World of Agile, 2018
Testing on-the-road
!
Simulation-based Testing
è Optimization Problem
+ Q&A

35. 19.05.23 ICSE 2023 Journal-First 35