IEEE International Conference on Software Testing, Verification and Validation (ICST) 2025 4 April 2025

1. ICST TOOL COMPETITION: SELF-DRIVING
CAR TESTING
Christian Birchler
University of Bern
Zurich University of Applied Sciences
Switzerland
Stefan Klikovits
Johannes Keppler University Linz
Austria
Mattia Fazzini
University of Minnesota
USA
Sebastiano Panichella
University of Bern
Switzerland
1

2. SELF-DRIVING CAR COMPETITION
2

3. SELF-DRIVING CAR COMPETITION
CONTEXT
2.1

4. 3

5. 3.1

6. Cost
3.2

7. Cost
Replicability
3.3

8. Cost
Replicability
3.4

9. Cost
Replicability
3.5

10. Cost
Replicability
3.6

11. Cost
Replicability
4

12. Cost
Replicability
5

13. BeamNG.tech Simulator
6

14. 7

15. 8

16. HOW IS A TEST DEFINED?
9

17. HOW IS A TEST DEFINED?
9.1

18. HOW IS A TEST DEFINED?
10

19. WHEN IS A TEST FAILING OR PASSING?
11

20. 12

21. 13

22. PASSED
13.1

23. 14

24. 15

25. FAILED
15.1

26. 16

27. REGRESSION TESTING
17

28. REGRESSION TESTING
18

29. REGRESSION TESTING
19

30. REGRESSION TESTING
20

31. REGRESSION TESTING
21

32. 22

33. What tests should be selected?
22.1

34. What tests should be selected?
How should tests be selected?
22.2

35. 23

36. Competition code on GitHub
23.1

37. Competition code on GitHub
Troubleshooting with Issues and Discussion Forum
23.2

38. Competition code on GitHub
Troubleshooting with Issues and Discussion Forum
6 Submissions as Pull Request
23.3

39. Competition code on GitHub
Troubleshooting with Issues and Discussion Forum
6 Submissions as Pull Request
Docker Images of all Tools!
23.4

40. INFRASTRUCTURE
ToolX
Evaluator
ToolX
Evaluator
initialize
ok
select
return selection
Evaluator (Docker Container) gRPC ToolX (Docker Container)
Docker with Nvidia Runtime
24

41. PROTOCOL BUFFERS
syntax = "proto3";
service CompetitionTool {
rpc Name(Empty) returns (NameReply) {}
rpc Initialize (stream Oracle) returns (InitializationReply) {}
rpc Select (stream SDCTestCase) returns (stream SelectionReply) {}
}
message Empty {}
message NameReply {
string name = 1;
}
message Oracle {
SDCTestCase testCase = 1;
bool hasFailed = 2;
}
message SDCTestCase {
string testId = 1;
repeated RoadPoint roadPoints = 2;
}
message RoadPoint {
int64 sequenceNumber = 1;
float x = 2;
float y = 3;
}
message InitializationReply {
bool ok = 1;
}
message SelectionReply {
string testId = 1;
}
25

42. syntax = "proto3";
service CompetitionTool {
rpc Name(Empty) returns (NameReply) {}
rpc Initialize (stream Oracle) returns (InitializationReply) {}
rpc Select (stream SDCTestCase) returns (stream SelectionReply) {}
}
...
26

43. Now, let us see the tools!
27

44. What are the evaluation metrics?
@dataclass
class EvaluationReport:
"""This class holds evaluation metrics of a tool."""
tool_name: str
benchmark: str
test_suite_cnt: int
selection_cnt: int
time_to_initialize: float
time_to_select_tests: float
time_to_fault_ratio: float
fault_to_selection_ratio: float
diversity: float
28

45. BENCHMARK
32,580 Test Cases
Generated by three test generators
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
29

46. EXPERIMENTS
The experiments are conducted on an Open Stack
virtual machine (VM) with 16GB of RAM, eight virtual
CPUs, and an Nvidia Tesla T4 GPU.
30

47. FINDING 1
On average, the random test selector selects the most
test cases (µ = 88.17) compared to all tools.
31

48. FINDING 2
DETOUR required on average (µ = 61.77sec) to initialize
the tool with the provided initialization data. All other
tools did not require more than a second.
32

49. FINDING 3
For the selection phase, DETOUR requires, on average,
most of the time (µ = 15.67sec) followed by DRVN Tool
(µ = 10.49sec). The remaining tools did not require
more than three seconds on average.
33

50. FINDING 4
The random selector has, on average, the worst
Simulation Time to Fault Ratio with µ = 156.23. With an
average of µ = 65.39, the ITS4SDC tool has the best
ratio.
34

51. FINDING 5
The ITS4SDC tool has, on average, the best Fault to
Selection Ratio (µ = 0.8) and the random baseline the
worst with µ = 0.38.
35

52. FINDING 6
There is no clear indication that the diversity metric
differs among all tools.
36

53. 37

54. ITS4SDC
Ali ihsan Güllü, Faiz Ali Shah, Dietmar Pfahl
University of Tartu, Estonia
37.1

55. 38

56. 39

57. THANK YOU ALL!
40

58. THANK YOU ALL!
Any feedback and/or ideas are welcome for future
editions.
github.com/christianbirchler-org/sdc-testing-competition
40.1