This is the final presentation of the master thesis "DriveBuild: Automation of Simulation-based Testing of Autonomous Vehicles". (Reference to thesis: <To be published>)

1. DriveBuild
Automation of Simulation-based Testing
of Autonomous Vehicles
�Stefan Huber
University of Passau
November 27, 2019

2. 2

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11

12. 12

13. 13

14. 14

16. 15

17. Goals of Testers
16

18. Shared Problems
17

19. Shared Problems
Setup simulator
17

20. Shared Problems
Setup simulator
Distribute and manage test runs
17

21. Shared Problems
Setup simulator
Distribute and manage test runs
Interact with AIs
17

22. Shared Problems
Setup simulator
Distribute and manage test runs
Interact with AIs
Collect data
17

23. Shared Problems
Setup simulator
Distribute and manage test runs
Interact with AIs
Collect data
⇒ Time consuming
17

24. Shared Problems
Setup simulator
Distribute and manage test runs
Interact with AIs
Collect data
⇒ Time consuming
⇒ Error prone
17

25. DriveBuild
Formalization
Execution
Runtime
Verification
Data
Collection
18

26. DriveBuild
Formalization
Execution
Runtime
Verification
Data
Collection
18

27. DriveBuild
Formalization
Execution
Runtime
Verification
Data
Collection
18

28. DriveBuild
Formalization
Execution
Runtime
Verification
Data
Collection
18

29. DriveBuild
Formalization
Execution
Runtime
Verification
Data
Collection
18

30. Why Formalization?
Formalization
19

31. Why XML?
Formalization
19

32. Execution
Why distribute?
20

33. Execution
Why micro
services?
20

34. RuntimeVerification
Why synchronous simulation?
21

35. DataCollection
Why serialization?
22

36. Evaluation
1 Generality of formalization
2 Scalability
3 How supportive is DriveBuild?
23

38. Generator ⇒ 1 method
AI ⇒ 2 methods
24

39. Generator ⇒ 1 method
AI ⇒ 2 methods
1114 tests
24

40. Conclusions
Abstraction supports various approaches
Sensor data
Feedback
Criteria
25

41. Conclusions
Abstraction supports various approaches
Sensor data
Feedback
Criteria
DriveBuild yields metrics for analysis
Efficiency/effectiveness of generators
Generated curvature profiles
Traveled distances of AVs
25

43. Timeout
26

44. 27

45. 28

46. Conclusions
Utilization of parallelism
Linear runtime behavior (until 10 tests)
Unreliable in VMs
29

47. Experience
30

48. Experience
Where did students struggle?
Implement interfaces
Exchange data
Specify required data
Specify criteria
Understand XML
30

49. Experience
Where did students struggle?
Implement interfaces
Exchange data
Specify required data
Specify criteria
Understand XML
Basically everywhere…
30

50. 31

51. 31

52. 31

53. 31

54. 31

55. On Going Work
32

56. On Going Work
32

57. On Going Work
32

58. On Going Work
32

59. Sources
33

60. Sources I
url: https://thelastdriverlicenseholder.files.wordpress.com/2018/11/waymo_10_million_
miles.png?w=1140 (visited on 11/10/2019).
url: https://i.kinja-img.com/gawker-media/image/upload/s--o04XLtGl--
/c_fill,fl_progressive,g_center,h_900,q_80,w_1600/lsaqi1oct4aguj5dd7jm.png (visited on
11/10/2019).
url: https://www.aeye.ai/wp-content/uploads/2019/06/Autonomous-Vehicle-Investments-Are-
Poised-to-Set-a-New-Record-in-2019.png (visited on 11/10/2019).
url: https://s3.amazonaws.com/cbi-research-portal-uploads/2017/06/07221804/corps-in-
autonomous-header.png (visited on 11/10/2019).
url: https://2.bp.blogspot.com/-GOeB3jwNrII/WTU6bstp-DI/AAAAAAABQzQ/d4FButF-
7YcwswbNbFoARckVl1R_Pv3vACLcB/s1600/system-failure.jpg (visited on 11/10/2019).
url: https://cdn.vox-cdn.com/thumbor/kCdl0QIIM11m5wTqjQy4wplYrU0=/800x0/filters:
no_upscale()/cdn.vox-cdn.com/uploads/chorus_asset/file/9194311/unnamed__1_.jpg (visited on
11/10/2019).
34

61. Sources II
url: http://cdn1.fiverrcdn.com/photos/1469216/v2_680/url.jpg?1369210040 (visited on
11/10/2019).
url: https://cdn.pixabay.com/photo/2017/01/10/23/01/icon-1970474_960_720.png (visited on
11/10/2019).
url: http://media-cdn.beamng.com/8X1IZnX5ubEmhA3p (visited on 11/11/2019).
url: http://2.bp.blogspot.com/_OPdnnH8MxiM/S-wiDO7DSCI/AAAAAAAAAZw/wi-
ZwqB5zok/s1600/Crazy_Road_Network.jpg (visited on 11/11/2019).
url: https://thefourthrevolution.org/wordpress/wp-content/uploads/2012/03/obstacle-on-
your-road.jpg (visited on 11/11/2019).
url: https://i.ytimg.com/vi/fXR5PH4Xu1E/maxresdefault.jpg (visited on 11/11/2019).
url: http://wavelab.uwaterloo.ca/wp-content/uploads/2016/08/nlp.jpg (visited on 11/11/2019).
url: https://i.ytimg.com/vi/xlAh7CIaMzA/maxresdefault.jpg (visited on 11/11/2019).
35

62. Sources III
url:
http://cimg1.ibsrv.net/cimg/www.carsdirect.com/680x382_85/349/Finish-Line-Wide--1349.jpg
(visited on 11/11/2019).
url: https://cdn.onlinewebfonts.com/svg/img_535483.png (visited on 11/11/2019).
url: http://cdn01.androidauthority.net/wp-
content/uploads/2015/07/location_marker_gps_shutterstock.jpg (visited on 11/12/2019).
url: https://www.homeschoolmath.net/teaching/g/measure_angles/measure-angle7.gif (visited
on 11/12/2019).
url: https://clipground.com/images/tacho-clipart-13.jpg (visited on 11/12/2019).
url: https://www.ifwemadeit.com/wp-content/uploads/2018/12/beamng-drive-350x350bb.jpg
(visited on 11/12/2019).
36

63. Sources IV
url: https://media.springernature.com/lw785/springer-
static/image/art%3A10.1007%2Fs12544-018-0284-x/MediaObjects/12544_2018_284_Fig1_HTML.gif
(visited on 11/12/2019).
url: https://clipground.com/images/seminar-clipart-7.jpg (visited on 11/12/2019).
url: https://pictogram-free.com/material/244-pictogram-free.jpg (visited on 11/12/2019).
url: http://www.clker.com/cliparts/D/6/P/c/j/S/gears-hi.png (visited on 11/12/2019).
url: https://www.appzaps.co.uk/relaxing-your-mind/wp-content/uploads/edd/2013/04/after-
a-difficult-day.jpg (visited on 11/20/2019).
url: https://assets.rocketstock.com/uploads/2017/06/Interface-Hero.jpg (visited on
11/20/2019).
url: https://www.statista.com/chart/13450/perceived-safety-of-self-driving-cars/ (visited
on 11/25/2019).
37

64. Sources V
url: https://copyforchange.files.wordpress.com/2015/04/save-the-world-2.jpg (visited on
11/25/2019).
url: https://www.statista.com/statistics/191530/fatal-passenger-car-crashes-in-the-us/
(visited on 11/25/2019).
url: https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/autonomousveh_ol316+ (visited
on 11/25/2019).
url: http://sitevolt.com/ontariocarsblog/wp-content/uploads/sites/145/2018/12/self-
parking-vehicles-i-3-1000x675.jpg (visited on 11/25/2019).
url: https://officechai.com/wp-content/uploads/2017/11/testing.jpg (visited on 11/25/2019).
url: https://www.dailydot.com/wp-content/uploads/016/84/2ef78b50bedc31ec-2048x1024.jpg
(visited on 11/25/2019).
url: http://dontlettheforexdriveyouupthewall.com/wp-content/uploads/2015/01/clocks.jpg
(visited on 11/25/2019).
38

65. Sources VI
url:
https://www.letstalkcoaching.com/wp-content/uploads/How-good-are-you-at-I-dont-know.jpg
(visited on 11/25/2019).
url: https://abovethelaw.com/wp-content/uploads/2011/06/Lots-of-Papers.jpg (visited on
11/25/2019).
url: https://www.froglogic.com/wp-content/uploads/2016/05/Distributed-770x770.png (visited
on 11/26/2019).
url: http://cdn.onlinewebfonts.com/svg/img_71102.png (visited on 11/26/2019).
url: http://comunytek.com/wp-content/uploads/2017/03/Microservices.png (visited on
11/26/2019).
url: https://www.telegraph.co.uk/content/dam/news/2016/04/10/chess-
xlarge_trans_NvBQzQNjv4BqgsaO8O78rhmZrDxTlQBjdLdu0TL-Cg_AMOUqySXmFgU.jpg (visited on
11/26/2019).
39

66. Sources VII
url: https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Text-xml.svg/1116px-Text-
xml.svg.png (visited on 11/26/2019).
url: https://things.eu-1.bosch-iot-
suite.com/dokuwiki/lib/exe/fetch.php?cache=&media=3_basic_concepts:topo-schema-icon.png
(visited on 11/26/2019).
url: http://worldartsme.com/images/clip-art-no-workout-today-clipart-1.jpg (visited on
11/26/2019).
url: https://icdn-3.motor1.com/images/mgl/KVEmG/s1/nissan-tsuru-crash-test.jpg (visited on
11/26/2019).
url: http://www.digitaldataservices.com/images/analyze.png (visited on 11/26/2019).
url: https://clipground.com/images/curving-clipart-12.jpg (visited on 11/26/2019).
url: https://www.seoclerk.com/pics/562891-1aRurq1507581612.png (visited on 11/26/2019).
40

67. Sources VIII
url: http://pngimg.com/uploads/equals/equals_PNG46.png (visited on 11/26/2019).
url: https://png.icons8.com/win8/1600/107C10/checkmark (visited on 11/26/2019).
url: http://www.clker.com/cliparts/L/j/U/f/3/O/thunder-bolt-plain-hi.png (visited on
11/26/2019).
url: http://i.stack.imgur.com/6Ssd1.png (visited on 11/27/2019).
url: http://media-cdn.beamng.com/dXOTx2tC8AWW5bsn/screenshot_2018-12-14_12-21-57.jpg
(visited on 11/27/2019).
url: https://www.paramedics.org/wp-content/uploads/2017/01/research.jpg (visited on
11/27/2019).
url: https://png.pngtree.com/svg/20160606/9f4dd6559e.png (visited on 11/27/2019).
41

68. Backup Slides
42

69. Environment Test Setup
Formalization

70. Environment Test Setup
Formalization

71. Environment Test Setup
Formalization

72. Environment Test Setup
Formalization

73. Environment Test Setup
Formalization

74. Environment Test Setup
Formalization

75. Environment Test Setup
Formalization

76. Environment Test Setup
Formalization
43

77. ...
Execution
44

78. Verify
criteria
Request
AIs
Control
Step
Pause
RuntimeVerification
45

79. :Client :MainApp
/runTests
SubmissionResult
/ai/waitForSimulatorRequest
Wait for requestAiFor
SimStateResponse
/ai/requestData
DataResponse
Calculate control commands
/ai/control
Apply control commands
Void
Handle AIHandle AI For each AI in parallel
RuntimeVerification
46

80. DataCollection
47

81. 48

82. 49

83. 50

84. 51

85. 52

86. 53

87. DeepDriving Example
54