Test and Evaluation Tools for Unmanned Vehicles

1. Delivering Test and Evaluation
Tools for Autonomous
Unmanned Vehicles to the
Fleet[1]
Nitin Varghese
University of Passau
1Slide

2. Contents
 Motivation
 Technology
 Introduction
 RAPT
 GPR SEARCH
 DB SCAN
 Scenario and Inference
 Conclusion
 Questions
2Slide

3. Motivation
 Autonomy in Vehicles
 NP hard Problem
[1][2]
 Test and Evaluation tools for maritime,
air and ground Autonomous Systems
 Limited Test to Evaluate Test Systems
Important or Not Important
 Cost and Reliability
Slide 3

4. Technology
 BeamNG Research
Open Platform solve problems of
mostly vehicle autonomy
 Python
API interface language
 MATLAB
Data Transformations and
Visualizations
Slide 4

5. Introduction
 Paper focus: relation between Scenario
configuration and Performance
 Not Focused on Fault Detection or Model
Checking
 Identify regions indicating performance
boundaries
 Classify test cases based on
performance
Slide 5

6. RAPT
[1]
Slide 5

7. GPR Search
 Gaussian Function[4]
 Performance Boundary: Small
Changes in Scenario large
performance Difference
 Exploration vs Exploitation
 Mean,gradient of Mean, Covariance
leads to uniform sampled input
Slide 7

8. Input to the Search Algorithm
Slide 7

9. DB SCAN
 It is a density-based
clustering algorithm
[4]
 Boundary Detection Classification
Problem
 Performance modes treated as
continuous space thus we use DB scan
 K nearest neighbour to identify final
Slide 9

10. Completed Process
Slide 9

11. 11Slide

12. 12Slide

13. 13Slide

14. Slide 13

15. Conclusion
 Narrows down the input space
 Helps Testers focus on the important
scenarios
 Relating Scenarios help design
suitable test cases
 Reduce Cost and increase reliability
Slide 14

16. Questions
Slide 15

17. References
 Mullins, G. E., Stankiewicz, P. G., Hawthorne, R. C., Appler, J. D., Biggins, M. H., Chiou, K., ... &
Watkins, A. S. (2017). Delivering Test and Evaluation Tools for Autonomous Unmanned Vehicles to the
Fleet. JOHNS HOPKINS APL TECHNICAL DIGEST, 33(4), 279-288. [1]
 Abdessalem, R. B., Nejati, S., Briand, L. C., & Stifter, T. (2018, May). Testing vision-based control
systems using learnable evolutionary algorithms. In 2018 IEEE/ACM 40th International Conference on
Software Engineering (ICSE) (pp. 1016-1026). IEEE.[2]
 Wittmann, D., Wang, C., & Lienkamp, M. (2015). Definition and identification of system boundaries of
highly automated driving. In 7. Tagung Fahrerassistenz[3]
 Ester, Martin; Kriegel, Hans-Peter; Sander, Jörg; Xu, Xiaowei (1996). Simoudis, Evangelos; Han,
Jiawei; Fayyad, Usama M., eds. A density-based algorithm for discovering clusters in large spatial
databases with noise. Proceedings of the Second International Conference on Knowledge Discovery
and Data Mining (KDD-96). AAAI Press. pp. 226–231. CiteSeerX 10.1.1.121.9220. ISBN 1-57735-004-
9.[4]
 https://en.wikipedia.org/wiki/Gaussian_function[5]
17Slide

18. Thank You!
Page 18