Digital Twins Testing Yields Different Results
The increasing levels of software- and data-intensive driving automation call for an evolution of automotive software testing. As a recommended practice of the Verification and Validation (V&V) process of ISO/PAS 21448, a candidate standard for safety of the intended functionality for road vehicles, simulation-based testing has the potential to reduce both risks and costs. There is a growing body of research on devising test automation techniques using simulators for Advanced Driver-Assistance Systems (ADAS). However, how similar are the results if the same test scenarios are executed in different simulators? We conduct a replication study of applying a Search-Based Software Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision detection system) using two different commercial simulators, namely, TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we compare critical test scenarios generated using our SBST solution in these two simulators. We show that SBST can be used to effectively and efficiently generate critical test scenarios in both simulators, and the test results obtained from the two simulators can reveal several weaknesses of the ADAS under test. However, executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs, in particular, related to (1) safety violations revealed by tests, and (2) dynamics of cars and pedestrians. Based on our findings, we recommend future V&V plans to include multiple simulators to support robust simulation-based testing and to base test objectives on measures that are less dependant on the internals of the simulators.
Recommended
Recommended
More Related Content
What's hot
What's hot (14)
Similar to Digital Twins Testing Yields Different Results
Similar to Digital Twins Testing Yields Different Results (20)
More from Markus Borg
More from Markus Borg (20)
Recently uploaded
Recently uploaded (20)
Digital Twins Testing Yields Different Results
- 1. Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators ICST’21 April 14, 2021 RISE Research Institutes of Sweden Markus Borg, Raja Ben Abdessalem, Shiva Nejati, François-Xavier Jegeden, and Donghwan Shin
- 2. One scenario, two simulators – quite different test results
- 4. Who is Markus? • Development engineer, ABB 2007-2010 – Process automation – Editor and compiler development • PhD student, Lund University 2010-2015 – Requirements engineering and testing – Traceability, change impact analysis • Senior researcher, RISE 2015- – Software engineering for AI/ML
- 9. 9 Early testing • Simulators let you test whatever is ready whenever you need to Effective testing • Simulators let you execute scenarios that matter! Efficient testing • Simulators cost CPU and GPU – and scale Why testing in simulators? They enable 3Es!
- 10. Further perks 10 Simulators… • … provide annotated data • … are intrinsically GDPR compliant (CC BY 2.0 Flickr:Brickset)
- 11. Siemens/TASS PreScan - Seamless MATLAB/Simulink integration ESI Pro-SiVIC - High-fidelity sensor modelling BeamNG - Strong focus on crash modelling CARLA - Open source, growing community Custom simulator with Unreal Engine - Focus on photo realism LGSVL (by LG Electronics) – Too early for me to say… Our hands-on simulator experience 11
- 12. Does ADAS testing generalize X-Sim? 12
- 13. Method
- 14. Research visit in Luxembourg
- 15. PeVi
- 18. 18 Reproduction: A Minimal Scene
- 20. Reproduction: Search-Based Software Testing 20 • NSGA-II • 150 min search budget • Fitness functions – Two distances – Time to collision (TTC)
- 21. 21
- 22. RQ1
- 23. 23 RQ1. Is SBST effective if we replace PreScan with Pro-SiVIC? – Analyze 400 scenarios from NSGA-II in PreScan and Pro-SiVIC, respectively.
- 24. The principal findings from PreScan can be reproduced in Pro-SiVIC Results 24 RQ1
- 25. RQ2
- 26. 26 RQ2. Is the diagnostic info from SBST using PreScan reproducible in Pro-SiVIC? – Plot which inputs lead to safety violations – Learn decision trees to explain patterns
- 27. 27
- 28. 28
- 29. 29
- 30. 30
- 31. 31 Characteristics of safety violations differ. Only agreement regarding high vehicle speeds. RQ2
- 32. RQ3
- 33. 33 RQ3. Can critical test scenarios identified in Sim A be reproduced in Sim B? – Reproduce the same scenarios in the other simulator
- 35. • Unsafe scenario – Critical scenario, PeVi fails • Safe scenarios – Critical scenario, PeVi works – Non-critical scenario Three Possible Scenario Outcomes 35
- 36. 36 171 229 164 236 284 116 188 212 PreScan Pro-SiVIC Reproducing Pro-SiVIC Scenarios in PreScan 236 Unsafe Scenarios (1a) Unsafe 212 (1b) Critical but Safe 0 (1c) Non-Critical 24 164 Safe Scenarios (2a) Unsafe 0 (2b) Critical but Safe 64 (2c) Non-critical 100 X-Sim Reproduction Reproducing PreScan Scenarios in Pro-SiVIC 229 Unsafe Scenarios (1a) Unsafe 78 (1b) Critical but Safe 45 (1c) Non-Critical 106 171 Safe Scenarios (2a) Unsafe 38 (2b) Critical but Safe 114 (2c) Non-critical 19 X-sim reproduction differs in distances and PeVi detections. RQ3
- 37. Implications for Research and Practice
- 38. (CC BY 4.0 G. Chernilevsky) Simulators are different Try the ADAS in another one before on-road testing
- 39. (CC BY-NC-ND 4.0 US Army Garrison Casey) Select fitness functions that don’t depend on simulator internals
- 40. One scenario, two simulators – quite different test results markus.borg@ri.se @mrksbrg mrksbrg.com RISE Research Institutes of Sweden https://github.com/mrksbrg/adas-pro-sivic