This document outlines a strategy for scenario-based validation of autonomous vehicles using the Eagle strategy with local search for selecting effective test scenarios. It discusses data from executed scenarios to estimate edge scenario boundaries and visualizes bug profiles using Voronoi diagrams. The conclusion emphasizes the efficiency of this approach in prioritizing interesting scenarios compared to traditional random sampling methods.

1. Eagle Strategy with Local Search
for Scenario Based Validation of
Autonomous Vehicles
Quentin Goss and Mustafa Ilhan Akbas
Department of Electrical Engineering and Computer Science
Embry-Riddle Aeronautical University
Daytona Beach, Florida
gossq@my.erau.edu, akbasm@erau.edu

2. Outline
• Introduction
• Eagle Strategy with Local Search
• Estimating Edge Scenario Boundaries
• Demonstration
• Conclusion

3. Introduction

4. Introduction
Previous Research
• Foretify + MSDL by Foretellix
• Beta Testing
• Modeling AV scenarios based on real-life CAV Accidents
• accident report --> scenario --> simulation
• General-case + modular
• IEEE IV 2021

5. Introduction
Previous Research
• Foretify + MSDL by Foretellix
• Beta Testing
• Modeling AV scenarios based on real-life CAV Accidents
• accident report --> scenario --> simulation
• General-case + modular
• IEEE IV 2021
• Observations
• Black box scenarios
• Parameters + Outcome
• Run in batches
• Lots of them are uneventful/uninteresting.

6. Introduction
Previous Research
• Foretify + MSDL by Foretellix
• Beta Testing
• Modeling AV scenarios based on real-life CAV Accidents
• accident report --> scenario --> simulation
• General-case + modular
• IEEE IV 2021
• Observations
• Black box scenarios
• Parameters + Outcome
• Run in batches
• Lots of them are uneventful/uninteresting.
How can we build a testing regime
which selects scenario parameters
in such a way that prioritizes
finding interesting scenarios first?

7. Introduction
• Scope/Focus of Research
• Autonomous Vehicles (AV) Scenarios
• Complex AV Scenarios
• Blackbox AV Scenarios

8. Introduction
• Scope/Focus of Research
• Autonomous Vehicles (AV) Scenarios
• Complex AV Scenarios
• Blackbox AV Scenarios
• AV Scenarios have many parameters.
• Complexity increases with realism.

9. Introduction
• Scope/Focus of Research
• Autonomous Vehicles (AV) Scenarios
• Complex AV Scenarios
• Blackbox AV Scenarios
• AV Scenarios have many parameters.
• Complexity increases with realism.
• How to choose scenarios?
• Interesting
• Efficient
• How to process/visualize it?

10. Eagle Strategy with Local Search

11. Choosing Scenarios
with Eagle Strategy with Local Search
• Eagle Strategy (Yang and Deb, 2010)
• Stochastic Optimization
• Levy Walk + Firefly Algorithm
• Eagles' search for food.

12. Choosing Scenarios
with Eagle Strategy with Local Search

13. Choosing Scenarios
with Eagle Strategy with Local Search

14. Choosing Scenarios
with Eagle Strategy with Local Search

15. Choosing Scenarios
with Eagle Strategy with Local Search
Choose a point/parameters
far away from the last.

16. Choosing Scenarios
with Eagle Strategy with Local Search
Choose a point/parameters
far away from the last.
Choose a point/parameters
near the last known bug.

17. Parameter Selection
and representing scenarios by their parameters.
• Representation by parameters.
• How are scenarios parameters selected currently?
• Exhaustively
• Randomly
• Quasi-randomly

18. Parameter Selection
and representing scenarios by their parameters.
• Representation by parameters.
• How are scenarios parameters selected currently?
• Exhaustively
• Randomly
• Quasi-randomly
But we want to find the interesting scenarios!
• Bug Scenarios

20. Estimating Edge Scenario Boundaries

21. • Data from executed scenarios:
• Input parameters
• Outcome
• Properties
• N-dimensional
• Locations of bug scenarios unknown.
Estimating Edge Scenario Boundaries

22. • Data from executed scenarios:
• Input parameters
• Outcome
• Properties
• N-dimensional
• Locations of bug scenarios unknown.
Estimating Edge Scenario Boundaries
Review of data collected from.
1. How to visualize?
2. What data structure?
3. How can we estimate where
the edge scenario boundaries
are?

23. Estimating Edge Scenario Boundaries
The Voronoi Diagram

24. Estimating Edge Scenario Boundaries
The Graph

25. Demonstration

26. Demonstration
Scenario Design + Bug Classification
• Car 1 begins [0..1000]m from node C on edge 1 and
has a constant speed of [1..100]kph.
• Car 2 begins [0..1000]m from node C on edge 2 and
has a constant speed of [1..100]kph.

27. Demonstration
Comparison of Approaches

28. Demonstration
Visualizing the Bug Profile

29. Conclusion

30. Conclusion
• Blackbox Approach for AV Scenario Generation
• Parameter Range Sampling
• Efficiently prioritizes "interesting scenarios"
• Better than Random or Quasi-random range sampling
• More than double!
• Bug Profile Generation
• Voronoi Diagram
• Graph
• Scalable to N-dimensions!

31. Questions ?