The document discusses the development of a modular framework for validating autonomous vehicles (AVs) using accident data, emphasizing the classification and parameterization of scenarios. It introduces an atomic block framework and outlines methods for scenario selection, execution, and assessment metrics, including collision detection and high-risk moment detection. The end result is a comprehensive database of scenarios sourced from real accident reports that facilitate AV testing and safety improvements.

1. Generation of Modular and Measurable
Validation Scenarios for Autonomous Vehicles
Using Accident Data
Quentin Goss, Yara AlRashidi, and Mustafa Ilhan Akbas
Department of Electrical Engineering and Computer Science
Embry-Riddle Aeronautical University, Daytona Beach, Florida
{gossq, alrashy1}@my.erau.edu, akbasm@erau.edu
32nd IEEE Intelligent Vehicles Symposium

2. Contents
● Levels of Abstraction
● Atomic Block framework
○ The Atomic Block
○ From Scenario Source to Assessment techniques
● Conclusion

3. Three Levels of Abstraction
Functional Concrete
Logical

4. Framework

5. Sources for Scenarios

6. Classification
Who/What
● Actors
○ Device under test (DUT)
○ Non-player character (NPC)
Where
S - Actors travel same path.
T- 3-way intersection
X - 4-way intersection
M - 5-way intersection

7. Classification
Atomic Block IDs
bl_2TP3
● Actors: (4) DUT, NPC1, NPC2, NPC3 (pedestrian)
● Location: (T) 3-way intersection
● Pedestrian?: Yes
● Index: (3) After bl_2TP1 and bl_2TP2.
bl_<# non-dut/pedestrian actors><location><pedestrian><index>

8. Parameterization
Actors Types
Car
● 0-180 kmh
● -32.40-10.44 kmh/s
● ….

9. Parameterization
Atomic Blocks
● Actors position/speeds
● Weather
● Traffic
● Injected Faults
● ….

10. Parameterization
“An AV was involved in a collision while operating in autonomous mode,
traveling westbound on McAllister St. at the intersection of Polk Street.
Another vehicle attempted to pass the AV on the right but crossed into its
lane, scraping the AV’s right sensor. There were no injuries, police were
not called.”

11. Parameterization

12. Parameterization

13. Atomic Blocks
Scenario Database
+
+
+ + +
Complex Logical Scenarios

14. Selection of Concrete Scenarios

15. Concrete Scenario Execution

16. AV Assessment

17. AV Assessment
Coverage Metrics

18. AV Assessment
Understanding Coverage Metrics

19. AV Assessment
Collision Detection
● 150 concrete scenario tests
● Count collisions

20. AV Assessment
High Risk Moment Detection
● Simulated DUT is not the physical DUT
● No faults is not injected faults
● Clear weather is not icy weather
● ….
But we can still find potential edge scenarios.

21. AV Assessment
High Risk Moment Detection
● Key Performance Indicator (KPI)
● Any metric
○ Time-to-collision
○ Comfortable braking/acceleration/steering
○ Unsatisfactory recovery after injected fault
○ ….
Anything that is pass/fail.

22. AV Assessment
Experiment
● For each bl_1, bl_2, bl_1 + bl_2
○ 150 Concrete Scenarios
○ 3 Street networks
○ Record (x,y) and speed 20ms interval
● Post-processing
○ Find KPI each interval
■ Distance between NPC and DUT
■ Time-to-Collision
○ Count tests with min(KPI) under threshold

23. AV Assessment

24. Conclusion
● Atomic Block
○ Modular and Measurable
● Framework (Source to Assessment)
● Sourced from AV accident reports.
○ Florida, California, Texas
● Classification/Parameterization
● Database of Scenarios
○ Atomic Blocks + Complex logical scenarios with ranges
● AV Assessment
○ Tuning scenarios
○ Identify edge scenarios

25. Special thanks to Foretellix Ltd.!

26. Questions?