The autonomous vehicles need to be validated with a reliable and repeatable methodology to be accepted by the public. In this paper, we present our methodology to develop a validation regime for the decision making system of an autonomous vehicle operating in a certain road network. The methodology starts with the thorough analysis of the selected roads. Then these roads are divided into atomic units, each of which is unique for testing purposes. The atomic units are modeled in simulation using our existing scenario generation framework, which allows for the stress testing and edge scenario discovery. Then the decision making software of the vehicle under test is taken in the loop to execute the tests. The methodology is applied to the autonomous campus shuttle currently operating at the Tallinn University of Technology campus. The shuttle’s route is analyzed and modeled in simulation to create the testing scenarios. The methodology will be a complete validation scheme as the shuttle is tested in the field with a variety of the corner test cases discovered by our methodology.

1. Development of a Validation Regime for an
Autonomous Campus Shuttle
03/13/2020
RKA 2016-06-22
Christopher Medrano-Berumen˚, Mohsen Malayjerdi˟, M. Ilhan Akbas*,
Raivo Sell˟ and Rahul Razdan˚
*Embry-Riddle Aeronautical University
˚Florida Polytechnic University
˟Tallinn University of Technology

2. Outline
• Application Domain
• Problem Definition and Objective
• Approach
• Implementation
• Summary and Future Work

3. Application Domain
• Autonomous Vehicle Validation is Critical
– AVs will make a significant impact on various industries
– ADAS systems have been deployed commercially
– There are AV companies testing on streets
– More rigorous testing is required
• Tallinn University of Technology (TalTech)
– TalTech (Estonia) has an operational AV shuttle ISEAUTO
– The campus of TalTech is the operation domain for the ISEAUTO
– Slower speeds and low complexity provides an ideal testing environment

4. Problem Definition
• Problem
– No fundamental structure exists to demonstrate the robustness of an AV
– The shadow driving testing is not sufficient and simulation testing is required
– Given the operation domain, there is no guidance on the exact type and
number of scenarios to test an AV in this domain
• Objective
– Creation of a testing regime for a given operation design domain

5. Approach
• Project and Methodology
– Phase 1: The TalTech route and environment is modeled in the validation flow
to build a framework for coverage and generate representative test cases
– Phase 2: Test scenarios are fed into software-in-the-loop (SiL) simulation to
test the decision-making system of the shuttle
– Phase 3: The physical demonstration in the TalTech campus
The progress towards the first two phases are given in this presentation

6. Approach
• Development of a Validation Regime
– The scenario generation focuses on the decision-making component
– We had developed a semantic language to generate validation scenarios*
– The goal of the approach is to create a validation regime using this language
specifically for the ISEAUTO
*C. Medrano-Berumen and M. I. Akbas, “Abstract Simulation Scenario Generation for Autonomous Vehicle Verification,” in Proceedings of the
IEEE SoutheastCon, IEEE, April 2019.

7. Approach
• Validation Approach
– Simulation system is aimed for:
− Modeling of relevant driving scenario environment
− Analysis of sections along the pilot route for safety
− Simulation of critical scenarios and identification of improvement options
− Software-in-the-loop (SiL) testing of the vehicle using the generated scenarios
− Use of collected data through a physical AV test vehicle
– The validation approach will include people and vehicle movement
– A solution with lowest risk level is the goal

8. Approach
• Scenario Generation for TalTech AV Route
– We break down the route of TalTech AV shuttle into segments
– Segments define the road pieces needed to model the route
– MATLAB AD Toolbox is used to create simulation scenarios

9. Approach
• Unique Road Segments
– Side Lot Enter
– Single Pedestrian Crosswalk
– Three-way Intersection
– Extra Road Area
– Open Area
– Garbage Truck Entry
– Sidewalk Entry
– Grass Path Entry

10. Approach
• Each Unique Road Segment is Modeled in MATLAB AD Toolbox
– A simulation model database is created
– The database will grow as new operation domains are modeled
– Larger database will allow efficient modeling of new domains

11. Approach
• New Road Piece Attributes: Added new attributes to the existing
validation methodology
– Median
– Outlet
– AV Only Segment

12. Approach
• Software-in-the-Loop Testing with Generated Scenarios
– SiL is a critical part of self-driving technology development.
– ISEAUTO uses Autoware platform to associate sensor data for a unified
observation of the environment and decision making
– Our scenarios in MATLAB AD Toolbox are used by Autoware and ROS

13. Approach
• Software-in-the-Loop Testing with Generated Scenarios
– Control algorithms are developed through simulation of different scenarios
– The algorithms are modified according to the results of simulation scenarios
– This process continues until an adequate number of cases are examined
– The algorithms are then deployed to the real vehicle for test rides

14. Implementation Example
• Scenario Generation for a Specific Segment
– A parking lot entrance on TalTech route
– All related features of the segment are parameterized and modeled
– The new atomic road segment is added to the model database

15. Implementation Example
• Translating the Road Pieces
– The components of the parking lot entrance is modeled and a simulation
scenario is generated
– The elements of the scenario are converted into our semantic language

16. Summary and Future Work
• Validation of AVs is critical for their deployment in real life
• We build a framework to create a validation regime for a given AV
operation domain
• We use TalTech campus as the operation domain and ISEAUTO as
the AV under test
• We plan to use the simulated tests for real-life testing of ISEAUTO
• We also plan to generalize the approach for other operational
domains

17. Thank You!
For more information about our research:
Dr. M. Ilhan Akbas
Assistant Professor of Electrical and Computer Engineering
Embry-Riddle Aeronautical University
• akbasm@erau.edu
• https://sites.google.com/site/miakbas/