"How to Test and Validate an Automated Driving System," a Presentation from MathWorks

Copyright © 2017 MathWorks, Inc 1
How to Test and Validate an Automated
Driving System
Avinash Nehemiah
May 2017

1. Obvious Reasons
• Safety critical software driving cars on public roads
• Human lives depend on this
• Integration of new sensors and software into existing vehicles
2. Less Obvious Reasons
• Test the interaction of automated driving with human drivers
• Understand the driving passenger experience
Why Test Automated Driving Systems

1. In-vehicle testing on a road or test track
2. Open loop testing of defined scenarios
3. Virtual driving
Audience Poll: Current Testing Methodologies
Answer: All are necessary for different phases of development.

• Automated driving sub-systems
• Most common challenges faced testing automated driving
• Perception system testing
• Controls and system-level testing
• Summary
Talk Outline
How to Test and Validate an Automated Driving System

Simplified Automated Driving Sub-Systems
Embedded Perception Software
SensorFusion
PlanningandControl
(Steering,Brakingetc.)
LiDAR
Camera
Radar
Actuation/
Driving

• Challenges faced testing automated driving
• Summary
Talk Outline

1. In-vehicle testing is expensive and time consuming
2. Too dangerous for testing corner-cases
3. Difficult to validate “black box” perception algorithms
Challenges Faced
Test and Validate an Automated Driving System

• Summary
Talk Outline

Perception systems include vision, deep learning, sensor fusion, etc.
Testing Perception Systems
Different flavors of testing:
1. Ground truth labeling
• Used for: computer vision , radar and deep learning
2. Scenario testing with synthetic data
• Used for: sensor fusion, control algorithms

How do auto companies verify changes to
perception algorithms ?

Test
Fleet
Video Data
RADAR
Data
CAN
Logs
t=1454335205,
{‘Pedestrian’,10m,(12
3,450)},{‘Car’,100m,(1
12,235)}
…
…
…
t= 1454335215,
{‘Road’,’wet’},
light=‘Day’
What is Ground Truth Labeling ?
Query specific driving
scenarios ( E.g., Car
approaching stop sign
with speed 30 mph )
Characterize system
performance (E.g.,
pedestrian detection in
day vs. night )
Comprehensive
regression testing new
ADAS features
Recorded Sensor Data Human-verified ground “truth”
Manual labeling
(1000s of hours
of tedious work)

Re-simulation and Comparison vs.
Ground Truth
Recorded Sensor Data
Ground Truth
Perception
System
Embedded Hardware, C
Code, MATLAB model
COMPARE
Most common workflow used to test “black box” perception systems
Uses millions of miles of test data

Testing vs. Ground Truth
How can I verify this
detection is correct?

Example: Ground Truth Labeling & Re-Simulation

Case Study: Traffic Sign Recognition (Continental)
System Includes:
• Computer vision
• Machine
learning
Traffic Sign Recognition for Driver Assistance Systems
Dr. Alexander Behrens, Continental AG
MAC Stuttgart , 2015

Case Study: Traffic Sign Recognition (Continental)
Traffic Sign Recognition for Driver Assistance Systems
Dr. Alexander Behrens, Continental AG
MAC Stuttgart, 2015
Testing vs.
ground truth
Perception
Systems

Case Study: Advanced Emergency Braking (Scania)
Developing Advanced Emergency Braking Systems at Scania – Jonny Andersson
Control systemPerception system

In-vehicle testing

Review Recorded Data
Testing Workflow
Test case:
• Interesting
scenarios
• Expected output
(ground truth)

How would you test a dangerous driving situation ?

When to use synthetic data?
Recorded data does not exist for
• Dangerous scenarios
• Different sensor configurations
Enables testing of dangerous scenarios and corner cases
Scenario Testing with Synthetic Data

How does my vision algorithm respond to changes
to acceleration and steering ?

• Summary
Talk Outline

Algorithm Models
Vehicle and
Environment
Models
• All testing methodologies shown previously are open loop
• Open loop testing: Input does not change with algorithm changes
Open Loop Testing
Forward
Collision
Warning
Autonomous
Emergency
Braking
Dynamic Static

• All testing methodologies shown previously are open loop
• Open loop testing: Input does not change with algorithm
changes
Testing Control Systems – What Else?
• Closed loop testing is required to test control systems
• Closed loop testing: Environment changes with
algorithm.

Algorithm Models
Vehicle and
Environment
Models
Closed Loop Testing
Forward
Collision
Warning
Autonomous
Emergency
Braking
Dynamic Dynamic

Ego Vehicle Dynamics
Target Vehicle Dynamics
Coordinate Transforms
Sensor Models
Closed loop simulation – reduces need for in-vehicle testing

Closed Loop Simulation with Hardware in Loop
Algorithm Models
Vehicle and
Environment
Models
Forward
Collision
Warning
Autonomous
Emergency
Braking
CAN Cable
Why Use HIL ?
• Run algorithm in lab first
• Move hardware in-vehicle for
test-drive

Closed Loop Simulation with Hardware in Loop

Challenge #1: In-vehicle testing is expensive and
time consuming
• Closed-loop simulation for virtual test-drives
Challenge #2: Too dangerous for testing corner-
cases
• Simulate dangerous driving scenarios
Challenge #3: Difficult to validate “black box”
perception algorithms
• Ground truth labeling and re-simulation
How Engineers Solve Common Challenges
Test and Validate an Automated Driving System

• Simulation and offline testing augment (nor replace) in-vehicle tests
• Two key methodologies used for testing:
• Ground truth labeling and re-simulation – testing perception
• Closed loop simulation with a gaming engine – system level testing
Key Takeaways

Automated Driving System Toolbox
Design, simulate, and test ADAS and autonomous driving systems
Resources
More Information (Links)
• Traffic Sign Recognition for Driver Assistance Systems, Dr. Alexander Behrens,
Continental AG, MAC Stuttgart , 2015
• Developing Advanced Emergency Braking Systems at Scania - Jonny Andersson
• Jayaraman, A., Micks, A., and Gross, E., "Creating 3D Virtual Driving Environments
for Simulation-Aided Development of Autonomous Driving and Active Safety," SAE
Technical Paper 2017-01-0107, 2017, doi:10.4271/2017-01-0107.

"How to Test and Validate an Automated Driving System," a Presentation from MathWorks

More Related Content

What's hot

Similar to "How to Test and Validate an Automated Driving System," a Presentation from MathWorks

More from Edge AI and Vision Alliance

Recently uploaded

"How to Test and Validate an Automated Driving System," a Presentation from MathWorks