This document proposes a methodology for assessing autonomous vehicle performance using real-world data, artificial intelligence, and optimization. It uses scenario mining of sensor data to build a database of parameterized scenarios and distributions. Simulations are run to identify critical scenarios, optimize autonomous functions, and evaluate risk. This was applied to adaptive cruise control optimization, identifying edge cases, tuning control parameters, and evaluating safety performance. The approach aims to objectively validate functions using real-world data and optimize performance for autonomous systems' operational design domains.

1. Assessing ADS real-life performance:
Uniting real data, AI and optimization
for scenario-driven design validation
and optimization
Alexandre Mugnai
Business Development Manager

2. ESTECO
The Challenge
ADAS & AD Validation Approach
Conclusions
What have we learned?
Agenda
The Proposition

3. ESTECO

4. © 2021 ESTECO SpA
ESTECO is an independent
software company, highly
specialized in numerical
optimization and simulation
process and data management.

5. © 2021 ESTECO SpA
Our products
The leading software solution for
simulation process automation and
design optimization
The innovative enterprise platform for
Simulation Process and Data Management
(SPDM) and design optimization

6. Find the optimal design
Handle your design parameters and
balance conflicting objectives.
Maximize IT resources
Exploit all computational resources
and engineering solvers.
Deliver results on time
Accelerate the engineering process
and run multiple simulations.

7. Make simulation data accessible
Expand the usage of engineering
simulation across teams.
Reduce time-to-market
Fast deliver the best product by applying
intelligent algorithms to the simulation
process.
Lower costs
Maximize the investment in
engineering solvers and IT resources.

8. © 2021 ESTECO SpA
Our customers and industries
Embraer
Leonardo
Lockheed Martin
Bombardier
FCA
Ford
Honda
PSA Group
Toyota
Volvo Cars
Corporation
Mahindra
TAFE
Volvo Trucks
ABB
Bajaj
BASF
Cummins
FAW
Whirlpool
Sony
Automotive and
Ground Transportation
Aerospace Architecture, Engineering
and Construction
Manufacturing and
Industrial Equipment
Marine Energy
Healthcare Consumer Goods Electronics

9. The challenge

10. © 2021 ESTECO SpA
Autonomous Vehicles: The verification challenge
”Autonomous vehicle shall safely manage every possible situation on the road”
Safety proof by driving Identify all possible situations
Test AV in these situations
1-10 billion km
Accidents may happen
Scenario types
Parameter ranges
Edge cases
Realistic interactions between actors
Performance evaluation
Assuring test coverage
Finding & solving performance gaps

11. © 2021 ESTECO SpA
• Data, AI & Optimization to Aid Scenario-Based Validation
REAL
DATA
AI analysis
and
modelling
Test
Coverage
& Opt.
Evaluation
Test coverage
Find & solve issues
Scenario types
Parameter ranges
Edge cases
Realistic interactions
Understand complex scenarios
Test only relevant conditions
Sample complexity of real world
Data-driven scenario-based
development and validation
process
REAL
DATA
Data Collection
Test vehicles
High cost & time
Complex infrastructure
Driver bias
Customer vehicles
Low accuracy
Sensor bias
Huge investment
Complex infrastructure
Infrastructure sensors
Anywhere
Anytime
Cost-efficient

12. The proposition

13. © 2021 ESTECO SpA
Vehicle data
Sensor data
#
RAW DATA
STORAGE
- vehicles
- sensors, GPS
- annotations
Scenario category
definition
Scenario Mining
Run identification
and classification algorithms
SCENARIO DATABASE
incl. all characteristic
parameter distributions
Specify Operational
Design Domain
Statistical based
sampling
Simulations
Residual Risk
&
Uncertainty
Vehicle
Validation
Behavioural model
(Inv. Reinforc. Learning)
Critical scenarios
Algorithms
Sensors
Actuators
DB Completeness
Edge cases
Severity
Exposure
Optimisation
Propose a methodology for the assessment of ADAS/AD systems
based on scenario mining and scenario based testing (Streetwise – )
The offering
OEM / Tier

14. Validation Approach Applied To
ACC Function

15. © 2021 ESTECO SpA
• Scenario: ACC
• First impression
• Very basic, yet relatively complicated case when considering function parameters and various vehicle conditions (i.e. ego
vehicle & target vehicle)
• Simple enough to confirm method is functioning properly
• Partners
• TNO providing scenario database, measurables (i.e. severity – exposure – risk)
• ESTECO developing process automation and optimization tools (i.e. specialized algorithms)
• Scope: provide customers a methodology to optimize and validate a given function operating in a specific ODD
The Use Case: ACC

16. © 2021 ESTECO SpA
Lead vehicle decelerating scenario
The Use Case: ACC (Cont.)
ACC parameters
• k1: distance gain
• k2: speed gain ve,0
Scenario parameters
• ve,0
• vl,0
Identify the ACC parameters such that:
• The driving style is similar to human drive style
• The risk of a collision is minimized
Optimisation

17. © 2021 ESTECO SpA
The Use Case: ACC (Cont.)
ACC parameter optimisation

18. © 2021 ESTECO SpA
The Use Case: ACC (Cont.)
k1 = 1.0
k2 = 0.4
Pareto Front
ACC parameter optimisation

19. © 2021 ESTECO SpA
The Use Case: ACC (Cont.)
Edge case identification

20. © 2021 ESTECO SpA
The Use Case: ACC (Cont.)
Edge case identification
Pareto Front
Safe and Probable
Scenario
Edge cases
Collisions

21. Conclusion

22. © 2021 ESTECO SpA
• By using cutting-edge AI to understand and model the real-world data, and by bringing these
AI models into simulation and optimization, the approach optimizes and validates the function
and its operational design domain (ODD)
• Gather real-life unbiased scenario data in relevant locations and time
• Produce a catalogue of parametrized scenarios with parameter distributions
• Identify most critical cases for the validation of a function
• Objectively quantify a system function and allow studies on:
• Sensor positions
• Sensor selection
• Algorithm parameter tuning for function optimization
Conclusion

23. What have we learned?

24. © 2021 ESTECO SpA
• The proposed approach has proven to work on an ACC study and can be extended to
various ADS functions that need to be developed, optimized and validated.
• The method can be applied for any domain in which vehicle requires best performance to
be identified in any real-life condition
• Automate process to simplify system assessment and optimization
• Algorithms play a significant role
• Building statistical understanding on the environment in which system needs to operate
• Identifying edge cases which challenge system performance
• Optimizing system performance
• Identify possible challenges ground truth system is facing
• Design a system that is not over spec-ed (i.e. excessive sensors) understanding its
strengths and weaknesses
• Provide safety argumentation, based on right real-life data
Take Away Ideas

25. esteco.com
Read the ESTECO Copyright Policy
Thank you!
Alexandre Mugnai
mugnai@esteco.com
+31 6 27392381