1.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Edge case testing in a
sensor realistic simulation
environment
Mike Dempsey
Managing Director
Claytex

2.
Copyright Claytex Services Limited 2022
• Multi-domain modelling and simulation
with Dymola, Modelica and FMI
• Virtual testing for ADAS and
Autonomous Vehicles with AVSandbox
• Requirements Traceability with Reqtify
• Software, Consulting, Training
• Part of TECHNIA, PLM & Engineering
Knowledge Leaders

3.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Edge case testing in a sensor
realistic simulation environment
Why do we want to do this?

4.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
The journey to AV safety
• Proving an AV is safe to deploy requires
testing on diverse scenarios
• Edge cases are those scenarios that occur
infrequently but cover all the risk
• Due to the risk involved and number of
edge cases we need to use simulation for
these tests
• The simulation platform must provide a
realistic, immersive, environment for the AV
controller
• New EU regulations place stringent
requirements on the simulation platform
used to generate evidence for the safety
case

5.
Copyright Claytex Services Limited 2022
• UK government funded project, led by ∂RISK, other
partners are Imperial College London and DG Cities
• Funded through Innovate UK and CCAV
• https://www.drisk-project.org/
• 3 year project, concluded September 2020
• Project goal: to develop the ultimate driving test for
autonomous vehicles
• Key output is a Virtual Verification and Validation platform
• Edge case database
• Sensor realistic simulation platform
• Research on safety topics including “What is safety” and
“How do we measure safety”

6.
Copyright Claytex Services Limited 2022
edge case database

7.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Simulation Platform
• Scenario basedAutonomous
Vehicle simulation platform
• Sensor realistic simulation​ with
physics-basedmodels
• Multibody vehicle dynamics
• Real-time, high-fidelity and fully
synchronous modes
• Supports open standards including
OpenSCENARIO, OpenDRIVE and
FMI
• Built around rFpro

8.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Powered by rFpro
• Engineering simulation tool, not a
gaming engine
• rFpro focuses on physically modelling
the real world
• Physics based rendering
• High fidelity road surface
• Millimetre accurate digital twins of real
world locations
• Fully synchronous, deterministic
simulation when distributed across
multiple processes and multiple
machine
• Real-time simulation mode to support
HiL and DiL (Passenger-in-the-Loop)

9.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
System under test
• Autoware based controller
• Setup with assistance from
WMG, University of Warwick
• No changes required to use
with the simulator
• Sensor setup
• Velodyne Ultra Puck
• Single forward-facing camera
with RCCC filter
• GPS
• ODD limited to handling basic
junctions, no traffic lights, with
traffic and pedestrians in
good weather conditions

10.
Copyright Claytex Services Limited 2022
System capabilities - ODD

11.
Copyright Claytex Services Limited 2022
11

12.
Copyright Claytex Services Limited 2022
Camera Simulation
• Lens distortion models including
360-degree fisheye camera
• Colour filter
• Dynamic exposure control
• Detailed shutter effects
• Apply masks to simulate dirt on
the lens
• Publish direct to ROS camera
topic
• HiL solutions available for real-
time camera simulation

13.
Copyright Claytex Services Limited 2022
LiDAR model details
• Include rotational dynamics of spinning sensors
• Range measurement
includes noise that
effects precision
• Also has a deterministic
mode
• Weather model effects
measurement precision,
intensity and includes
random drop-outs
• Output the same
message stream as the
real device

14.
Copyright Claytex Services Limited 2022
ROS Interface
• ROS bridge developed for
Autoware and Project Aslan
• Subscribes and publishes
ROS topics directly
• Sends state information
• Reads driving demands
• Publishes /clock to ensure
synchronization between
simulator and ROS
• ROS-camera sensor
published camera topics
direct to ROS

15.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Testing our AV
Using a sensor realistic simulation
platform allows us to integrate the AV
controller without modification

16.
Copyright Claytex Services Limited 2022
Test Automation
• Parameter sweeps can be
configured to cover nominal and
critical scenarios
• Interface to scenario database
which uses adaptive testing to
cover ODD
• OpenSCENARIO format used to
exchange scenario and results
• Integrate with CI/CD through
command line driven testing and
test-series input files

17.
Copyright Claytex Services Limited 2022
KPI calculation
• Open API to allow you to
define your own metrics
and dashboards
• For example:
• RSS
• Safe space
• Accurate parking
• TTC, TTLC, etc.
• Standard reports linked
to the scenario

18.
Copyright Claytex Services Limited 2022
T-junction edge case

19.
Copyright Claytex Services Limited 2022
Crossroad failures

20.
Copyright Claytex Services Limited 2022
Pedestrian in the road

21.
Copyright Claytex Services Limited 2022
How does this get us to a safe AV?
• Sensor realistic simulation
platform means no changes
needed to the controller to run in
simulation
• High fidelity models include noise
to explore robustness
• Open and extendible with support
for open standards
• Interfaces to scenario databases
provide the test gauntlet
• We continue to develop and
increase the level of detail
available and the validation of
sensors against real-world data

22.
Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Thanks
Questions?
Contact: mike.dempsey@claytex.com