An approach towards sotif with ansys medini analyze

1 © 2018 ANSYS, Inc. September 27, 2018 ANSYS Confidential
An Approach towards SOTIF with ANSYS medini analyze
Bernhard Kaiser
Sep 2018

Before we start: Who are we, and what is medini analyze?
▪ Integrated solution for functional safety
and reliability engineering
▪ Large variety of analysis techniques
(e.g. FMEA, Fault Tree Analysis, HAZOP)
acc. to all relevant industry standards
▪ Full support for safety concept creation
and safety management (e.g.
requirements, architecture, tasks)
▪ Single source of truth: SysML model
▪ Safety and reliability analysis at
vehicle/plant, system, software, circuit
board and silicon level
▪ Reduce up to 57% of effort and time-
to-market for safety and reliability
assurance
▪ 150+ customers worldwide from
different industries
Our main Product: medini analyze
▪ Founded in Berlin - member of
ANSYS since Nov. 2016
▪ Leading supplier of software
products for safety, reliability
and quality engineering
▪ Team of experienced safety and
modelling experts, application
engineers and skilled developers
▪ Technology partnership with
e.g. IBM Rational, PTC, JAMA,
Cadence, Synopsis
The Team: ANSYS medini Technologies

Agenda
New Safety Aspects to be Considered for Automated Driving
An outlook on upcoming PAS 21448
Putting AV Development into Practice
How can medini analyze support safe AV development?
What's next on the ANSYS / medini roadmap?

Agenda

An Unpreceded Level of Complexity to be Mastered by Technical Systems

What’s so new about Safety in the Domain of Automated Vehicles?
Functional Safety is all about prevention of accidents caused by electronic systems.
This means prevention of failures and malfunctions, or detection at runtime and taking the system into a safe state.
Safety analysis tracks malfunctions down either to defective hardware parts, or to design errors (“bugs”, “flaws”).
• Sensors and perception algorithms used in automated vehicles (e.g. machine learning, neural networks) have
inherent limitations of nominal performance, and can detect or decide wrong, even in absence of any failure.
• Automotive systems at higher speeds don’t have an immediately reachable passive safe state (“fail-operational”).
• Accidents of automated vehicles often cannot be tracked down to a single root cause, but are caused by misfit of
environment and system assumptions or by unfavorable chains of events (“systemic” or “emergent” behavior)
BUT

What new safety aspects will have to be considered for automated vehicles?
The part of safety that is not related to failures, but to limitations of the nominal performance has been termed
Safety of the Intented Functionality (SOTIF)
The discipline of SOTIF comprises:
• Verifiably sufficient performance of sensors and object detection algorithms
• Considering sensor limitations like field of view, resolution, calibration, sensitivity, disturbances by environment
• Considering performance limitations of machine learning (e.g. “holes” in training data, false negative/positive rates, accuracy)
• Appropriateness of decision making and control algorithms for the intended purpose
• Validation of assumptions about possible environmental situations (reduce the “unknown unknown”, mastering
even rare and initially unforeseen situations)
• Coping with inappropriate human/machine interaction (e.g. unsuccessful hand-over, distraction, intentional misuse)

Facets of Safety
Safety in a broader sense acc. Product Liability Legislation
(= Absence of any kind of hazards)
Functional Safety (cross domain)
ISO 26262 Scope
26262
SOTIF
(Safety of the intended
Functionality)
Security

Agenda

Safety of the Intended Functionality (SOTIF) - Upcoming PAS 21448
ISO 26262 (Functional Safety – FuSa) addresses hazards that arise from malfunctions due to failures of
the E/E systems in vehicle
Upcoming PAS 21448 (SOTIF) addresses hazards that arise from unintended behavior of the failure-free
system, due to performance limitations or disturbances of sensors and algorithms, or failed human
interaction, including misuse.

Scope (in terms of levels of vechicle automation)
Reference: http://safety.trw.com/autonomous-cars-must-progress-through-these-6-levels-of-automation/0104/
present scope Possible future scope

Safety of the intended functionality (SOTIF) - Scope
ISO PAS 21448: Road vehicles -- Safety of the intended functionality
adresses:
• Performance limitations (in particular of sensors and perception algorithms)
• Lack of robustness w.r.t. environmental influences that might disturb sensors
• Insufficient situational awareness
• Reasonable foreseeable misuse and incorrect/insufficient HMI
in the context of ADAS and automation (first edition only automation level 1 and 2, extension possible)
Examples:
• Reflection of a pedestrian on wet street surface triggers a braking maneuver
• Camera images blurred by rain may result in reduced perception of a pedestrian, thus failing to brake
• Driver is not taking over control when required
Status:
FDIS – for voting in second half of 2018, publication expected end of 2018
IMPORTANT:
At present, PAS 21448 is not
targeting AVs at all!
And, of course, AV Safety is
more than just SOTIF!

PAS 21448 Central Objective
1. Reduce amount of known unsafe situations by improving and verifying the system
2. Reduce (= uncover) amount of initially unknown unsafe situations
Legend:
Repeat until
Remaining risk is
acceptable

Upcoming PAS 21448 - Content The proposed proceeding in the current draft comprises
(by numbers of chapters in PAS 21448):
5. Functional and System Specification
(comparable to Item Definition in ISO 26262)
6. Identification and Evaluation of hazards caused
by the intended functionality (comparable to
HARA in ISO 26262)
7. Identification and Evaluation of triggering events
8. Functional modifications to reduce SOTIF related
risks
9. Definition of the Verification and Validation
strategy
10. Verification of the SOTIF (Area 2 in diagram)
11. Validation of the SOTIF (Area 3 in diagram)
12. Methodology and criteria for SOTIF release
Annexes provide examples and guidance for
application of the standard.PAS 21448 Process Flow

Agenda

Putting SOTIF Into a Development Context
PAS 21448 proposes a SOTIF process flow…
… which should be embedded into an overall approach
comprising function engineering, SOTIF and FuSa with
their respective V&V activities

Putting SOTIF Into a Development Context
• For complex system development, there is no use in handling aspects in isolation
• Nominal Function development, SOTIF and FuSa must go hand-in-hand
• The V-Model is just for orientation, in practice the development is iterative, considering more and
more safety aspects in later stages
Develop and
Validate
Nominal
Function
Analyse,
Improve,
Validate SOTIF
Analyze,
Improve,
Validate FuSa

Structured Architecture Design and Requirements Refinement
• There is no use in fighting against failures and weaknesses until the nominal application runs smoothly
• Sometimes, projects fail due to avoidable problems, such as imprecise requirements, chaotic architecture, state
machines that miss out possible triggers or block each other etc. (mainly due to increased complexity of the function)
– This can be avoided or reduced by a structured and formal proceeding
• Sometimes, AV functions fail in standard situations, like left-turn or merging-in onto a crowded highway
– Don’t look for the black swans until you get along with the white ones!
• Defining the architecture for the nominal function is the place where many important decisions are made regarding
SOTIF and FuSa, e.g.
– Selecting suitable perception and control architectures
– Defining monitoring and fallback architectures
– Deciding where to put the ASIL on
• “Normal” validation should be passed before looking for edge case scenarios

Sensor (e.g.
Radar)
Pre-
Processing
Detektion/
Tracking
Sensor Data
Fusion
D
A
...
... Feature
Extractor 3
Track Planner
EPS
ESC
Engine
Control
Feature
Extractor 2
Feature
Extractor 1
Arbiter
Sensor (e.g.
Camera)
Pre-
Processing
Detektion/
Tracking
D
A
Decider 3
Decider 2
Decider 1
Structured Architecture Design and Requirements Refinement
The SCADE Toolchain from
ANSYS can support formal
specification of algorithms
and architectures

Validation of Requirements for all Known Cases
HIL Testing
Test Drive
Open Loop / Closed Loop Simulation

HARA issues when making the transition to SOTIF
• Instead of only individual failures of individual actuators, all facets of the combined behavior of the
whole vehicle may be of relevance
• E.g. radar has a blind spot for a certain object, and camera cannot see it because blinded by the sun
• E.g. intended trajectory is left because limited performance of steering actuator w.r.t. speed adjusted by engine control
• Focus shift from actuator side (trad. HARA) to sensor side (SOTIF HARA)
• Already traditional ISO26262 HARA suffers from combinatorial explosion…
• E.g. Road Type x Speed Range x Wheather Conditions x Maneuver x Other Traffic Participants x …
• … but considering sensor weaknesses involves even more details and more specific scenarios
• E.g. driving under a metal bridge, sunset straight ahead in combination with wet road
• … and instead of static scenarios, temporal sequences of events will become relevant
• E.g. other vehicle joins in from neighbor lane and then, within 2 seconds, bruskely brakes (cf. Open Scenario catalog)
Means of reducing and/or automated analysis is essential to be able to claim that „everything“ has been
properly considered while keeping the effort manageable
→ Combination with simulation, evolutionary testing etc.?
→ Feedback from HARA runs and validation into future scenario catalogs (and runtime decision making policies)?

How to find unknown problematic cases
• Systematic recording and evaluation of test ride data from many miles and road types
• Incident and accident logs with in-depth analysis (as in aviation)
• Simulations and lab tests with sensors
• Systematic reasoning based on sensor and algorithm working principles
• Variation analysis applied to situation parameters as well to sensor / algorithm parameters
• Evolutionary Testing/Simulation, AI-based Testing/Simulation
• Structured brainstorming by domain experts
Sensor insufficiencies and mismatches of sensor capabilities with particular situations are no failures,
but can have similar consequences. The same applies to disturbances!

Mismatch and Disturbance Analysis
Sensor Triggering Event: Target Object and Pose Environmental Conditions Senor Capability
Camera
Pedestrian, 10..50 m away, partly hidden
behind signpost Clear sight
reduced
detectability
Pedestrian, 10..50 m away, crossing road darkness (at night)
almost no
detectability
Tire parts on road
clear sight, at high speed (on
highway)
almost no
detectability
Radar
Pedestrian, 10..50 m away, crossing road heavy rain
almost no
detectability
Tire parts on road
clear sight, at high speed (on
highway)
almost no
detectability
Representation Example - other possible representations:
• Matrix style (Sensor A x Sensor B)
• Matrix style (Sensor x Scenario)
• FMEA (cause-effect relations) and/or HAZOP (deviation keywords) table style
FuSa is about failures. SOTIF is rather about mismatch of a sensor or algorithm with a certain (perhaps unforseen)
relevant situation aspect. A sensor can do fine in one situation or application, but cause a hazard in another!

Probabilistic Performance Measures for Sensors
Sensors and Object Detection/Classification/Tracking algorithms cannot always be exact (even without any failure)
• False Positive Rate = Objects are reported that do not exist
• False Negative Rate = Existing objects are not reported
• Probability of wrong classification (i.e. pedestrian is reported when there is actually a motorcycle)
• Imprecision of measured distance, speed etc.
• … and due to non-linear decision making, small errors can lead to significant consequences (to brake or not to brake)
Perception Interpretation
Measured
Signal
True
Value
e.g. true presence of object,
true class of object, true
distance to object
e.g. radar scatters with
their time of flight
Inner
Concept
of Situation
e.g. object is a pedestrian,
80m ahead and in my
driving corridor
Trajectory /
Action
Planning
Tgt Trajectory
or Action Cmd
e.g. trigger emergency
braking
Trajectory /
Action
Execution
From ROC (Receiver Operator Characteristics) Curve

Probabilistic Analysis for Safety Goal Violation
Requirement:
A vehicle shall be considered as
critical cross-traffic if it drives at
30 km/h or more and crosses in
an angle above 45° and below
135°
„Failure“ Probability? „Failure“ = Requirement Violation

Probabilistic Analysis for Safety Goal Violation
Safety Requirement: A vehicle shall be considered critical for collision, if it approaches with 30 km/h or more.

Hazardous Event: A vehicle driving at 30km/h or more is classified as below 30km/h.
True value in real world
e.g. speed of other car = 32 km/h
Reported value
e.g. radar sensor says 29 km/h
if(obj.speed >= 30.0)
{
setAlert();
}Hazard
For a sensor with = 1km/h:
• For a vehicle running at 32 km/h, the probability
of not being classified correctly is 2.28%
• Things would get better if SW threshold was
lowered e.g. to 26 km/h, if acceptable
(For the entire population of vehicles with any speed, an
integral will have to be solved, taking probability distribution
of all vehicles into accoint…)
Threshold 30 km/h
Implementation
ERROR
Simulation will
be key in
estimating
hazard
probabilities!
ANSYS builds up
simulation
solutions for all
kinds of AV
sensors!

FuSa in the Context of Automated Vehcles
ASIL D
ASIL D
ASIL D
QM
… but basically, FuSa is as in traditional automotive systems and covered by ISO 26262.
With medini analyze, you have got an industry-proven solution for all aspects of FuSa!
Main challenge:
Putting an ASIL on
perception algorithms
/ finding an ASIL-
capable solution for
monitors!

Agenda

Managing the SOTIF Process in medini analyze – Guided by Checklists
Link to related artifact

Defining Automated Driving Functions and Candidate Errors in medini
Describing the intended functions and external
interfaces in the Item Definition is key for all further
SOTIF and FuSa steps
→ Describe and model architecture, interfaces and
functions like in existing ISO 2626 approach
Errors already known from gained experience can be
put in error collections as candidates for HARA and
Triggering Event Analysis

Item Definition and Architecture Modeling for SOTIF and FuSa
Medini analyze is based on SysML models and offers graphical editors.
Nominal Function Development can be carried out in ANSYS SCADE
and reused in medini for SOTIF and FuSa
Other import filters for Rhapsody, Enterprise Architect and Simumlink!

SOTIF HARA acc. PAS 21448: Draft implementation in medini analyze

Triggering Event Analysis acc. PAS 21448: Draft implementation in medini analyze
Example: Driver Interaction Checklist (implemented acc. PAS 21448 Table E.1)

Deriving SOTIF Measures for SOTIF Concept Using SOTIF FMEA in medini analyze

Safety Concept Incorporating SOTIF and FuSa in medini analyze
SOTIF Requirements FuSa Requirements
Of course, development in separate models is also possible!
Safety Goal
(from common FuSa/SOTIF HARA)

Creating Tasks for Verification/Validation Duties Resulting from SOTIF Analyses

All Our SOTIF Analyses Have not Yet Been Finally Fixed!
In these days, we are all just learners!
The SOTIF standard is not even out, and a true AV
Safety standard far away –
but AV development is happening now!
So, let‘s collaborate!
Now we have got something to start -
Let‘s refine these analyses together!

Agenda

Further Suggestions for SOTIF Analyses (in discussion)
• SOTIF FMEA (Causes need not be failures, but can be triggering events, weaknesses etc., effects can be unintended behavior)
• SOTIF HAZOP (On different levels of detail. Apply appropriate keywords to systematically find deviations)
• Limitation, Mismatch and Disturbance Analysis
• (Which sensor type is bad for which kind of target under which conditions?)
• (Which sensor type can be disturbed by what environmental conditions?)
• (Which underlying assumptions of perception algorithms or prediction models are wrong under which conditions?)
• Qualitative SOTIF FTA (What are potential reasons for a potential unintended behavior?)
• Quantitative SOTIF FTA (Taking into account probabilities for false positive, false negative, wrong classification, inaccuracy…)
• Event Tree Analysis (ETA) (How can a scenario evolve further after a triggering event has occurred)
• STPA (Systemic and process-oriented analysis involving also human interaction)
• (Dynamic) Bayesian Network (How to reason probabilistically from obervations back to the actual real-world situations)
• Edge Case Analysis (Find out at which point within a continuous parameter value space the behavior suddenly pivots.)

Example: Limitation, Mismatch and Disturbance Analysis for Sensors
Inability /
Impairment
Disturbance

Quantitative FTA Based on Statistical Sensor / Traffic Data From Simulation
6,0%
3,0%
3,5%
3,0%
2,5%2,0%2,5%
3,0%
5,0%
6,0%
7,0%
9,0%
13,0%
12,0%
9,0%
5,0%
3,5%
3,0%
2,0%
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180
Distribution of Speed
0,1%0,1%0,4%0,9%1,8%3,3%
5,5%
8,1%
10,6%
12,6%
13,3%
12,6%
10,6%
8,1%
5,5%
3,3%1,8%0,9%0,4%0,1%0,1%
-10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10
Sensor Inaccuracy
Vehicle detected below 30
km/h although faster in
reality
Quantitative Profile from Traffic Statistics Results from Sensor Tests or Simulations
Quantitative Fault Tree Analysis for probability
of a specific requirement violation

Integration of Safety Analysis with ANSYS‘ AV Simulation Environment
Possible Settings:
• Safety Analyses (e.g. Triggering Event Analysis) suggest critical factors, but the effect is unknown
→ Use open-loop simulation with Failure/Deviation Injection and monitor on acceptable output behavior
→ Safety analyses defines simulation duties and pass criteria to ensure safety goals
• Triggering Events / Edge Cases are unknown or cannot be enumerated for Safety Analysis
→ Use Simulation with random / statistical / evolutionary testing to find out when situation reaches a tilting point
→ Feedback into HARA / Triggering Event Analysis (+ Test ride duties catalog)
• Safety Analyses result in function improvement + validation duties, but driving is too much effort
→ Use closed-loop simulation with models for sensors, vehicle dynamics and environment for virtual validation

Integration of Safety Analysis with ANSYS‘ AV Simulation Environment
Edge Case AnalysisEquivalence Class Partitioning Evolutionary Testing Random/Statistical Testing
Scenario Builder
FTA
TEA
HARASafety Concept
Test Cases
Resulting Hazards
Causal Chains
Fault Injection Cases

Conclusion
• Safety of Automated Vehicles poses a bunch of new challenges
• Not just failures, but also limitations of the nominal performance (SOTIF) become safety issues
• For a quick start into SOTIF, medini analyse will offer the current analyses from PAS 21448 as early-
adopter versions as early as 2019
• We are ready to learn along with our customers in pilot projects and adapt our implemnetations
• Advanced analysis techniques are currently under research
• In combination with ANSYS‘ unique modelling and simulation capabilities, medini analyze can adress
novel problem fields that will forever remain unaccessible to pure safety analysis tools
Want to get involved in this discussion?
→ Visit us at Medini User Conference in Berlin 26+27 Sept 2018 and in Troy, Michigan 16+17 Nov 2018
→ Follow us on blog.ansys.com and give us your feedback!
→ If a sufficient number of people expresses interest, we will set up a regular web conference or dedicated blog.
→ See your ANSYS sales representative and ask for an on-site demo.
→ Become a strategic pilot customer, as some well-known AV market players have already done!

medini™ analyze
Realize Your Product Promise

An approach towards sotif with ansys medini analyze

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to An approach towards sotif with ansys medini analyze

Similar to An approach towards sotif with ansys medini analyze (20)

Recently uploaded

Recently uploaded (20)

An approach towards sotif with ansys medini analyze