While Deep Neural Networks (DNN) have revolutionized applications that rely on computer vision, their characteristics introduce substantial challenges to automotive safety engineering. The behavior of a DNN is not explicitly expressed by an engineer in source code, instead enormous amounts of annotated data are used to learn a mapping between input and output. Functional safety as defined by ISO 26262 is not sufficient to match the needs for the new generation of data-driven software. Earlier this year, ISO/PAS 21148 Safety of the Intended Functionality (SOTIF) was published by ISO. SOTIF is a Publicly Available Specification (PAS), a response to a pressing need of an automotive safety standard appropriate for machine learning. A PAS is a stepping stone toward a new ISO standard, and SOTIF is intended to complement conventional functional safety as defined in ISO 26262. In this presentation, we introduce the SOTIF process and present our contributions on how to support safety of the intended function. First, we present search-based software testing to efficiently and effectively idenify test scenarios that cause safety violations in simulated environments. Second, we present a safety cage architecture that helps percepiont systems reject input that does not resemble the training data.

1. Trained,
Not Coded
– Still Safe?
Software Technology Exchange Workshop
Lund, Nov 14, 2019
Markus Borg
@mrksbrg
mrksbrg.com
RISE Research Institutes of Sweden
AB

2. With ML, not only bugs are dangerous…
SOTIF: Safety of the Intended Function
CC BY-NC 2.0
Flickr: @andreas_komodromos

3. Who is Markus?
Board member
Senior researcher, Lund
Adjunct lecturer, Lund University

4. 4
Machine
Learning
Automotive Safety
SOTIF

5. Machine Learning
and
Functional Safety

6. ”a large portion of real-world problems have the
property that it is significantly
easier to collect the data
than to explicitly write the
program”
https://medium.com/@karpathy/software-2-0-a64152b37c35
Andrej Karpathy
Director of AI at
Tesla

7. Karpathy’s Software
2.0Software 1.0
• Humans write source code
• Other humans comprehend the source
code
Software 2.0
• Humans curate data and specify goals
• Backprop. and gradient descent
produces millions of weights
• Humans cannot comprehend mapping
from input to output

8. Neural
network
YOLO (You Only Look Once) by Redmon et al. (2016)

9. Well Hello
There
Management
Core processes for
- Requirements
- Architecture / Design
- Verification & Validation
- Traceability
Supporting processes

10. Definition of Functional Safety
”absence of unreasonable risk due to
hazards resulting from malfunctions of
the electrical/electronic system”
10
What if…
Not a bug – functionality delivered according to the training!
No object
detected

11. “Neither Autopilot nor the driver noticed the
(Fred Lambert, Electrek)
(US NTSB)
white side of the tractor trailer
against a brightly lit sky…”
- Tesla Team, June 30, 2016

12. Safe Machine
Learning

13. ISO/PAS 21448 – SOTIF
Safety of the Intended Functionality

14. Automotive Software Safety
…functional insufficiencies
14
…malfunctions of the
electrical/electronic system
Absence of
unreasonable risk due
to…
ISO/PAS
21448
ISO
26262

15. Structure of ISO/PAS 21448
15
1
43
2
SafeUnsafe
Known
Unknown
4
1
2
3

16. Goal of the SOTIF process
16
SafeUnsafe
Known
Unknown

17. How to Minimize the Unsafe Areas?
17
SafeUnsafe
Known
Unknown
Hazard mitigation
Hazard identification
Verification
Validation

18. The SOTIF Process
- and our Contributions

19. Intended function:
Pedestrian Detection

20. YOLO trained, not
coded

21. SafeUnsafe
Known
Unknown
Requirements
specs.

22. When
active?
Dependencies?
Sensors?
… ego car shall detect pedestrians
crossing the road…
… country road… daylight…
… front-facing camera…

23. SafeUnsafe
Known
Unknown
Requirements
specs.
Risk analysis
Consequences
Ok?
Causes
Ok?
N
Ok
Risks
Y = no harm
Update
requirements
N

24. Avoidance
Reduction
Mitigation
SOTIF
Hazard
Mitigation

25. … ego car shall detect pedestrians
crossing the road…
… country road… daylight…
… front-facing camera + radar…

26. Y
Verification
SafeUnsafe
Known
Unknown
Requirements
specs.
Risk analysis
Consequences
Ok?
Causes
Ok?
N
Ok
Risks
Y = no harm
Update
requirements
N
Known
scenarios can be
covered?
Sensors
Algorithms
Actuators

27. 27
Testing pedestrian detection in
simulators
PreScan

29. SafeUnsafe
Known
Unknown
Requirements
specs.
Risk analysis
Consequences
Ok?
Causes
Ok?
N
Ok
Risks
Y = no harm
Change
requirements
N
N
Risk in real-life
Scenarios Ok?
Validation
• Randomized input
• Worst case
scenarios
• Simulation
• Testbeds
• Fleet tests
• …Y
Verification
Known
scenarios can be
covered?

30. #didyouthinkofthat?
Phillip Koopman, Edge Case Research
30

32. Avoidance
Mitigation
SOTIF
Hazard
Mitigation
Reduction
+

33. 33
Safety Cage Architecture
… Known
input?
N
Warn
YOLO
Y
Hand-
over?
Y
Graceful degradationN

35. SafeUnsafe
Known
Unknown
Requirements
specs.
Risk analysis
Consequences
Ok?
Causes
Ok?
N
Ok
Risks
Y = no harm
Change
requirements
N
N
Risk in real-life
Scenarios Ok?
N
Validation
Y
Verification
Known
scenarios can be
covered?
Prepare
Release
Y

36. Wrap-up

37. Automotive
software
hazards
Malfunctions
Functional
insufficiencies
Known
input
Unknown
input
CC BY-SA 3.0
Alpha Stock Images

38. With ML, not only bugs are dangerous…
SOTIF: Safety of the Intended Function
CC BY-NC 2.0
Flickr: @andreas_komodromos
markus.borg@ri.se
@mrksbrg
mrksbrg.com