Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Trained, Not Coded - Still Safe?

13 views

Published on

While Deep Neural Networks (DNN) have revolutionized applications that rely on computer vision, their characteristics introduce substantial challenges to automotive safety engineering. The behavior of a DNN is not explicitly expressed by an engineer in source code, instead enormous amounts of annotated data are used to learn a mapping between input and output. Functional safety as defined by ISO 26262 is not sufficient to match the needs for the new generation of data-driven software.

Earlier this year, ISO/PAS 21148 Safety of the Intended Functionality (SOTIF) was published by ISO. SOTIF is a Publicly Available Specification (PAS), a response to a pressing need of an automotive safety standard appropriate for machine learning. A PAS is a stepping stone toward a new ISO standard, and SOTIF is intended to complement conventional functional safety as defined in ISO 26262.

In this presentation, we introduce the SOTIF process and present our contributions on how to support safety of the intended function. First, we present search-based software testing to efficiently and effectively idenify test scenarios that cause safety violations in simulated environments. Second, we present a safety cage architecture that helps percepiont systems reject input that does not resemble the training data.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Trained, Not Coded - Still Safe?

  1. 1. Trained, Not Coded – Still Safe? Software Technology Exchange Workshop Lund, Nov 14, 2019 Markus Borg @mrksbrg mrksbrg.com RISE Research Institutes of Sweden AB
  2. 2. With ML, not only bugs are dangerous… SOTIF: Safety of the Intended Function CC BY-NC 2.0 Flickr: @andreas_komodromos
  3. 3. Who is Markus? Board member Senior researcher, Lund Adjunct lecturer, Lund University
  4. 4. 4 Machine Learning Automotive Safety SOTIF
  5. 5. Machine Learning and Functional Safety
  6. 6. ”a large portion of real-world problems have the property that it is significantly easier to collect the data than to explicitly write the program” https://medium.com/@karpathy/software-2-0-a64152b37c35 Andrej Karpathy Director of AI at Tesla
  7. 7. Karpathy’s Software 2.0Software 1.0 • Humans write source code • Other humans comprehend the source code Software 2.0 • Humans curate data and specify goals • Backprop. and gradient descent produces millions of weights • Humans cannot comprehend mapping from input to output
  8. 8. Neural network YOLO (You Only Look Once) by Redmon et al. (2016)
  9. 9. Well Hello There Management Core processes for - Requirements - Architecture / Design - Verification & Validation - Traceability Supporting processes
  10. 10. Definition of Functional Safety ”absence of unreasonable risk due to hazards resulting from malfunctions of the electrical/electronic system” 10 What if… Not a bug – functionality delivered according to the training! No object detected
  11. 11. “Neither Autopilot nor the driver noticed the (Fred Lambert, Electrek) (US NTSB) white side of the tractor trailer against a brightly lit sky…” - Tesla Team, June 30, 2016
  12. 12. Safe Machine Learning
  13. 13. ISO/PAS 21448 – SOTIF Safety of the Intended Functionality
  14. 14. Automotive Software Safety …functional insufficiencies 14 …malfunctions of the electrical/electronic system Absence of unreasonable risk due to… ISO/PAS 21448 ISO 26262
  15. 15. Structure of ISO/PAS 21448 15 1 43 2 SafeUnsafe Known Unknown 4 1 2 3
  16. 16. Goal of the SOTIF process 16 SafeUnsafe Known Unknown
  17. 17. How to Minimize the Unsafe Areas? 17 SafeUnsafe Known Unknown Hazard mitigation Hazard identification Verification Validation
  18. 18. The SOTIF Process - and our Contributions
  19. 19. Intended function: Pedestrian Detection
  20. 20. YOLO trained, not coded
  21. 21. SafeUnsafe Known Unknown Requirements specs.
  22. 22. When active? Dependencies? Sensors? … ego car shall detect pedestrians crossing the road… … country road… daylight… … front-facing camera…
  23. 23. SafeUnsafe Known Unknown Requirements specs. Risk analysis Consequences Ok? Causes Ok? N Ok Risks Y = no harm Update requirements N
  24. 24. Avoidance Reduction Mitigation SOTIF Hazard Mitigation
  25. 25. … ego car shall detect pedestrians crossing the road… … country road… daylight… … front-facing camera + radar…
  26. 26. Y Verification SafeUnsafe Known Unknown Requirements specs. Risk analysis Consequences Ok? Causes Ok? N Ok Risks Y = no harm Update requirements N Known scenarios can be covered? Sensors Algorithms Actuators
  27. 27. 27 Testing pedestrian detection in simulators PreScan
  28. 28. SafeUnsafe Known Unknown Requirements specs. Risk analysis Consequences Ok? Causes Ok? N Ok Risks Y = no harm Change requirements N N Risk in real-life Scenarios Ok? Validation • Randomized input • Worst case scenarios • Simulation • Testbeds • Fleet tests • …Y Verification Known scenarios can be covered?
  29. 29. #didyouthinkofthat? Phillip Koopman, Edge Case Research 30
  30. 30. Avoidance Mitigation SOTIF Hazard Mitigation Reduction +
  31. 31. 33 Safety Cage Architecture … Known input? N Warn YOLO Y Hand- over? Y Graceful degradationN
  32. 32. SafeUnsafe Known Unknown Requirements specs. Risk analysis Consequences Ok? Causes Ok? N Ok Risks Y = no harm Change requirements N N Risk in real-life Scenarios Ok? N Validation Y Verification Known scenarios can be covered? Prepare Release Y
  33. 33. Wrap-up
  34. 34. Automotive software hazards Malfunctions Functional insufficiencies Known input Unknown input CC BY-SA 3.0 Alpha Stock Images
  35. 35. With ML, not only bugs are dangerous… SOTIF: Safety of the Intended Function CC BY-NC 2.0 Flickr: @andreas_komodromos markus.borg@ri.se @mrksbrg mrksbrg.com

×