This is the presentation slides for WAISE (Workshop on Artificial Intelligence and Safety Engineering) 2021.

1. Experimental Conformance Evaluation on Uber ATG
Safety Case Framework with ANSI/UL 4600
National Institute of Informatics
Kenji Taguchi
Fuyuki Ishikawa
2021/09/07
ERATO Metamathematics for Systems Design Engineerable AI Techniques for Practical Applications of
High-Quality Machine Learning-based Systems
1) 2)
1)
2)

2. Aims of This Research
1. Assurance framework for Autonomous Driving Systems (ADS)
‒ which encompasses assurance processes and their associated assurance artefacts.
2. Generic safety argument structure, which comprises of fundamental influential factors for safety
for ADS
‒ Structural aspects of safety argument
 Macro argument structure
 top-level argument structure, which defines how the fundamental safety principles (safety
concept, safety policy, safety culture, etc) are dealt with to assure safety
 Micro argument structure
 each particular issue, e.g., safety assurance for ML components
 Bridge structure
 Interfaces safety case and assurance process to collect assurance artefacts for evidence
‒ Safety/cybersecurity aspects
 Functional Safety (FuSa)
 Intended Functionality (SOTIF)
 Cybersecurity
 Their integrations
3. Assessment method for a safety case in compliance with standards and guidelines
‒ ANSI/UL 4600, ISO 26262, ISO/PAS 21448, ISO/SAE 21434, …
Assessment results on an existing safety case for ADS with ANSI/UL 4600
This presentation

3. Safety Case Framework (SCF) by Uber ATG/Aurora
• Remark: Some of the following information is now obsolete, since the latest version of the SCF became
publicly available in Aug/2021.
• Publicly available safety case for automated driving systems by Uber ATG (Advanced Technology Group)
‒ 1st version （Jul/2019）, 2nd version（Jul/2020）, 3rd version (Aug/2021)
• Remarks: Those dates and versions are not officially provided by Uber ATG or Aurora. We just used those
for explanation purposes.
• This is the only publicly available safety case for automated driving systems
‒ Unfortunately, the version (Jul/2020) we used for this work is no longer available on the net
• The latest version by Uber ATG and Aurora are available under the following urls
 https://uberatgresources.com/safetycase/gsn
 https://safetycaseframework.aurora.tech/gsn
• Copyright notice (Remark: Old version)
‒ Creative Commons 0
• Modelled in Goal Structuring Notation (GSN)
‒ Flat argument structure without modules and patterns
• Safety Case Framework is modelled without using any particular tool and provided as a web page with I/F to
expand/shrink GSN diagram and search by keywords
• No technical articles/documents are available and only blogs are posted on medium
‒ https://medium.com/@UberATG/uber-atg-issues-enhancements-to-safety-case-framework-1b5a02c62159
‒ https://medium.com/@UberATG/trailblazing-a-safe-path-forward-e02f5f9ef0cc
• The SCF is not developed in compliance with ANSI/UL 4600.

4. Safety Case Framework
【The top-level structure of the Safety Case Framework 】
We re-modelled the SCF by
astah System Safety tool for reuse.
We used modules instead of
a flat argument structure in the SCF.

5. Macro Argument Architecture of the SCF
＋G1: Proficient: In the absence of system faults, how do we
demonstrate that our system is acceptably safe during nominal
operations?
＋G2: Fail-Safe: How do we ensure that the system is acceptably
safe in the presence of faults and failures? How will the system
mitigate harm in the event of a fault or failure?
＋G3：Continuously improving: How do our developmental and
operational processes identify, evaluate, and resolve anomalies that
can potentially affect the safety of the self-driving vehicle? How can
we actively cultivate a strong culture of safety, where the
organization is engaged and empowered and holds all employees at
all levels accountable for their active participation?
＋G4: Resilient: How do we ensure the self-driving vehicle is
acceptably safe in case of reasonably foreseeable misuse or
unavoidable events?
＋G5: Trustworthy: How do we earn and keep the trust of our riders,
regulators, legislators, public safety officials, other road users, and
advocacy organizations and provide them evidence of the safety
measures of our self-driving enterprise?
• This top-level argument structure faithfully
follows that of the safety report by Uber ATG (A
principled approach to safety, 2020). The
report is submitted to NHTSA as a Voluntary
Safety Self-Assessment （VSSA) .
• The benefit of this approach is that the SCF
preserves the consistency with the safety
report.

6. ANSI/UL 4600
• ANSI/UL 4600 Standard for Safety for
Evaluation of Autonomous Products, April 1,
2020 (1st edition).
• New safety standard which specifies
assurance requirements for the safety of
autonomous products.
‒ This standard aims to cover a broad
categories of autonomous products including
autonomous driving systems.
‒ Annex for mapping to ISO 26262 and
ISO/PAS 21448.
• The safety case is adopted as the main
safety assurance framework and provides
the rigorous guidelines for the use of safety
case.
• The standard covers lifecycle, V&V, risk
assessment metrics, etc.
1 Preface
2 Scope
3 Reference Publications
4 Terms, Definitions, and Document Usage
5 Safety Case and Arguments
6 Risk Assessment
7 Interaction with Humans and Road Users
8 Autonomy Functions and Support
9 Software and System Engineering Processes
10 Dependability
11 Data and Networking
12 Verification, Validation, and Test
13 Tool Qualification, COTS, and Legacy
Components
14 Lifecycle Concerns
15 Maintenance
16 Metrics and Safety Performance Indicators (SPIs)
17 Assessment
Annex A (Informative) – Use with ISO 26262 and
ISO/PAS 21448
A.1 Compatibility
A.2 Safety Case
A.3 Clause Mapping to ISO 26262:2018
A.4 Clause Mapping to ISO/PAS 21448:2019

7. Format of Assurance Requirement in ANSI/UL 4600
• 5 Safety Case and Arguments
• 5.1 General
• 5.1.1 The safety case shall be a structured explanation in the form of claims, supported by argument
and evidence, that justifies that the item is acceptably safe for a defined operational design domain,
and covers the item’s lifecycle.
• 5.1.1.1 MANDATORY:
• （Abbreviated） Addressed by the safety case. Safety case deviations not permitted.
• 5.1.1.2 REQUIRED:
• （Abbreviated） Addressed by the safety case. Safety case deviation is permitted only if documented
by argument that the prompt element is intrinsically incompatible with the item and/or its safety
case.
• 5.1.1.3 HIGHLY RECOMMENDED – N/A
• （Abbreviated）These are best practices that should be followed, but may be omitted, especially for
low risk items.
• 5.1.1.4 RECOMMENDED – N/A
• (Abbreviated）These are optional prompt elements documenting good practices and/or suggestions
for helpful techniques.
• 5.1.1.5 CONFORMANCE:
• （Abbreviated）Conformance with each clause is evaluated via both self-audit and independent
assessment according to Section 17.

8. Assessment Methods
• Assessment for the conformance of the SCF for ANSI/UL 4600 is carried out for each assurance requirement in ANSI/UL
4600 and relevant goals and solutions in the SCF.
• Limitations
• Verdict on conformance is entirely subjective based on our understanding of ANSI/UL 4600 and through search on
relevant goals and solutions in the SCF.
• There are mismatches of terminology between the SCF and ANSI/UL 4600, which makes it hard to assess
conformance of the SCF with ANSI/UL 4600.
• In many cases, it is not a simple one to one correspondence between an assurance requirement and a goal/solution
in the SCF.
• Common understanding of ANSI/UL 4600 has not been established yet due to its release date is very recent.
• The SCF only provides an argument structure without any evidence. This sometimes makes it hard to judge
conformance with assurance requirements in ANSI/UL 4600.
• The SCF lacks strategies for goal decomposition and context in many places, which makes it hard to understand
why those safety argument is chosen.
[Examples of assessment results]

9. Gap Analysis：Conformance Analysis
• The conformance ratio varies significantly
from one chapter to another.
• The highest score is 8:1, which is marked
by Chapter 9 of Software and System
Engineering Process. This analysis result
is reflected by very detailed process-
related arguments in the SCF.
• The worst score is 0:6 by Chapter 15 of
Maintenance. This is mainly because
many arguments only partially conform
with assurance requirements. For instance,
an assurance requirement (15.1.1) in
4600 for mitigation of hazard and risks
related to maintenance and inspection is
only argued by maintenance and no
argument for inspection in the SCF.
C for confirmed, U for un-confirmed

10. Gap Analysis: Structural Analysis
• We map the previous analysis results onto the top-
level structure of the SCF.
• This table can be interpreted horizontally and
vertically, were a horizontal interpretation shows
which chapter in 4600 is dealt with in which sub-goal
of the SCF, and a vertical interpretation shows which
sub-argument structure contains how many different
assurance requirements in chapters in 4600.
• Some observations from the horizontal analyses:
• Safety case is covered almost all sub-argument
structures, since the fundamental mechanism for
safety assurance is based on safety case.
• Risk Assessment is covered by G2: Fail-Safe and
G3: Continuously Improvement, which is plausible
due to their intentions.
• Autonomy Functions and Support are mainly
covered in G1: Proficient, which is plausible for its
intention.
• Software and System Engineering Process is
mostly covered by G1: Proficient. We cannot judge
its plausibility but there is no reason against it.
• Dependability are either safety-related (G2) or
Resilient (G4), which results from the very nature of
dependability.
• Metrics and SPIs are entirely covered by G3:
Continuously Improving, which is plausible. Since
SPIs are used for risk assessment in the operation
phase.
• Assessment is only dealt with in Trustworthy (G5)
due to its role in assessment.

11. NHTSA Safety Elements/ATG Safety Principles Crosswalk
Uber ATG Safety Report (A principled approach to safety, 2020)
Our horizontal analysis is quite similar to safety principles crosstalk in Uber ATG’s
Safety report.

12. Observations
• Safety principles which will affect the entire argument structure may vary from
one company to another so that there would not be the generic and one-fit-all
top-level argument structure.
• Micro-argument adopted in the SFC is very traditional. There are several
proposals for micro-argument structures for machine learning and SOTIF so
that these results could be used as micro-argument patterns.
• Safety case approaches in Pegasus and SaFAD (Safety First for Automated
Driving) could be useful to connect artefact production pipeline (simulation
results, etc) with the argument structure.
• Interrelations between functional safety, SOTIF, cybersecurity are not clearly
presented in the SCF. Security informed safety type of approach or argument
patterns for safety and security could be used as micro-argument patterns.

13. Conclusions/Future directions
• Gap analysis reveals which part of the SCF does not comply with ANSI/UL 4600.
‒ Assessment methods and criteria for a safety case in compliance with a certain
standard should be established.
‒ Generally it is hard to access real safety cases, but the SCF provided us the
golden opportunity to investigate the real safety case thanks to Uber
AGT/Aurora.
• Our structural analyses on distribution of fragments of argument structure found in
each chapter in ANSI/UL4600 reveals some hidden characteristic of the SCF.
‒ We will further investigate the effectiveness of this type of analysis method.
• Third party evaluation on our analysis results:
‒ Evaluation by industry experts
 Four experts (two from OMEs, one from supplier and one from tool vendor)
gave us some feedback on our analysis results. The feedback was positive in
our results and approaches.

14. Thank you for your attention!
We are happy to hear your critical/constructive
comments and answer your questions at the last part
of this session!

15. Some remarks on the SCF
1. Safety case with and without Mission Specialist
2. SDV is a modified commercial vehicle which complies with safety regulations.
3. SDV is developed for rideshare and operated under specific ODD (Operational
Design Domain) for this purpose.
• Mission Specialist is not an ordinary driver. Some evaluation is required if it is used
for SDV with an ordinary driver.
• SDV is a modified commercial vehicle. It is not clear from the SCF how modification
would affect safety already assured in the original commercial vehicle.
• SDV is developed for ride share and ODDs used for SDV are different from those for
ordinary commercial vehicles.
‒ Remark: Any details of ODDs are not provided so that it would not affect the validity of an argument
structure.
（Extracted from contexts specified in the SCF ）
（Cautions if it is applied to commercial vehicles）