SiriusCon 2018 - Talk by Ewen W. Denney, NASA - Robust Software Engineering Group We describe AdvoCATE, the assurance case automation toolset, under development at NASA Ames Research Center. An assurance case provides a comprehensive basis for the stakeholders of a system to have justified confidence in its dependability attributes—safety, security, reliability, etc. A safety case is a specialized assurance case that focuses on providing a formal justification of acceptable safety. In many safety-critical industries, such as aviation, the production and evaluation of a safety case can be a prerequisite for regulatory approval to field and operate a system. A safety case developed in AdvoCATE comprises multiple interrelated artifacts, including a hazard log, a requirements repository, a safety architecture, and several structured arguments, all of which must be kept mutually consistent. The hazard log and requirements repository record the results of safety analysis activities; the safety architecture graphically describes the relevant safety-related scenarios along with the collection of safety risk reduction mechanisms, whilst providing the foundation for risk assessment and visualization. Structured arguments link various safety related claims to heterogeneous sources of evidence gathered during systems engineering, also capturing rationale pertinent to safety and other auxiliary concerns. AdvoCATE is an Eclipse Rich Client Platform (RCP) application that leverages a variety of models specified using Eclipse Modeling Framework (EMF), Sirius for creating graphical representations, NatTable for tabular environments, and Xtext for domain-specific languages. Thus, AdvoCATE facilitates model-based development, exploiting validations, transformations, and a range of views in the construction of (some of) the artifacts constituting a safety case. We have used AdvoCATE to develop safety cases and successfully obtain regulatory approval for NASA flight operations involving a fleet of small unmanned aircraft systems (UAS) flying beyond visual line of sight.

1. AdvoCATE: An Assurance Case
Automation Toolset
Based on Eclipse and Sirius
4th December, 2018
Ewen Denney & Robbie Henderson
(Joint work with Ganesh Pai and Dimo Petroff)
NASA Ames Research Center
Robust Software Engineering Group
ewen.denney@nasa.gov

2. Research Motivation
• High-hazard industries are moving to active safety
management
– Safety management system (SMS) in aviation
– Need to
• Unify reasoning about technical aspects of safety
• Support safety-related decision making
• Goals-based regulation is attractive for novel applications
– When regulations and performance standards are absent
• Unmanned aircraft systems (UAS), Autonomous systems, …
– Increases flexibility for regulated entity
– Evidence-based assurance  safety case / assurance case
2

3. Safety and Assurance Cases
‘A safety case is a structured argument, supported
by a body of evidence, that provides a compelling,
comprehensible and valid case that a system is
safe for a given application in a given operating
environment’
- NASA System Safety Handbook ver. 1 (2014)
• Essentially, a safety risk management artifact
– Other compatible definitions and guidance on content
– Based on application domain, standard, regulatory
paradigm, etc.
• An assurance case generalizes safety cases to
other assurance properties: reliability, security,
availability, … 3

4. Risk Control
Risk Analysis
and Assessment
Hazard Effect Severity Likelihood Initial Risk
Level
Hazard
Control
Residual
Risk Level
H1 - Airspace
encounter with
GA aircraft
NMAC
/ MAC
2 (Haz.)
1 (Cat.)
Probable
Probable
2B
1B
Detect & Avoid
Flt.
Termination
...
2D
2D
H2 – Stall CFIT
Safety Risk Management
4
System Analysis
Concept of Operations,
System/change description,
Regulations, …
HazID
Hazards
Operational, functional, …
Design target
Barrier Modeling – Abstract Safety Architecture
Safety
Requirements
Implementation
Mitigations
Safety requirements
Barrier and Control functions
Risk scenarios, design targets,
risk evaluation
Assurance Rationale
(Structured Argument)
Evidence Artifacts
Design, Analysis, Verification
Testing,
Assurance claims,
strategies, context,
rationale, …
Operational Safety
Assurance
(Monitoring and Update)
Safety performance
measures, monitors, …
Operational Evidence
Verification of safety performance targets
Assumption corroboration
Hazard tracking, Precursors, …

5. Example: UAS Safety
5
Combination of operating modes
• Visual line of sight (VLOS)
• Beyond visual line of sight (BVLOS)
• Beyond radio line of sight (BRLOS)
Varying access profiles
• Operating range
• Terminal airspace
• Transit (vertical / lateral)
Diverse environment
• Populated / urban / built-up areas
• Uncontrolled / controlled airspace
• Low / high density airspace
Varying mission concepts
• Package delivery
• Surveillance
• Aerial inspection
• Mapping, …
Different configurations
• Airborne sensors (Lidar, sonar,
FPV camera, Radar)
• Ground sensors (Radar)
• Multiple GCS, Roaming GCS, …
Increasing
complexity in
mission and
operations
UAS – Unmanned Aerial System (aka drone)

6. UTM: UAS Traffic Management
6

7. Tool Needs
• Creation and assessment of assurance cases
– Support variety of diagrams and for assurance artifacts
representations (graphical, tabular, textual)
– Views for diverse stakeholders and use cases
– Consistency and navigation between assurance artifacts
– Automation workflows
– Integration with 3rd party tools
• Tool technologies
– EMF: model-based assurance
– Sirius: graphical editing of industry standard safety notations
– Xtext: domain specific languages and querying of safety models
– NatTable: table editor for hazard/requirements analysis
7

8. Barrier Modeling
• Collection of barrier models providing a risk
basis
– Collection of all factors affecting risk
– Model for risk qualification/quantification
8
Event chain / accident trajectory
Barrier compromise/breach
Loss of
Control
State
Threats /
Causes /
Initiating
Events or
States
Accident /
Loss /
Harmful
States or
Events
Prevention Barriers Recovery Barriers
Hazard

9. Bow Tie Diagram (BTD)
9

10. Example: Loss of Separation
10

11. Rationale Capture via Assurance Arguments
11
Chain of
reasoning
Safety / Dependability Claims
Item of Evidence
Developed
claims
Documentation and Details
Goal Structuring
Notation (GSN)

12. Example: Battery Failure
12

13. AdvoCATE: Tables
• Assurance Case Automation Toolset
• Hazard analysis and risk assessment
– Conducting hazard identification
– Specification of hazard causes and
consequences
– Assessment of initial and residual risk levels
given in terms of probability and severity
• Safety and assurance requirements capture
13

14. Hazard Log and Tabular Editor
14

15. Safety Requirements Capture
15

16. AdvoCATE: Arguments and Patterns
• Structured argument development
– Pattern specification
– Automated pattern instantiation
• Integration of formal methods and formal tool-based
evidence
– Hierarchical and Modular organization
– Argument queries and views
• Metrics
16

17. Argument Editor
17

18. AdvoCATE: Safety Architectures
• Safety architecture development
– Composition of multiple bow tie diagrams
– Views
– Transformations (event and barrier split / merge)
• Sequential event split: Loss of safe separation  Loss
of “well-clear” separation + NMAC
• Parallel event split: MAC  MAC within OR || MAC
outside OR
• Barrier split: Ground-based surveillance  Radar
surveillance + Visual surveillance
– Risk computation: event probability along paths
18

19. Safety Architectures
19
Bow Tie Modeling
Automated View
Extraction

20. AdvoCATE: Traceability
• Navigation
• Traceability matrices
• Maintaining consistency between related
artifacts, e.g., between
– Entries in the hazard log and the relevant
assurance requirements
– Arguments and the corresponding requirements,
verification artifacts, etc.
20

21. Tracing and Consistency
21
Hazards
Safety and Assurance Requirements
Assurance Arguments / Rationale Bow Tie Diagrams / Safety Architecture

22. Amalgam Activity Explorer
• The Amalgam
activity explorer
is used in the
design of our
Safety, Mission
Assurance, and
Risk
management
(SMART)
dashboard
22

23. Amalgam Activity Explorer
• The (SMART) Dashboard allows us to:
– Provide a clear and directional workflow towards
a completed safety/assurance case
23

24. Amalgam Activity Explorer
24
• For each step we have one
EMF model
• Dependencies provide some
of the workflow, i.e. safety
architectures can require
“requirements” model
components
• Necessary components are
clearly prompted
• Sirius diagrams relevant to the
current model are accessible

25. Amalgam Activity Explorer
• The (SMART) Dashboard allows us to:
– Provide a clear and directional workflow towards
a completed safety/assurance case
– Provide feedback on the status of assurance
activities, and areas that need to be developed
further
– Provide a naive evaluation of the current system
safety
25

26. Amalgam Activity Explorer
26
• Problems with the safety case development are
clearly brought to the users attention, with
hyperlinking to the problem source

27. Amalgam Activity Explorer
• The (SMART) Dashboard allows us to:
– Provide a clear and directional workflow towards
a completed safety/assurance case
– Provide feedback on the status of assurance
activities, and areas that need to be developed
further
– Provide a naive evaluation of the current system
safety
– In future, provide real-time evaluation of system
based on feedback from a live platform
27

28. Activity Explorer Issues
• We don’t always have a Sirius “session”
– Amalgam works very well when provided a Sirius
“session”
– Some of our models are entirely developed in a DSL,
or NAT Table tabular editor
– Initially we created viewpoints for all
resources….even when it wasn’t useful
– We now manually load resources and open editors
by id, and only use the Sirius session for the
opening/creation of viewpoints
• Debugging is hard!
– Issues with activity explorer pages often result in no
activity explorer at all, with no logging – help!
28

29. BX of Safety Models
• Sirius viewpoints are used extensively, along
side various editors, to avoid complex bi-
directional transformations of the safety models
– The safety architecture of a system can be viewed as
a Controlled Event Structure, a single diagram
showing the temporal flow of all events
– One event in a CES may have a local bow tie, where
we only care about the event, its own causes and
effects
– Through a combination of Xtend model helpers and
multiple viewpoints, we managed to merge most
models containing similar information and just
provide viewpoints where necessary
29

30. BX of Safety Models
30

31. Sirius Custom Properties Panel
• Many of AdvoCATE’s graphical elements are the
product of multiple modelled constructs
– To handle this, we made use of Sirius custom
property panels
– Model elements, such as hazards, are edited from
many locations in AdvoCATE, and are viewed in
different forms all over the tool
– One custom property panel is added, allowing us to
define one uniform editing experience for the
combined feature, regardless of what is shown
– Certain semantic attributes can be shown, but not
edited to allow the user context while in a particular
viewpoint
31

32. Sirius Custom Properties Panel
32
Calculated Values –
Mitigation of Risk
A Hazard/Event
A Hazard in progress –
Event Instance

33. Property Panel Additions
• Some customizations to the custom
properties panel we have implemented:
– Enum Lists: We have many model features as
lists of enumerated values
– Xtext editor widgets (more on that later)
– Xtext index-query selection boxes – model cross
references
33

34. Xtext
34
• All models within AdvoCATE make use of Xtext
resources and the powerful index they provide
• Extensive cross-referencing between models
became cumbersome using pure EMF
• Integration of Xtext and Sirius has been very
smooth – with only minimal customizations to
Sirius widgets and some services to take
advantage of the Xtext index in diagrams
• Most models we use require an Xtext DSL to
keep all users happy…so extra effort is minimal

35. Xtext - Indexing
35
• With all models being Xtext resources we are
able to take advantage of the Xtext index as a
one-stop repository of safety elements
• Cross-referencing by loading resources
becomes quite cumbersome with large projects
• We wrap Xtext index querying in services used
by our Sirius diagrams, to take advantage of our
DSL scope providers
• Future plans will involve the DSL Devkit
Scope/Export framework, to allow us to fine tune
relevant safety artifacts, and export these to an
external repository (large scale safety case
development)

36. Xtext - Indexing
36
• We create an Xtext scope-provider-fed
custom property widget
• As the DSL is modified, the Sirius properties
view is updated automatically – it simply calls
our scope provider
• Relevant EObjects are resolved and the list of
choices is populated

37. Xtext – Serialized Models
37
• One important future feature of AdvoCATE is
collaborative safety case development
– When using pure EMF + Sirius, we found that
version control struggled a little…

38. Xtext – Serialized Models
38
• One way we thought to combat this problem
is a combination of:
– Really good auto-layout (if a little ambitious)
• We don’t necessarily need to version control the layout
if we can do so automatically, and reliably
• AIRD merge conflicts become huge, and impossible to
merge – we might not need to track them
– Serialize the model as a DSL, and parse
• The models themselves in XMI format can be hard to
merge
• New features cause compatibility problems

39. Xtext – Serialized Models
39
• By designing a robust Xtext DSL for each
model, we can more reliably track changes
– Git likes DSLs way more than XMI
– New features, or modified metamodels are less
likely to also break the parser, but XMI almost
always will
– We can auto-create appropriate diagrams for our
models in Sirius, and auto-layout on first opening
• We’re still in the process of finding a solution
to our problems – but this fits nicely so far

40. Xtext – Direct Edit Xtext Editor
40
• In some contexts, complex syntax had to be
embedded in our graphical editors
– Argument patterns, are a way to generate a GSN
argument based on given data and a “pattern”
providing the structure
– Parameters are defined, and then embedded in
node descriptions to be evaluated at generation
time
– To do so, we designed a DSL to define the
pattern and it’s parameters
– Great! We get all the content assist, linking, and
that fun stuff

41. Xtext – Direct Edit Xtext Editor
41
But wait…what’s the structure?

42. Xtext – Direct Edit Xtext Editor
42
• Clearly, a graphical layout gives a much more
manageable view of what the generated
result might be
– We needed a solution that combined the power of
the Xtext DSL, for what might become very
complex string-building expressions, with the
high-level view of a Sirius viewpoint
– We created a Sirius Direct Edit widget which
wrapped the Xtext Embedded editor
– Now we have content assist, syntax highlighting,
hyperlinking, and inline validation – all as part of
direct edit

43. Xtext – Direct Edit Xtext Editor
43

44. Perspectives
• Ongoing focus on design-time assurance
– Artifacts and rationale from development, prior to release-into-service
• Outlook towards operational assurance through lifecycle
– In-service safety performance monitoring
• Autonomy applications
– NASA System-wide Safety Project
– DARPA Assured Autonomy Program
– Expansion in application domain to spaceflight: initially robotic,
eventually, human spaceflight
• Future tool development
– User-customizable dashboards
– Query/view language
– Collaborative development
– Towards the Cloud …
44

45. We’re hiring!
Contact: ewen.denney@nasa.gov
45
Looking for software engineers with experience
in Eclipse, Sirius, Xtext, NatTable, ...

46. Please wait a few seconds before we
automatically bring you to the next session
(First Day Closing Session)
If you want to keep talking with the speakers of actual talk,
you will have to come back to this session.
Thanks for listening to (Ewen Denney|NASA Ames)
Any questions?