SlideShare a Scribd company logo
1 of 21
This project has received funding from the European Union's Horizon
Europe programme under grant agreement number 101069595.
‡
AI-FSM: Towards Functional Safety Management
for Artificial Intelligence-based Critical Systems
Presenter: Javier Fernández - IKERLAN
Javier Fernández†, Irune Agirre†, Jon Perez-
Cerrolaza†, Lorea Belategi†, Ana Adell†, Carlo
Donzella⋆, Jaume Abella‡
(†)
(⋆)
(‡)
CONTENTS
01
02
03
CONTENTS
CONTEXTUALIZATION
PROPOSED
LIFECYCLE
CONCLUSION & FUTURE RESEARCH
04 DISCUSSION
3
Functional Safety Management (FSM): encompasses all essential activities throughout the Functional Safety
lifecycle phases, as mandated by IEC 61508-1. FSM is designed to prevent errors during specification, design,
development, manufacturing, and commissioning.
Context
4
AI lifecycle phases
• Five main stages:
• Requirements Specification
• Data Management
• Development dataset
• Training + Validation* dataset
• Verification dataset
• Model training
• Trained model
• Model verification
• Verified model
• Model Deployment
• Inference model
Context
5
The general DL-based systems development process clashes frontally with traditional
safety development processes:
• DL SW is designed monolithically following empirical training processes with example training
data, rather than implementing specific safety requirements.
• DL SW, as opposed to any other kind of SW in safety critical systems, cannot be considered as
correct by design due to the predictive nature that comes along with mispredictions and
confidence values.
• DL SW design is no longer independent of data, and its parameters are set empirically based
on training datasets.
• DL SW imposes high-performance demands on the underlying HW and its inherent
complexity (both HW and SW) entails challenges to comply with safety standards.
Context
6
Context
Main phases affected by including DL • Definition of the ODD and operational
scenarios
• HARA shall identify potential hazards caused by the DL-
based system. The ODD and operational scenarios are
used as input for this stage.
• In traditional software, after a product release an update
involves a re-assessment process taking a lot of time. This
can be challenging in DL models since their product
lifecycle is more likely to be updated.
• New phases not contemplated by the traditional V-model:
• Data management
• Learning management
• Inference management
CONTENTS
01
02
03
04
CONTENTS
CONTEXTUALIZATION
PROPOSED AI-FSM LIFECYCLE
CONCLUSION & FUTURE RESEARCH
DISCUSSION
8
Scope:
The current version of this AI-FSM is restricted to DL constituents with the following features:
• DL algorithms based on supervised learning for visual perception classification tasks.
• Applications based on offline learning processes in which the model remains fixed at approval
time, while excluding online learning processes.
The AI-FSM is based on the recommendations drawn from:
• ISO 26262 and IEC 61508
• IEC TR 5469
• EASA Concept Paper
• AMLAS
• A-SPICE for ML
AI-FSM
9
AI-FSM
• Main procedure: It provides a set of steps required to generate the
basic structure for a specific safety-related project. It serves as an
internal guideline for fulfilling the procedure template.
• Procedure template: This document compiles how functional safety
has been assessed within the organization.
• Guidelines: These documents offer additional guidance for specific
processes.
• Templates: Standard documents used to collect the information
consistently. They often include examples and tables to be
completed.
• Internal Reviews (IRs): reviews based on the activities of the left side
of the safety lifecycle. Objective: Check that the activities defined in
each phase have been properly carried out:
• Quality Assurance
Types of documents: Structure of folders:
10
Proposed lifecycle
11
Proposed lifecycle
• IEC 61508 traditional functional safety lifecycle (Software V-model) + AI lifecycle
12
Proposed lifecycle: phases’ objectives
• Ph0 AI Overall Lifecycle: It is a transversal phase that collects all
the generic project information
• Documents generated
• Organization chart
• Tools selection
• Ph1 DL-Related Concept Specification: This phase encompasses the definition of the DL
Operational Design Domain (ODD) and operational scenarios in which the DL will operate. In the
case the safety-related system entails the use of DL, these definitions are required besides the
traditional description of the use case and the definition of the operation reflected in the
requirements.
• DL Modules (interfaces): This box highlights that Ph2.2 shall define all the interfaces of the DL
modules.
• Ph2 DL Requirements Specification: This phase allocates the software requirements to DL
constituents and refines them:
• Safety, operation, functional and non-functional requirements specification (among others)
13
Proposed lifecycle: phases’ objectives
• PhDM Data Management. It is responsible for collecting and
preparing the datasets. Four steps:
• Data req. Specifications. It allocates the DL req. to the data req. and
refine them. It shall collect:
• Data and datasets req.
• Req. Associated with the collection and preparation steps.
• Data filename policy.
• Data collection. It involves collecting all the data to generate the datasets:
• Data gathering. It involves gathering data from different sources.
• Data generation. It relates to generating new data to complete the data gathering.
• Data preparation. In this step, the previous data is cleaned, processed, or annotated to meet the reqs.
• Data verification. This phase checks if the datasets meet the data req. specification.
• Inputs:
• DL reqs specifications
• ODD
• Operational scenarios
All
actions
and
decisions
taken
shall
be
documented
• Ouputs generated:
• Development dataset (training + validation*)
• Verification dataset
14
Proposed lifecycle: phases’ objectives
• PhLM Learning Management. It is responsible for generating a DL
model that meets the DL req. specification. Five steps:
• Learning req. Specifications. It allocates the DL req. to learning reqs. and
refine them. It shall collect:
• Qualitative and quantitative learning reqs.
• Model post-training selection criteria.
• Reqs. associated with the model design and training.
• Model design. It focuses on the specification of a set of DL models that best suit the application.
• Model training. In this step, the specified models are generated employing the training dataset.
• Model evaluation. Once the model(s) are trained, they are evaluated employing the validation dataset.
• Model verification. This step not only evaluates the generalization capabilities and identifies potential
issues using the verification dataset but also checks if the reqs. are met.
• Inputs:
• Development dataset (training + validation*) from PhDM
• Verification dataset from PhDM
• DL req. specification
All
actions
and
decisions
taken
shall
be
documented
• Ouputs:
• Trained model
• Evaluated model
• Verified learning model
15
Proposed lifecycle: phases’ objectives
• PhIM Inference Management. Its purpose is to adapt the verified
model for its deployment on the target HW while ensuring that it still
meets the DL reqs. after converting and even optimising it. Five steps:
• Inference req. specification. It allocates the DL and learning reqs. to
inference reqs. and refine them. It shall collect:
• Inference reqs.
• Req. associated with the model conversion, optimization and deployment
• Model conversion. The model is transformed into a format suitable for deployment that must ensure
compatibility with the specific target inference platform.
• Model optimisation. the model may undergo optimization to enhance its performance, reduce its
size, or adapt it for resource-constrained environments.
• Deployment. This steps entails the implementation of the model in the target platform.
• Inference verification. This phase not only evaluates the generalization capabilities and identifies
potential issues using the verification dataset but also checks if the reqs. are met.
• Input:
• Verified learning model from PhLM
• Verification dataset from PhDM
• Learning and DL req. specification
• Ouput:
• Verified inference model
All
actions
and
decisions
taken
shall
be
documented
CONTENTS
01
02
03
04
CONTENTS
CONTEXTUALIZATION
PROPOSED LIFECYCLE
CONCLUSION & FUTURE RESEARCH
DISCUSSION
17
In recent years various emerging initiatives and standards have been developed to align the
use of AI in safety-critical systems, but it remains an open research challenge. In this context,
this paper addresses the updates required in the realization phase of traditional functional
safety standards to support the safe development of AI in safety-critical systems.
In the future, the AI-FSM may be updated to align with upcoming iterations of emerging
standards such as:
• ISO/CD PAS 8800: Road Vehicles - Safety and artificial intelligence
• IEC TS 6254: Information technology - Artificial intelligence - Objectives and approaches
for explainability and interpretability of ML models and AI systems
Conclusions and Future Work
CONTENTS
01
02
03
04
CONTENTS
DISCUSSION
CONTEXTUALIZATION
PROPOSED LIFECYCLE
CONCLUSION & FUTURE RESEARCH
19
QUESTIONS ?
DISCUSSION
IKERLAN
P.º José María Arizmendiarrieta, 2 - 20500 Arrasate-Mondragón
T. +34 943712400 F. +34 943796944
THANK YOU!
IKERLAN
P.º José María Arizmendiarrieta, 2 - 20500 Arrasate-Mondragón
T. +34 943712400 F. +34 943796944
NAME: JAVIER FERNÁNDEZ
EMAIL: JAVIER.FERNANDEZ@IKERLAN.ES
Acknowledgements:
• The research leading to these results has received funding from the European
Union’s Horizon Europe Programme under the SAFEXPLAIN Project
(www.safexplain.eu), grant agreement num. 101069595.
• Jaume Abella has also been partially supported by the Spanish Ministry of
Science and Innovation under grant PID2019-107255GB-
C21/AEI/10.13039/501100011033.

More Related Content

Similar to Javier_Fernandez_CARS_workshop_presentation.pptx

Software engineering jwfiles 3
Software engineering jwfiles 3Software engineering jwfiles 3
Software engineering jwfiles 3Azhar Shaik
 
Software engineering jwfiles 3
Software engineering jwfiles 3Software engineering jwfiles 3
Software engineering jwfiles 3Azhar Shaik
 
Software development life cycle
Software development life cycle Software development life cycle
Software development life cycle ParikshitTaksande1
 
Software development life cycle
Software development life cycle Software development life cycle
Software development life cycle ParikshitTaksande1
 
SDLC - Software Development Life Cycle
SDLC - Software Development Life CycleSDLC - Software Development Life Cycle
SDLC - Software Development Life CycleSaravanan Manoharan
 
SDLC - Software Development Life Cycle
SDLC - Software Development Life CycleSDLC - Software Development Life Cycle
SDLC - Software Development Life CycleSaravanan Manoharan
 
Generic Software Process Models
Generic Software Process ModelsGeneric Software Process Models
Generic Software Process ModelsEducation Front
 
Generic Software Process Models
Generic Software Process ModelsGeneric Software Process Models
Generic Software Process ModelsEducation Front
 
DISE - Introduction to Software Engineering
DISE - Introduction to Software EngineeringDISE - Introduction to Software Engineering
DISE - Introduction to Software EngineeringRasan Samarasinghe
 
DISE - Introduction to Software Engineering
DISE - Introduction to Software EngineeringDISE - Introduction to Software Engineering
DISE - Introduction to Software EngineeringRasan Samarasinghe
 
Lect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPMLect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPMMubashir Ali
 
Lect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPMLect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPMMubashir Ali
 

Similar to Javier_Fernandez_CARS_workshop_presentation.pptx (20)

Process Models
Process ModelsProcess Models
Process Models
 
Process Models
Process ModelsProcess Models
Process Models
 
Process Models
Process ModelsProcess Models
Process Models
 
Process Models
Process ModelsProcess Models
Process Models
 
Seminar on Project Management by Rj
Seminar on Project Management by RjSeminar on Project Management by Rj
Seminar on Project Management by Rj
 
Seminar on Project Management by Rj
Seminar on Project Management by RjSeminar on Project Management by Rj
Seminar on Project Management by Rj
 
Software engineering jwfiles 3
Software engineering jwfiles 3Software engineering jwfiles 3
Software engineering jwfiles 3
 
Software engineering jwfiles 3
Software engineering jwfiles 3Software engineering jwfiles 3
Software engineering jwfiles 3
 
Software development life cycle
Software development life cycle Software development life cycle
Software development life cycle
 
Software development life cycle
Software development life cycle Software development life cycle
Software development life cycle
 
SDLC - Software Development Life Cycle
SDLC - Software Development Life CycleSDLC - Software Development Life Cycle
SDLC - Software Development Life Cycle
 
SDLC - Software Development Life Cycle
SDLC - Software Development Life CycleSDLC - Software Development Life Cycle
SDLC - Software Development Life Cycle
 
Generic Software Process Models
Generic Software Process ModelsGeneric Software Process Models
Generic Software Process Models
 
Generic Software Process Models
Generic Software Process ModelsGeneric Software Process Models
Generic Software Process Models
 
DISE - Introduction to Software Engineering
DISE - Introduction to Software EngineeringDISE - Introduction to Software Engineering
DISE - Introduction to Software Engineering
 
DISE - Introduction to Software Engineering
DISE - Introduction to Software EngineeringDISE - Introduction to Software Engineering
DISE - Introduction to Software Engineering
 
Lect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPMLect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPM
 
Lect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPMLect-4: Software Development Life Cycle Model - SPM
Lect-4: Software Development Life Cycle Model - SPM
 
sdlc.pptx
sdlc.pptxsdlc.pptx
sdlc.pptx
 
sdlc.pptx
sdlc.pptxsdlc.pptx
sdlc.pptx
 

Recently uploaded

Passive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.pptPassive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.pptamrabdallah9
 
Software Engineering Practical File Front Pages.pdf
Software Engineering Practical File Front Pages.pdfSoftware Engineering Practical File Front Pages.pdf
Software Engineering Practical File Front Pages.pdfssuser5c9d4b1
 
Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...
Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...
Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...drjose256
 
Artificial Intelligence in due diligence
Artificial Intelligence in due diligenceArtificial Intelligence in due diligence
Artificial Intelligence in due diligencemahaffeycheryld
 
5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...archanaece3
 
What is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, FunctionsWhat is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, FunctionsVIEW
 
Interfacing Analog to Digital Data Converters ee3404.pdf
Interfacing Analog to Digital Data Converters ee3404.pdfInterfacing Analog to Digital Data Converters ee3404.pdf
Interfacing Analog to Digital Data Converters ee3404.pdfragupathi90
 
Adsorption (mass transfer operations 2) ppt
Adsorption (mass transfer operations 2) pptAdsorption (mass transfer operations 2) ppt
Adsorption (mass transfer operations 2) pptjigup7320
 
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdflitvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdfAlexander Litvinenko
 
Basics of Relay for Engineering Students
Basics of Relay for Engineering StudentsBasics of Relay for Engineering Students
Basics of Relay for Engineering Studentskannan348865
 
UNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptxUNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptxkalpana413121
 
8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...josephjonse
 
analog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptxanalog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptxKarpagam Institute of Teechnology
 
21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological university21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological universityMohd Saifudeen
 
15-Minute City: A Completely New Horizon
15-Minute City: A Completely New Horizon15-Minute City: A Completely New Horizon
15-Minute City: A Completely New HorizonMorshed Ahmed Rahath
 
Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..MaherOthman7
 
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptxSLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptxCHAIRMAN M
 
Raashid final report on Embedded Systems
Raashid final report on Embedded SystemsRaashid final report on Embedded Systems
Raashid final report on Embedded SystemsRaashidFaiyazSheikh
 
Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1T.D. Shashikala
 
Augmented Reality (AR) with Augin Software.pptx
Augmented Reality (AR) with Augin Software.pptxAugmented Reality (AR) with Augin Software.pptx
Augmented Reality (AR) with Augin Software.pptxMustafa Ahmed
 

Recently uploaded (20)

Passive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.pptPassive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.ppt
 
Software Engineering Practical File Front Pages.pdf
Software Engineering Practical File Front Pages.pdfSoftware Engineering Practical File Front Pages.pdf
Software Engineering Practical File Front Pages.pdf
 
Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...
Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...
Tembisa Central Terminating Pills +27838792658 PHOMOLONG Top Abortion Pills F...
 
Artificial Intelligence in due diligence
Artificial Intelligence in due diligenceArtificial Intelligence in due diligence
Artificial Intelligence in due diligence
 
5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...
 
What is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, FunctionsWhat is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, Functions
 
Interfacing Analog to Digital Data Converters ee3404.pdf
Interfacing Analog to Digital Data Converters ee3404.pdfInterfacing Analog to Digital Data Converters ee3404.pdf
Interfacing Analog to Digital Data Converters ee3404.pdf
 
Adsorption (mass transfer operations 2) ppt
Adsorption (mass transfer operations 2) pptAdsorption (mass transfer operations 2) ppt
Adsorption (mass transfer operations 2) ppt
 
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdflitvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
 
Basics of Relay for Engineering Students
Basics of Relay for Engineering StudentsBasics of Relay for Engineering Students
Basics of Relay for Engineering Students
 
UNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptxUNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptx
 
8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...
 
analog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptxanalog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptx
 
21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological university21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological university
 
15-Minute City: A Completely New Horizon
15-Minute City: A Completely New Horizon15-Minute City: A Completely New Horizon
15-Minute City: A Completely New Horizon
 
Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..
 
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptxSLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
 
Raashid final report on Embedded Systems
Raashid final report on Embedded SystemsRaashid final report on Embedded Systems
Raashid final report on Embedded Systems
 
Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1
 
Augmented Reality (AR) with Augin Software.pptx
Augmented Reality (AR) with Augin Software.pptxAugmented Reality (AR) with Augin Software.pptx
Augmented Reality (AR) with Augin Software.pptx
 

Javier_Fernandez_CARS_workshop_presentation.pptx

  • 1. This project has received funding from the European Union's Horizon Europe programme under grant agreement number 101069595. ‡ AI-FSM: Towards Functional Safety Management for Artificial Intelligence-based Critical Systems Presenter: Javier Fernández - IKERLAN Javier Fernández†, Irune Agirre†, Jon Perez- Cerrolaza†, Lorea Belategi†, Ana Adell†, Carlo Donzella⋆, Jaume Abella‡ (†) (⋆) (‡)
  • 3. 3 Functional Safety Management (FSM): encompasses all essential activities throughout the Functional Safety lifecycle phases, as mandated by IEC 61508-1. FSM is designed to prevent errors during specification, design, development, manufacturing, and commissioning. Context
  • 4. 4 AI lifecycle phases • Five main stages: • Requirements Specification • Data Management • Development dataset • Training + Validation* dataset • Verification dataset • Model training • Trained model • Model verification • Verified model • Model Deployment • Inference model Context
  • 5. 5 The general DL-based systems development process clashes frontally with traditional safety development processes: • DL SW is designed monolithically following empirical training processes with example training data, rather than implementing specific safety requirements. • DL SW, as opposed to any other kind of SW in safety critical systems, cannot be considered as correct by design due to the predictive nature that comes along with mispredictions and confidence values. • DL SW design is no longer independent of data, and its parameters are set empirically based on training datasets. • DL SW imposes high-performance demands on the underlying HW and its inherent complexity (both HW and SW) entails challenges to comply with safety standards. Context
  • 6. 6 Context Main phases affected by including DL • Definition of the ODD and operational scenarios • HARA shall identify potential hazards caused by the DL- based system. The ODD and operational scenarios are used as input for this stage. • In traditional software, after a product release an update involves a re-assessment process taking a lot of time. This can be challenging in DL models since their product lifecycle is more likely to be updated. • New phases not contemplated by the traditional V-model: • Data management • Learning management • Inference management
  • 8. 8 Scope: The current version of this AI-FSM is restricted to DL constituents with the following features: • DL algorithms based on supervised learning for visual perception classification tasks. • Applications based on offline learning processes in which the model remains fixed at approval time, while excluding online learning processes. The AI-FSM is based on the recommendations drawn from: • ISO 26262 and IEC 61508 • IEC TR 5469 • EASA Concept Paper • AMLAS • A-SPICE for ML AI-FSM
  • 9. 9 AI-FSM • Main procedure: It provides a set of steps required to generate the basic structure for a specific safety-related project. It serves as an internal guideline for fulfilling the procedure template. • Procedure template: This document compiles how functional safety has been assessed within the organization. • Guidelines: These documents offer additional guidance for specific processes. • Templates: Standard documents used to collect the information consistently. They often include examples and tables to be completed. • Internal Reviews (IRs): reviews based on the activities of the left side of the safety lifecycle. Objective: Check that the activities defined in each phase have been properly carried out: • Quality Assurance Types of documents: Structure of folders:
  • 11. 11 Proposed lifecycle • IEC 61508 traditional functional safety lifecycle (Software V-model) + AI lifecycle
  • 12. 12 Proposed lifecycle: phases’ objectives • Ph0 AI Overall Lifecycle: It is a transversal phase that collects all the generic project information • Documents generated • Organization chart • Tools selection • Ph1 DL-Related Concept Specification: This phase encompasses the definition of the DL Operational Design Domain (ODD) and operational scenarios in which the DL will operate. In the case the safety-related system entails the use of DL, these definitions are required besides the traditional description of the use case and the definition of the operation reflected in the requirements. • DL Modules (interfaces): This box highlights that Ph2.2 shall define all the interfaces of the DL modules. • Ph2 DL Requirements Specification: This phase allocates the software requirements to DL constituents and refines them: • Safety, operation, functional and non-functional requirements specification (among others)
  • 13. 13 Proposed lifecycle: phases’ objectives • PhDM Data Management. It is responsible for collecting and preparing the datasets. Four steps: • Data req. Specifications. It allocates the DL req. to the data req. and refine them. It shall collect: • Data and datasets req. • Req. Associated with the collection and preparation steps. • Data filename policy. • Data collection. It involves collecting all the data to generate the datasets: • Data gathering. It involves gathering data from different sources. • Data generation. It relates to generating new data to complete the data gathering. • Data preparation. In this step, the previous data is cleaned, processed, or annotated to meet the reqs. • Data verification. This phase checks if the datasets meet the data req. specification. • Inputs: • DL reqs specifications • ODD • Operational scenarios All actions and decisions taken shall be documented • Ouputs generated: • Development dataset (training + validation*) • Verification dataset
  • 14. 14 Proposed lifecycle: phases’ objectives • PhLM Learning Management. It is responsible for generating a DL model that meets the DL req. specification. Five steps: • Learning req. Specifications. It allocates the DL req. to learning reqs. and refine them. It shall collect: • Qualitative and quantitative learning reqs. • Model post-training selection criteria. • Reqs. associated with the model design and training. • Model design. It focuses on the specification of a set of DL models that best suit the application. • Model training. In this step, the specified models are generated employing the training dataset. • Model evaluation. Once the model(s) are trained, they are evaluated employing the validation dataset. • Model verification. This step not only evaluates the generalization capabilities and identifies potential issues using the verification dataset but also checks if the reqs. are met. • Inputs: • Development dataset (training + validation*) from PhDM • Verification dataset from PhDM • DL req. specification All actions and decisions taken shall be documented • Ouputs: • Trained model • Evaluated model • Verified learning model
  • 15. 15 Proposed lifecycle: phases’ objectives • PhIM Inference Management. Its purpose is to adapt the verified model for its deployment on the target HW while ensuring that it still meets the DL reqs. after converting and even optimising it. Five steps: • Inference req. specification. It allocates the DL and learning reqs. to inference reqs. and refine them. It shall collect: • Inference reqs. • Req. associated with the model conversion, optimization and deployment • Model conversion. The model is transformed into a format suitable for deployment that must ensure compatibility with the specific target inference platform. • Model optimisation. the model may undergo optimization to enhance its performance, reduce its size, or adapt it for resource-constrained environments. • Deployment. This steps entails the implementation of the model in the target platform. • Inference verification. This phase not only evaluates the generalization capabilities and identifies potential issues using the verification dataset but also checks if the reqs. are met. • Input: • Verified learning model from PhLM • Verification dataset from PhDM • Learning and DL req. specification • Ouput: • Verified inference model All actions and decisions taken shall be documented
  • 17. 17 In recent years various emerging initiatives and standards have been developed to align the use of AI in safety-critical systems, but it remains an open research challenge. In this context, this paper addresses the updates required in the realization phase of traditional functional safety standards to support the safe development of AI in safety-critical systems. In the future, the AI-FSM may be updated to align with upcoming iterations of emerging standards such as: • ISO/CD PAS 8800: Road Vehicles - Safety and artificial intelligence • IEC TS 6254: Information technology - Artificial intelligence - Objectives and approaches for explainability and interpretability of ML models and AI systems Conclusions and Future Work
  • 20. IKERLAN P.º José María Arizmendiarrieta, 2 - 20500 Arrasate-Mondragón T. +34 943712400 F. +34 943796944 THANK YOU!
  • 21. IKERLAN P.º José María Arizmendiarrieta, 2 - 20500 Arrasate-Mondragón T. +34 943712400 F. +34 943796944 NAME: JAVIER FERNÁNDEZ EMAIL: JAVIER.FERNANDEZ@IKERLAN.ES Acknowledgements: • The research leading to these results has received funding from the European Union’s Horizon Europe Programme under the SAFEXPLAIN Project (www.safexplain.eu), grant agreement num. 101069595. • Jaume Abella has also been partially supported by the Spanish Ministry of Science and Innovation under grant PID2019-107255GB- C21/AEI/10.13039/501100011033.

Editor's Notes

  1. The development of safety-critical systems follows a well known V-model, moving from safety goals to safety requirements, system architecture design, Software (SW) and Hardware (HW) architecture design, and implementation to obtain a system that is intended to be safe by construction. Then, the testing phase takes place from unit testing up to full system testing against its safety requirements. The Functional Safety Management (FSM) defines the required systematic approach (e.g., steps, actions, technical considerations) for developing safety-critical systems and other lifecycle phases, from concept definition up to decommissioning and disposal.
  2. Although AI-FSM maintains the name "validation" according to AI nomenclature, it would not correspond to validation in the context of safety. Simplified DL Lifecycle
  3. Traditionally, safety-critical systems incorporating soft ware are developed, tested, and validated during design. When the software is updated after the product release, this is done only in a time interval of sev eral months or years, since the re-assessment process takes a lot of time. In con trast to traditional software, which ensures the correctness in every corner case, ML models are created based on a data set, which can not cover all possible corner cases. Thus, it is very likely that the ML model will be updated continuously over the product lifetime
  4. Moreover, there is a lack of guidance in the development process for safety-critical systems incorporating AI. To address these challenges, this paper proposes a new development process that maps the traditional lifecycle of safety-critical systems with the AI lifecycle, addressing their interactions.
  5. Traditional functional safety standards such as IEC 61508 and ISO 26262 [7] define separate V-model based procedures for safety-related SW and HW.
  6. The Phase 0 (Ph0) is a transversal phase that collects all the generic project information, e.g., project document list, organizational chart, or tools selection. It must consider information collected in an AI-related safety project.
  7. All development data, even in the traditional models, are subject to the “CIA principle” of Confidentiality, Integrity, and Accessibility, especially so for Configuration Parameters
  8. Qualitative requirements: - a methodology for searching the hyperparameters of the model - defining the initial parameters of the model. Quantitative: - robustness ((i.e., the model shall perform within the performance thresholds within unseen data in the training dataset). - performance (accuracy, precisión, recall) Once the learning requirements specifications are defined, the Model Design step focuses on the specification of a set of DL models that best suit the application. On the one hand, a scenario may arise in which none of the previous candidates achieves the expected performance. In such cases, an iterative repetition of the model design, training, and evaluation becomes necessary. These iterations continue until the stipulated performance requirements are successfully met. Otherwise, a new iteration of the PhDM should be carried out. On the other hand, if one or multiple candidate models demonstrate the anticipated performance levels, they will be verified in the next step.