SlideShare a Scribd company logo
Matteo Bordin
bordin@adacore.com
How does it feel to qualify a tool for DO-178?
The Big Freeze problem
Is Tool Qualification equivalent to Baselining?
Verify a slightly different coding standard
Change the code generation strategy
Regenerate a source-to-obj traceability study
The Big Freeze problem (II)
A new bug is found!
Please, do NOT fix it!
Really cheaper/safer to invent workarounds?
The Big Freeze problem (III)
So, what is our goal?
AdaCore engineers improve our tech every day
Constraining engineers’ creativity cannot work
…still, we target the high-integrity domain
QUALIFICATION
How to Avoid the Big Freeze Problem (I)
Continuous Qualification
Mirrors the “Continuous Integration” concept
Qualification artifacts evolve along source code
A tool shall always be in a “semi-qualifiable” state
Maximize automation of qualification-oriented activities
How to Avoid the Big Freeze Problem (II)
Delta Qualification
What is the minimum effort to re-achieve qualifiability?
Which artifact shall be (re-)produced
Which activities shall be (re-)performed
An example of Continuous Certification
Requirement R1
Test Case TC1
Source File SF1
Test Case TC2
Test Case TC3
Test Procedure TP1
Test Procedure TP2
Test Procedure TP3
Structural Coverage
Report SCR1
Coding Standard CS1
Coding Standard
Compliance Report CSC1
Modified
Re-generated
Re-generated
A modification of an artifact triggers
the qualification infrastructure to
execute a set of activities
(possibly overnightly)
Continuous Certification + Delta Certification
Sounds nice eh?
Requires an integrated certification infrastructure
 The topic of this talk!
Focus on automation & optimization
Difficult to generalize for different teams/clients
A drastically different way of working
The core problem
Establishing a dependency & traceability graph
To use to trigger automated activities
To calculate the Delta Qualification effort
Low creation & maintenance effort
Understand when an artifact has changed
Propagate un-valid status
Presenting current results…
…as well as un-implemented ideas
A good vision of our way of thinking…
…and how we develop TQL5 kits
From now on, this is a long-term view
Introducing the Qualifying Machine
An intelligent repository for qualification artifacts
Integrated with build/test infrastructure
Provides hooks to invoke scripts to trigger activities
Support for change impact analysis
QM back-end
Qualifying Machine Architecture (I)
SVN, Git, …
Repository
Traceability tracking
Dependency tracking
Action tracking
Un-validity propagation
Automated activity triggering
QM front-end
Development EngineerProcess Engineer
Project Manager
Certification Authority
Testing/Building infrastructure
Qualifying Machine Architecture (II)
Major design decisions & challenges
What to track (artifacts, activities, …)
How to build a traceability & dependency graph
How to understand if an artifact has changed
Un-validity propagation algorithm
What do we track?
Everything is an artifact!
Each artifact is a (set of) file(s)
Metadata are placed in a single folder
Activities ALWAYS produce qualification artifacts
An activity not producing an artifact IS NOT WORTH IT
State of an activity = state of the artifacts it produces
Leverage as much as possible on naming conventions
Configuring the QM
project <NAME> is
artifact Requirement is
files: “req.txt”
activity Decompose produces
derived : Requirement [0..*] in “./Req_<name>”
activity Write_Test_Case produces
test_cases : Test_Case [1..*] in “./TC_<name>”
activity QA_Review produces
qa_report : Report [0..1] in “qa.txt”
traceability:
standard : Doc [1] in “<ROOT>/Standards/req_std.txt”
Building a traceability & dependency graph
My_Project
Req_SysReq1
Req_SwReq1
TC_Flying
TC_Departure
QA
Req_SysReq1
Req_SwReq1
Configuration File
project <NAME> is
artifact Requirement is
files: “req.txt”
activity Decompose produces
derived : Requirement [0..*]
in “./Req_<name>”
activity Write_Test_Case produces
test_cases : Test_Case [1..*]
in “./TC_<name>”
+ =
When does an artifact change?
At the heart of un-validity propagation
Timestamp, SVN diff, MD5, …
How to cope with complex artifacts (UML or Simulink models)?
How to identify changes not requiring formal re-validation?
Example: a typo
TC1
6.3.2bcd
Un-validity propagation algorithm (II)
LLR1
6.4.4.1
Re-validate 6.4.4.1
(test coverage of LLR),
possibly editing TC1
Re-measure
structural coverage
TP1
Source File SF1
Manual activity
Depending on the status of artifacts &
activities, the QM:
- Proposes a minimal list of manual activities
- Triggers automated activities
- Track each single action of the user
LLRs are accurate,
consistent, compatible
and verifiable
TORs
Un-validity propagation algorithm (I)
Known Constraints and
Open Problems
References, but does
not depends on
Tool Qualification Plan
Depends on
Verification Results
Analysis
Delivery File
Modifying the Known Constraints and Open
Problems triggers the production of a minimal
set of manual activities to be performed in order
to re-achieve qualification:
- on TORS
- on elements derived from TORs
How do we use the QM? (I)
Not targeting the typical DOORS users…
Lightweight artifact editing/managing
How do we use the QM? (II)
Un-validity propagation feedback
How do we use the QM? (III)
How do we fill the QM repository? (IV)
However, we understand other tools are used
Develop a set of plug-in
Import artifact metadata in the QM
The QM focuses on continuous/delta qualification
Integrate external dev. tools in the QM
Wrap-up
Our experience
From a semi-agile infrastructure to the QM
Qualification kit for a verification tool (TQL5): can we scale?
Lessons learnt
Fragmentation & Automation!
What about workflow modeling/tracking?
Future developments
www.forge.open-do.org/projects/qmachine
SafeCer Project (Artemis)
Presented by

More Related Content

What's hot

Software Testing with a TDD Application
Software Testing with a TDD ApplicationSoftware Testing with a TDD Application
Software Testing with a TDD Application
Yelda Gurbuz Erdogan
 
How do Software Architects consider Non-Functional Requirements
How do Software Architects consider Non-Functional RequirementsHow do Software Architects consider Non-Functional Requirements
How do Software Architects consider Non-Functional RequirementsGESSI UPC
 
Software Engineering (An Agile View of Process)
Software Engineering (An Agile View of Process)Software Engineering (An Agile View of Process)
Software Engineering (An Agile View of Process)
ShudipPal
 
Agile & Secure SDLC
Agile & Secure SDLCAgile & Secure SDLC
Agile & Secure SDLC
Paul Yang
 
The Software Development Process
The Software Development ProcessThe Software Development Process
The Software Development Process
Cesar Augusto Nogueira
 
Security in the Software Development Life Cycle (SDLC)
Security in the Software Development Life Cycle (SDLC)Security in the Software Development Life Cycle (SDLC)
Security in the Software Development Life Cycle (SDLC)
Frances Coronel
 
Software Process Models
Software Process ModelsSoftware Process Models
Software Process Models
Rody Middelkoop
 
SDLC Models - testing
SDLC Models - testingSDLC Models - testing
SDLC Models - testing
Prasad Gali
 
Sdlc (software development life cycle)
Sdlc (software development life cycle)Sdlc (software development life cycle)
Sdlc (software development life cycle)
Nadeesha Thilakarathne
 
IT Software Development Life Cycle
IT Software Development Life CycleIT Software Development Life Cycle
IT Software Development Life Cycle
Preshita Chaurasiya
 
How do Software Architects consider Non-Functional Requirements - An explorat...
How do Software Architects consider Non-Functional Requirements - An explorat...How do Software Architects consider Non-Functional Requirements - An explorat...
How do Software Architects consider Non-Functional Requirements - An explorat...
Jordi Cabot
 
Software development PROCESS
Software development PROCESSSoftware development PROCESS
Software development PROCESS
Ivano Malavolta
 
SDLC or Software Development Life Cycle
SDLC or Software Development Life CycleSDLC or Software Development Life Cycle
SDLC or Software Development Life Cycle
Jyothi Vbs
 
Software development life cycle
Software development life cycleSoftware development life cycle
Software development life cycle
Vaibhav యం.
 
Agile & Iconix sdlc
Agile & Iconix sdlcAgile & Iconix sdlc
Agile & Iconix sdlc
Ahmed Nehad
 
Software development methodologies
Software development methodologiesSoftware development methodologies
Software development methodologies
Ankita Lachhwani
 
6 basic steps of software development process
6 basic steps of software development process6 basic steps of software development process
6 basic steps of software development processRiant Soft
 
Software development life cycle
Software development life cycleSoftware development life cycle
Software development life cycleA Subbiah
 

What's hot (20)

Software Testing with a TDD Application
Software Testing with a TDD ApplicationSoftware Testing with a TDD Application
Software Testing with a TDD Application
 
How do Software Architects consider Non-Functional Requirements
How do Software Architects consider Non-Functional RequirementsHow do Software Architects consider Non-Functional Requirements
How do Software Architects consider Non-Functional Requirements
 
Software Engineering (An Agile View of Process)
Software Engineering (An Agile View of Process)Software Engineering (An Agile View of Process)
Software Engineering (An Agile View of Process)
 
Agile & Secure SDLC
Agile & Secure SDLCAgile & Secure SDLC
Agile & Secure SDLC
 
The Software Development Process
The Software Development ProcessThe Software Development Process
The Software Development Process
 
Security in the Software Development Life Cycle (SDLC)
Security in the Software Development Life Cycle (SDLC)Security in the Software Development Life Cycle (SDLC)
Security in the Software Development Life Cycle (SDLC)
 
Software Process Models
Software Process ModelsSoftware Process Models
Software Process Models
 
SDLC Models - testing
SDLC Models - testingSDLC Models - testing
SDLC Models - testing
 
Mt s2 sdlc
Mt s2 sdlcMt s2 sdlc
Mt s2 sdlc
 
Sdlc (software development life cycle)
Sdlc (software development life cycle)Sdlc (software development life cycle)
Sdlc (software development life cycle)
 
IT Software Development Life Cycle
IT Software Development Life CycleIT Software Development Life Cycle
IT Software Development Life Cycle
 
How do Software Architects consider Non-Functional Requirements - An explorat...
How do Software Architects consider Non-Functional Requirements - An explorat...How do Software Architects consider Non-Functional Requirements - An explorat...
How do Software Architects consider Non-Functional Requirements - An explorat...
 
Software development PROCESS
Software development PROCESSSoftware development PROCESS
Software development PROCESS
 
SDLC or Software Development Life Cycle
SDLC or Software Development Life CycleSDLC or Software Development Life Cycle
SDLC or Software Development Life Cycle
 
Unit 8
Unit 8Unit 8
Unit 8
 
Software development life cycle
Software development life cycleSoftware development life cycle
Software development life cycle
 
Agile & Iconix sdlc
Agile & Iconix sdlcAgile & Iconix sdlc
Agile & Iconix sdlc
 
Software development methodologies
Software development methodologiesSoftware development methodologies
Software development methodologies
 
6 basic steps of software development process
6 basic steps of software development process6 basic steps of software development process
6 basic steps of software development process
 
Software development life cycle
Software development life cycleSoftware development life cycle
Software development life cycle
 

Similar to Challenges of Agile Qualification

Automating the Quality
Automating the QualityAutomating the Quality
Automating the Quality
Dejan Vukmirovic
 
SCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome ThemSCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome Them
Compuware
 
SCA in an Agile World | June 2010
SCA in an Agile World | June 2010SCA in an Agile World | June 2010
SCA in an Agile World | June 2010
Klocwork
 
DevOPs Transformation Workshop
DevOPs Transformation WorkshopDevOPs Transformation Workshop
DevOPs Transformation Workshop
Jules Pierre-Louis
 
Luiz Fernando Testa Contador - Aplicando DevOps em grandes corporações
Luiz Fernando Testa Contador - Aplicando DevOps em grandes corporaçõesLuiz Fernando Testa Contador - Aplicando DevOps em grandes corporações
Luiz Fernando Testa Contador - Aplicando DevOps em grandes corporações
Agile Trends
 
Gcs day1
Gcs day1Gcs day1
Gcs day1
Sriram Angajala
 
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8sShipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
Andreas Grabner
 
From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018
Christophe Rochefolle
 
Dances with bits - industrial data analytics made easy!
Dances with bits - industrial data analytics made easy!Dances with bits - industrial data analytics made easy!
Dances with bits - industrial data analytics made easy!
Julian Feinauer
 
DesignState Intralink to AgilePLM
DesignState Intralink to AgilePLMDesignState Intralink to AgilePLM
DesignState Intralink to AgilePLM
Zero Wait-State
 
ALM (Application Lifecycle Management)
ALM (Application Lifecycle Management)ALM (Application Lifecycle Management)
ALM (Application Lifecycle Management)
Terry Cho
 
The Need for Speed
The Need for SpeedThe Need for Speed
The Need for Speed
Capgemini
 
Qtp interview questions
Qtp interview questionsQtp interview questions
Qtp interview questionsRamu Palanki
 
Qtp interview questions
Qtp interview questionsQtp interview questions
Qtp interview questionsRamu Palanki
 
An Introduction to Microservices
An Introduction to MicroservicesAn Introduction to Microservices
An Introduction to Microservices
Ad van der Veer
 
Workshop: Delivering chnages for applications and databases
Workshop: Delivering chnages for applications and databasesWorkshop: Delivering chnages for applications and databases
Workshop: Delivering chnages for applications and databases
Eduardo Piairo
 
Critical Capabilities to Shifting Left the Right Way
Critical Capabilities to Shifting Left the Right WayCritical Capabilities to Shifting Left the Right Way
Critical Capabilities to Shifting Left the Right Way
SmartBear
 
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...
Simon Storm
 
How to develop a common sense of "DONE"?
How to develop a common sense of "DONE"?How to develop a common sense of "DONE"?
How to develop a common sense of "DONE"?
Dr. Alexander Schwartz
 
FME World Tour 2015 - FME & Data Migration Simon McCabe
FME World Tour 2015 -  FME & Data Migration Simon McCabeFME World Tour 2015 -  FME & Data Migration Simon McCabe
FME World Tour 2015 - FME & Data Migration Simon McCabe
IMGS
 

Similar to Challenges of Agile Qualification (20)

Automating the Quality
Automating the QualityAutomating the Quality
Automating the Quality
 
SCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome ThemSCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome Them
 
SCA in an Agile World | June 2010
SCA in an Agile World | June 2010SCA in an Agile World | June 2010
SCA in an Agile World | June 2010
 
DevOPs Transformation Workshop
DevOPs Transformation WorkshopDevOPs Transformation Workshop
DevOPs Transformation Workshop
 
Luiz Fernando Testa Contador - Aplicando DevOps em grandes corporações
Luiz Fernando Testa Contador - Aplicando DevOps em grandes corporaçõesLuiz Fernando Testa Contador - Aplicando DevOps em grandes corporações
Luiz Fernando Testa Contador - Aplicando DevOps em grandes corporações
 
Gcs day1
Gcs day1Gcs day1
Gcs day1
 
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8sShipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
 
From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018
 
Dances with bits - industrial data analytics made easy!
Dances with bits - industrial data analytics made easy!Dances with bits - industrial data analytics made easy!
Dances with bits - industrial data analytics made easy!
 
DesignState Intralink to AgilePLM
DesignState Intralink to AgilePLMDesignState Intralink to AgilePLM
DesignState Intralink to AgilePLM
 
ALM (Application Lifecycle Management)
ALM (Application Lifecycle Management)ALM (Application Lifecycle Management)
ALM (Application Lifecycle Management)
 
The Need for Speed
The Need for SpeedThe Need for Speed
The Need for Speed
 
Qtp interview questions
Qtp interview questionsQtp interview questions
Qtp interview questions
 
Qtp interview questions
Qtp interview questionsQtp interview questions
Qtp interview questions
 
An Introduction to Microservices
An Introduction to MicroservicesAn Introduction to Microservices
An Introduction to Microservices
 
Workshop: Delivering chnages for applications and databases
Workshop: Delivering chnages for applications and databasesWorkshop: Delivering chnages for applications and databases
Workshop: Delivering chnages for applications and databases
 
Critical Capabilities to Shifting Left the Right Way
Critical Capabilities to Shifting Left the Right WayCritical Capabilities to Shifting Left the Right Way
Critical Capabilities to Shifting Left the Right Way
 
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...
 
How to develop a common sense of "DONE"?
How to develop a common sense of "DONE"?How to develop a common sense of "DONE"?
How to develop a common sense of "DONE"?
 
FME World Tour 2015 - FME & Data Migration Simon McCabe
FME World Tour 2015 -  FME & Data Migration Simon McCabeFME World Tour 2015 -  FME & Data Migration Simon McCabe
FME World Tour 2015 - FME & Data Migration Simon McCabe
 

More from AdaCore

RCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standardsRCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standards
AdaCore
 
Have we a Human Ecosystem?
Have we a Human Ecosystem?Have we a Human Ecosystem?
Have we a Human Ecosystem?
AdaCore
 
Rust and the coming age of high integrity languages
Rust and the coming age of high integrity languagesRust and the coming age of high integrity languages
Rust and the coming age of high integrity languages
AdaCore
 
SPARKNaCl: A verified, fast cryptographic library
SPARKNaCl: A verified, fast cryptographic librarySPARKNaCl: A verified, fast cryptographic library
SPARKNaCl: A verified, fast cryptographic library
AdaCore
 
Developing Future High Integrity Processing Solutions
Developing Future High Integrity Processing SolutionsDeveloping Future High Integrity Processing Solutions
Developing Future High Integrity Processing Solutions
AdaCore
 
Taming event-driven software via formal verification
Taming event-driven software via formal verificationTaming event-driven software via formal verification
Taming event-driven software via formal verification
AdaCore
 
Pushing the Boundary of Mostly Automatic Program Proof
Pushing the Boundary of Mostly Automatic Program ProofPushing the Boundary of Mostly Automatic Program Proof
Pushing the Boundary of Mostly Automatic Program Proof
AdaCore
 
RCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standardsRCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standards
AdaCore
 
Product Lines and Ecosystems: from customization to configuration
Product Lines and Ecosystems: from customization to configurationProduct Lines and Ecosystems: from customization to configuration
Product Lines and Ecosystems: from customization to configuration
AdaCore
 
Securing the Future of Safety and Security of Embedded Software
Securing the Future of Safety and Security of Embedded SoftwareSecuring the Future of Safety and Security of Embedded Software
Securing the Future of Safety and Security of Embedded Software
AdaCore
 
Spark / Ada for Safe and Secure Firmware Development
Spark / Ada for Safe and Secure Firmware DevelopmentSpark / Ada for Safe and Secure Firmware Development
Spark / Ada for Safe and Secure Firmware Development
AdaCore
 
Introducing the HICLASS Research Programme - Enabling Development of Complex ...
Introducing the HICLASS Research Programme - Enabling Development of Complex ...Introducing the HICLASS Research Programme - Enabling Development of Complex ...
Introducing the HICLASS Research Programme - Enabling Development of Complex ...
AdaCore
 
The Future of Aerospace – More Software Please!
The Future of Aerospace – More Software Please!The Future of Aerospace – More Software Please!
The Future of Aerospace – More Software Please!
AdaCore
 
Adaptive AUTOSAR - The New AUTOSAR Architecture
Adaptive AUTOSAR - The New AUTOSAR ArchitectureAdaptive AUTOSAR - The New AUTOSAR Architecture
Adaptive AUTOSAR - The New AUTOSAR Architecture
AdaCore
 
Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...
Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...
Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...
AdaCore
 
Software Engineering for Robotics - The RoboStar Technology
Software Engineering for Robotics - The RoboStar TechnologySoftware Engineering for Robotics - The RoboStar Technology
Software Engineering for Robotics - The RoboStar Technology
AdaCore
 
MISRA C in an ISO 26262 context
MISRA C in an ISO 26262 contextMISRA C in an ISO 26262 context
MISRA C in an ISO 26262 context
AdaCore
 
Application of theorem proving for safety-critical vehicle software
Application of theorem proving for safety-critical vehicle softwareApplication of theorem proving for safety-critical vehicle software
Application of theorem proving for safety-critical vehicle software
AdaCore
 
The Application of Formal Methods to Railway Signalling Software
The Application of Formal Methods to Railway Signalling SoftwareThe Application of Formal Methods to Railway Signalling Software
The Application of Formal Methods to Railway Signalling Software
AdaCore
 
Bounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise EnvironmentBounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise Environment
AdaCore
 

More from AdaCore (20)

RCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standardsRCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standards
 
Have we a Human Ecosystem?
Have we a Human Ecosystem?Have we a Human Ecosystem?
Have we a Human Ecosystem?
 
Rust and the coming age of high integrity languages
Rust and the coming age of high integrity languagesRust and the coming age of high integrity languages
Rust and the coming age of high integrity languages
 
SPARKNaCl: A verified, fast cryptographic library
SPARKNaCl: A verified, fast cryptographic librarySPARKNaCl: A verified, fast cryptographic library
SPARKNaCl: A verified, fast cryptographic library
 
Developing Future High Integrity Processing Solutions
Developing Future High Integrity Processing SolutionsDeveloping Future High Integrity Processing Solutions
Developing Future High Integrity Processing Solutions
 
Taming event-driven software via formal verification
Taming event-driven software via formal verificationTaming event-driven software via formal verification
Taming event-driven software via formal verification
 
Pushing the Boundary of Mostly Automatic Program Proof
Pushing the Boundary of Mostly Automatic Program ProofPushing the Boundary of Mostly Automatic Program Proof
Pushing the Boundary of Mostly Automatic Program Proof
 
RCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standardsRCA OCORA: Safe Computing Platform using open standards
RCA OCORA: Safe Computing Platform using open standards
 
Product Lines and Ecosystems: from customization to configuration
Product Lines and Ecosystems: from customization to configurationProduct Lines and Ecosystems: from customization to configuration
Product Lines and Ecosystems: from customization to configuration
 
Securing the Future of Safety and Security of Embedded Software
Securing the Future of Safety and Security of Embedded SoftwareSecuring the Future of Safety and Security of Embedded Software
Securing the Future of Safety and Security of Embedded Software
 
Spark / Ada for Safe and Secure Firmware Development
Spark / Ada for Safe and Secure Firmware DevelopmentSpark / Ada for Safe and Secure Firmware Development
Spark / Ada for Safe and Secure Firmware Development
 
Introducing the HICLASS Research Programme - Enabling Development of Complex ...
Introducing the HICLASS Research Programme - Enabling Development of Complex ...Introducing the HICLASS Research Programme - Enabling Development of Complex ...
Introducing the HICLASS Research Programme - Enabling Development of Complex ...
 
The Future of Aerospace – More Software Please!
The Future of Aerospace – More Software Please!The Future of Aerospace – More Software Please!
The Future of Aerospace – More Software Please!
 
Adaptive AUTOSAR - The New AUTOSAR Architecture
Adaptive AUTOSAR - The New AUTOSAR ArchitectureAdaptive AUTOSAR - The New AUTOSAR Architecture
Adaptive AUTOSAR - The New AUTOSAR Architecture
 
Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...
Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...
Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of...
 
Software Engineering for Robotics - The RoboStar Technology
Software Engineering for Robotics - The RoboStar TechnologySoftware Engineering for Robotics - The RoboStar Technology
Software Engineering for Robotics - The RoboStar Technology
 
MISRA C in an ISO 26262 context
MISRA C in an ISO 26262 contextMISRA C in an ISO 26262 context
MISRA C in an ISO 26262 context
 
Application of theorem proving for safety-critical vehicle software
Application of theorem proving for safety-critical vehicle softwareApplication of theorem proving for safety-critical vehicle software
Application of theorem proving for safety-critical vehicle software
 
The Application of Formal Methods to Railway Signalling Software
The Application of Formal Methods to Railway Signalling SoftwareThe Application of Formal Methods to Railway Signalling Software
The Application of Formal Methods to Railway Signalling Software
 
Bounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise EnvironmentBounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise Environment
 

Recently uploaded

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Neo4j
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Nexer Digital
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
Aftab Hussain
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Alex Pruden
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 

Recently uploaded (20)

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 

Challenges of Agile Qualification

  • 2. How does it feel to qualify a tool for DO-178?
  • 3.
  • 4. The Big Freeze problem Is Tool Qualification equivalent to Baselining? Verify a slightly different coding standard Change the code generation strategy Regenerate a source-to-obj traceability study
  • 5. The Big Freeze problem (II) A new bug is found! Please, do NOT fix it! Really cheaper/safer to invent workarounds?
  • 6. The Big Freeze problem (III) So, what is our goal? AdaCore engineers improve our tech every day Constraining engineers’ creativity cannot work …still, we target the high-integrity domain QUALIFICATION
  • 7. How to Avoid the Big Freeze Problem (I) Continuous Qualification Mirrors the “Continuous Integration” concept Qualification artifacts evolve along source code A tool shall always be in a “semi-qualifiable” state Maximize automation of qualification-oriented activities
  • 8. How to Avoid the Big Freeze Problem (II) Delta Qualification What is the minimum effort to re-achieve qualifiability? Which artifact shall be (re-)produced Which activities shall be (re-)performed
  • 9. An example of Continuous Certification Requirement R1 Test Case TC1 Source File SF1 Test Case TC2 Test Case TC3 Test Procedure TP1 Test Procedure TP2 Test Procedure TP3 Structural Coverage Report SCR1 Coding Standard CS1 Coding Standard Compliance Report CSC1 Modified Re-generated Re-generated A modification of an artifact triggers the qualification infrastructure to execute a set of activities (possibly overnightly)
  • 10. Continuous Certification + Delta Certification Sounds nice eh? Requires an integrated certification infrastructure  The topic of this talk! Focus on automation & optimization Difficult to generalize for different teams/clients A drastically different way of working
  • 11. The core problem Establishing a dependency & traceability graph To use to trigger automated activities To calculate the Delta Qualification effort Low creation & maintenance effort Understand when an artifact has changed Propagate un-valid status
  • 12. Presenting current results… …as well as un-implemented ideas A good vision of our way of thinking… …and how we develop TQL5 kits From now on, this is a long-term view
  • 13. Introducing the Qualifying Machine An intelligent repository for qualification artifacts Integrated with build/test infrastructure Provides hooks to invoke scripts to trigger activities Support for change impact analysis
  • 14. QM back-end Qualifying Machine Architecture (I) SVN, Git, … Repository Traceability tracking Dependency tracking Action tracking Un-validity propagation Automated activity triggering QM front-end Development EngineerProcess Engineer Project Manager Certification Authority Testing/Building infrastructure
  • 15. Qualifying Machine Architecture (II) Major design decisions & challenges What to track (artifacts, activities, …) How to build a traceability & dependency graph How to understand if an artifact has changed Un-validity propagation algorithm
  • 16. What do we track? Everything is an artifact! Each artifact is a (set of) file(s) Metadata are placed in a single folder Activities ALWAYS produce qualification artifacts An activity not producing an artifact IS NOT WORTH IT State of an activity = state of the artifacts it produces Leverage as much as possible on naming conventions
  • 17. Configuring the QM project <NAME> is artifact Requirement is files: “req.txt” activity Decompose produces derived : Requirement [0..*] in “./Req_<name>” activity Write_Test_Case produces test_cases : Test_Case [1..*] in “./TC_<name>” activity QA_Review produces qa_report : Report [0..1] in “qa.txt” traceability: standard : Doc [1] in “<ROOT>/Standards/req_std.txt”
  • 18. Building a traceability & dependency graph My_Project Req_SysReq1 Req_SwReq1 TC_Flying TC_Departure QA Req_SysReq1 Req_SwReq1 Configuration File project <NAME> is artifact Requirement is files: “req.txt” activity Decompose produces derived : Requirement [0..*] in “./Req_<name>” activity Write_Test_Case produces test_cases : Test_Case [1..*] in “./TC_<name>” + =
  • 19. When does an artifact change? At the heart of un-validity propagation Timestamp, SVN diff, MD5, … How to cope with complex artifacts (UML or Simulink models)? How to identify changes not requiring formal re-validation? Example: a typo
  • 20. TC1 6.3.2bcd Un-validity propagation algorithm (II) LLR1 6.4.4.1 Re-validate 6.4.4.1 (test coverage of LLR), possibly editing TC1 Re-measure structural coverage TP1 Source File SF1 Manual activity Depending on the status of artifacts & activities, the QM: - Proposes a minimal list of manual activities - Triggers automated activities - Track each single action of the user LLRs are accurate, consistent, compatible and verifiable
  • 21. TORs Un-validity propagation algorithm (I) Known Constraints and Open Problems References, but does not depends on Tool Qualification Plan Depends on Verification Results Analysis Delivery File Modifying the Known Constraints and Open Problems triggers the production of a minimal set of manual activities to be performed in order to re-achieve qualification: - on TORS - on elements derived from TORs
  • 22. How do we use the QM? (I) Not targeting the typical DOORS users…
  • 24. Un-validity propagation feedback How do we use the QM? (III)
  • 25. How do we fill the QM repository? (IV) However, we understand other tools are used Develop a set of plug-in Import artifact metadata in the QM The QM focuses on continuous/delta qualification Integrate external dev. tools in the QM
  • 26. Wrap-up Our experience From a semi-agile infrastructure to the QM Qualification kit for a verification tool (TQL5): can we scale? Lessons learnt Fragmentation & Automation! What about workflow modeling/tracking? Future developments www.forge.open-do.org/projects/qmachine SafeCer Project (Artemis)