Matteo Bordin
bordin@adacore.com
How does it feel to qualify a tool for DO-178?
The Big Freeze problem
Is Tool Qualification equivalent to Baselining?
Verify a slightly different coding standard
Change ...
The Big Freeze problem (II)
A new bug is found!
Please, do NOT fix it!
Really cheaper/safer to invent workarounds?
The Big Freeze problem (III)
So, what is our goal?
AdaCore engineers improve our tech every day
Constraining engineers’ cr...
How to Avoid the Big Freeze Problem (I)
Continuous Qualification
Mirrors the “Continuous Integration” concept
Qualificatio...
How to Avoid the Big Freeze Problem (II)
Delta Qualification
What is the minimum effort to re-achieve qualifiability?
Whic...
An example of Continuous Certification
Requirement R1
Test Case TC1
Source File SF1
Test Case TC2
Test Case TC3
Test Proce...
Continuous Certification + Delta Certification
Sounds nice eh?
Requires an integrated certification infrastructure
 The t...
The core problem
Establishing a dependency & traceability graph
To use to trigger automated activities
To calculate the De...
Presenting current results…
…as well as un-implemented ideas
A good vision of our way of thinking…
…and how we develop TQL...
Introducing the Qualifying Machine
An intelligent repository for qualification artifacts
Integrated with build/test infras...
QM back-end
Qualifying Machine Architecture (I)
SVN, Git, …
Repository
Traceability tracking
Dependency tracking
Action tr...
Qualifying Machine Architecture (II)
Major design decisions & challenges
What to track (artifacts, activities, …)
How to b...
What do we track?
Everything is an artifact!
Each artifact is a (set of) file(s)
Metadata are placed in a single folder
Ac...
Configuring the QM
project <NAME> is
artifact Requirement is
files: “req.txt”
activity Decompose produces
derived : Requir...
Building a traceability & dependency graph
My_Project
Req_SysReq1
Req_SwReq1
TC_Flying
TC_Departure
QA
Req_SysReq1
Req_SwR...
When does an artifact change?
At the heart of un-validity propagation
Timestamp, SVN diff, MD5, …
How to cope with complex...
TC1
6.3.2bcd
Un-validity propagation algorithm (II)
LLR1
6.4.4.1
Re-validate 6.4.4.1
(test coverage of LLR),
possibly edit...
TORs
Un-validity propagation algorithm (I)
Known Constraints and
Open Problems
References, but does
not depends on
Tool Qu...
How do we use the QM? (I)
Not targeting the typical DOORS users…
Lightweight artifact editing/managing
How do we use the QM? (II)
Un-validity propagation feedback
How do we use the QM? (III)
How do we fill the QM repository? (IV)
However, we understand other tools are used
Develop a set of plug-in
Import artifac...
Wrap-up
Our experience
From a semi-agile infrastructure to the QM
Qualification kit for a verification tool (TQL5): can we...
Presented by
Challenges of Agile Qualification
Upcoming SlideShare
Loading in...5
×

Challenges of Agile Qualification

1,460

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,460
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Challenges of Agile Qualification

  1. 1. Matteo Bordin bordin@adacore.com
  2. 2. How does it feel to qualify a tool for DO-178?
  3. 3. The Big Freeze problem Is Tool Qualification equivalent to Baselining? Verify a slightly different coding standard Change the code generation strategy Regenerate a source-to-obj traceability study
  4. 4. The Big Freeze problem (II) A new bug is found! Please, do NOT fix it! Really cheaper/safer to invent workarounds?
  5. 5. The Big Freeze problem (III) So, what is our goal? AdaCore engineers improve our tech every day Constraining engineers’ creativity cannot work …still, we target the high-integrity domain QUALIFICATION
  6. 6. How to Avoid the Big Freeze Problem (I) Continuous Qualification Mirrors the “Continuous Integration” concept Qualification artifacts evolve along source code A tool shall always be in a “semi-qualifiable” state Maximize automation of qualification-oriented activities
  7. 7. How to Avoid the Big Freeze Problem (II) Delta Qualification What is the minimum effort to re-achieve qualifiability? Which artifact shall be (re-)produced Which activities shall be (re-)performed
  8. 8. An example of Continuous Certification Requirement R1 Test Case TC1 Source File SF1 Test Case TC2 Test Case TC3 Test Procedure TP1 Test Procedure TP2 Test Procedure TP3 Structural Coverage Report SCR1 Coding Standard CS1 Coding Standard Compliance Report CSC1 Modified Re-generated Re-generated A modification of an artifact triggers the qualification infrastructure to execute a set of activities (possibly overnightly)
  9. 9. Continuous Certification + Delta Certification Sounds nice eh? Requires an integrated certification infrastructure  The topic of this talk! Focus on automation & optimization Difficult to generalize for different teams/clients A drastically different way of working
  10. 10. The core problem Establishing a dependency & traceability graph To use to trigger automated activities To calculate the Delta Qualification effort Low creation & maintenance effort Understand when an artifact has changed Propagate un-valid status
  11. 11. Presenting current results… …as well as un-implemented ideas A good vision of our way of thinking… …and how we develop TQL5 kits From now on, this is a long-term view
  12. 12. Introducing the Qualifying Machine An intelligent repository for qualification artifacts Integrated with build/test infrastructure Provides hooks to invoke scripts to trigger activities Support for change impact analysis
  13. 13. QM back-end Qualifying Machine Architecture (I) SVN, Git, … Repository Traceability tracking Dependency tracking Action tracking Un-validity propagation Automated activity triggering QM front-end Development EngineerProcess Engineer Project Manager Certification Authority Testing/Building infrastructure
  14. 14. Qualifying Machine Architecture (II) Major design decisions & challenges What to track (artifacts, activities, …) How to build a traceability & dependency graph How to understand if an artifact has changed Un-validity propagation algorithm
  15. 15. What do we track? Everything is an artifact! Each artifact is a (set of) file(s) Metadata are placed in a single folder Activities ALWAYS produce qualification artifacts An activity not producing an artifact IS NOT WORTH IT State of an activity = state of the artifacts it produces Leverage as much as possible on naming conventions
  16. 16. Configuring the QM project <NAME> is artifact Requirement is files: “req.txt” activity Decompose produces derived : Requirement [0..*] in “./Req_<name>” activity Write_Test_Case produces test_cases : Test_Case [1..*] in “./TC_<name>” activity QA_Review produces qa_report : Report [0..1] in “qa.txt” traceability: standard : Doc [1] in “<ROOT>/Standards/req_std.txt”
  17. 17. Building a traceability & dependency graph My_Project Req_SysReq1 Req_SwReq1 TC_Flying TC_Departure QA Req_SysReq1 Req_SwReq1 Configuration File project <NAME> is artifact Requirement is files: “req.txt” activity Decompose produces derived : Requirement [0..*] in “./Req_<name>” activity Write_Test_Case produces test_cases : Test_Case [1..*] in “./TC_<name>” + =
  18. 18. When does an artifact change? At the heart of un-validity propagation Timestamp, SVN diff, MD5, … How to cope with complex artifacts (UML or Simulink models)? How to identify changes not requiring formal re-validation? Example: a typo
  19. 19. TC1 6.3.2bcd Un-validity propagation algorithm (II) LLR1 6.4.4.1 Re-validate 6.4.4.1 (test coverage of LLR), possibly editing TC1 Re-measure structural coverage TP1 Source File SF1 Manual activity Depending on the status of artifacts & activities, the QM: - Proposes a minimal list of manual activities - Triggers automated activities - Track each single action of the user LLRs are accurate, consistent, compatible and verifiable
  20. 20. TORs Un-validity propagation algorithm (I) Known Constraints and Open Problems References, but does not depends on Tool Qualification Plan Depends on Verification Results Analysis Delivery File Modifying the Known Constraints and Open Problems triggers the production of a minimal set of manual activities to be performed in order to re-achieve qualification: - on TORS - on elements derived from TORs
  21. 21. How do we use the QM? (I) Not targeting the typical DOORS users…
  22. 22. Lightweight artifact editing/managing How do we use the QM? (II)
  23. 23. Un-validity propagation feedback How do we use the QM? (III)
  24. 24. How do we fill the QM repository? (IV) However, we understand other tools are used Develop a set of plug-in Import artifact metadata in the QM The QM focuses on continuous/delta qualification Integrate external dev. tools in the QM
  25. 25. Wrap-up Our experience From a semi-agile infrastructure to the QM Qualification kit for a verification tool (TQL5): can we scale? Lessons learnt Fragmentation & Automation! What about workflow modeling/tracking? Future developments www.forge.open-do.org/projects/qmachine SafeCer Project (Artemis)
  26. 26. Presented by
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×