Your SlideShare is downloading. ×
Integrating Proof &Testing in Verification      Strategies                 Cyrille Comar
Place of Testing in DO-178DO-178 define mainly 6 sets of objectives related to•  Planning (7)•  Development (7)•  Verifica...
Requirements in DO-178 •  There are 3 levels requirements               •  System Requirements (SLR)               •  High...
Requirements- Based (RB) Testing in DO-178                            RB                      Test Generation             ...
Robustness vs functional test cases in DO-178                    •  2 kinds of test cases                                 ...
Low-Level Testing is long & difficult•  Is it as relevant as HLR based testing ?•  LLR verification is important but is “t...
Programming by Contract                                •  Precondition	  	                                  •  Postconditi...
Programming By Contract ( Ada examples)           procedure Square_Root (Input : Integer; Res : out Natural)           wit...
What can Contracts be used for?•  Adding user-defined dynamic checks at runtime•  A way of expressing constraints, propert...
Robustness Testing & Contracts• Internal Robustness issues are related to Run-Time Errors• Ada guarantees (most cases of) ...
Requirement Testing & Contracts•  LLR is expressed as a subprogram contract•  Successful execution of Postcondition è Tes...
Executable Contracts vs Formal Contracts   procedure	  Square_Root	  	                                                    ...
Requirement based Verification     & Formal Contracts•  LLR is expressed as a subprogram formal contract•  Successful proo...
Hybrid Verification (1)•  Some LLRs are tested•  Some LLRs are proved•  Is the combination as “strong” as all LLRs are tes...
Hybrid Verification (2)Modular Formal verification is based on assumptions:  •  My Precondition is never violated by Calle...
Hybrid Verification (3)            Tested_Function	               calls           Proved_Function	                        ...
Conclusion: Hybrid Verification Benefits•  Helps with gradual introduction of Formal proofs•  The traditional 80 / 20 % ru...
Extra information•  Open-DO Initiative  •  www.open-do.org  •  http://www.open-do.org/projects/hi-lite/
Cyrille Comar +33-1-49-70-67-16comar@adacore.com46 Rue d’Amsterdam         Paris       75009
Integrating Proof and Testing in Verification Strategies for Safety Critical Systems
Upcoming SlideShare
Loading in...5
×

Integrating Proof and Testing in Verification Strategies for Safety Critical Systems

976

Published on

This talk was given by Cyrille Comar at the recent SPARK User Group. This talk reviews the prominent place and role testing holds in Safety Standards. It compares the strengths and weaknesses of testing with an alternative verification technique based on formal methods. It then explores specific instances where a combination of both approaches makes sense and can bring significant cost savings, without forcing dramatic changes in internal development procedures.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
976
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Integrating Proof and Testing in Verification Strategies for Safety Critical Systems"

  1. 1. Integrating Proof &Testing in Verification Strategies Cyrille Comar
  2. 2. Agenda•  Place of Testing in Safety Standards•  Programming with executable Contracts•  Executable Contracts vs Formal Contracts•  Reducing Testing with Formal Proofs
  3. 3. Place of Testing in DO-178DO-178 define mainly 6 sets of objectives related to•  Planning (7)•  Development (7)•  Verification (43)•  Configuration Management (6) ü  Various reviews & analysis•  Quality Assurance (5) ü  Requirements-Based Testing•  Certification Liaison (3) SLR HLR LLR
  4. 4. Requirements in DO-178 •  There are 3 levels requirements •  System Requirements (SLR) •  High-Level Requirements (HLR) •  Low-Level Requirements (LLR) LLRs SLRs HLRs Allocated to Software Software Architecture
  5. 5. Requirements- Based (RB) Testing in DO-178 RB Test Generation LLR HLR Hardware / Low-Level Software Software Tests Integration Integration Tests Tests RBTest Coverage Analysis Statements Structural Decisions Coverage Analysis MCDC
  6. 6. Robustness vs functional test cases in DO-178 •  2 kinds of test cases •  Normal Range Test Cases •  Robustness Test Cases normal    “  …  ability  of  the  software  to  respond  to                              inputs  and  conditions  …  ”   abnormal   From outside From inside (response should be specified in SLR/HLR) (Software error) Integrated software comp1 comp2 comp3
  7. 7. Low-Level Testing is long & difficult•  Is it as relevant as HLR based testing ?•  LLR verification is important but is “testing” the most appropriate one? subp2   subp1   subp5   subp3   subp4  Test subp6  harness Subp3 Subp4 stub
  8. 8. Programming by Contract •  Precondition     •  Postcondition    •  A CONTRACT is •  Invariant     •  Predicate     •  A logical property •  Assertion   •  subprogram   •  type   •  Associated with a program entity •  statement   •  It can be verified •  It can be relied upon
  9. 9. Programming By Contract ( Ada examples) procedure Square_Root (Input : Integer; Res : out Natural) with pre => Input >= 0, post => Res * Res <= Input and (Res+1)*(Res+1) > Input; Implicit contract Explicit contract type Positive_Stack is private with Invariant => Is_Empty (Stack) or else Top (Stack) > 0; subtype Odd_Positive is Natural with Predicate => (Odd_Positive/2)*2 /= Odd_Positive;
  10. 10. What can Contracts be used for?•  Adding user-defined dynamic checks at runtime•  A way of expressing constraints, properties LLRs “When a low level requirement can be expressed as a contract, it can be tested much more easily”
  11. 11. Robustness Testing & Contracts• Internal Robustness issues are related to Run-Time Errors• Ada guarantees (most cases of) RTEs è run-time exception• Preconditions are better than “defensive coding”• The more runtime checks the better Robustness Testing is
  12. 12. Requirement Testing & Contracts•  LLR is expressed as a subprogram contract•  Successful execution of Postcondition è Test successful•  No need for collecting & verifying output
  13. 13. Executable Contracts vs Formal Contracts procedure  Square_Root     procedure  Square_Root        (Input  :  Integer;  Res  :  out  Natural)      (Input  :  Integer;  Res  :  out  Natural)   with     -­‐-­‐#  pre    Input  >=  0;      pre    =>  Input  >=  0,   -­‐-­‐#  post        Res      *  Res        <=  Input        post  =>    Res      *  Res        <=  Input     -­‐-­‐#        and  (Res+1)*(Res+1)  >    Input;                and  (Res+1)*(Res+1)  >    Input;           (Ada  2012)   (SPARK  2005)  •  Execution of “executable” contracts can be disabled•  The syntactic difference is not relevant•  The same contracts can be interpreted in 2 different world Executable boolean expression First order logic formula
  14. 14. Requirement based Verification & Formal Contracts•  LLR is expressed as a subprogram formal contract•  Successful proof of Postcondition è LLR verified for any input•  Approach allowed by DO-333•  Proof of absence of RTEs provides for some Robustness verification
  15. 15. Hybrid Verification (1)•  Some LLRs are tested•  Some LLRs are proved•  Is the combination as “strong” as all LLRs are tested?
  16. 16. Hybrid Verification (2)Modular Formal verification is based on assumptions: •  My Precondition is never violated by Callers •  Postconditions of callees are true after the call •  all outputs are known •  Direct or Indirect •  Explicit (out parameters) or Implicit (globals) •  No surprises (e.g. unexpected aliasing)
  17. 17. Hybrid Verification (3) Tested_Function   calls Proved_Function   Precondition must be executed Proved_Function   calls Tested_Function   Postcondition must be executed•  Executable contracts “protect” assumptions of formal verification•  Globals can be computed by static analysis•  Surprises can be eliminated by subsetting
  18. 18. Conclusion: Hybrid Verification Benefits•  Helps with gradual introduction of Formal proofs•  The traditional 80 / 20 % rule is true for both formal verification and testing 80% remaining easily tested 80% easily proved
  19. 19. Extra information•  Open-DO Initiative •  www.open-do.org •  http://www.open-do.org/projects/hi-lite/
  20. 20. Cyrille Comar +33-1-49-70-67-16comar@adacore.com46 Rue d’Amsterdam Paris 75009

×