Integrating Proof and Testing in Verification Strategies for Safety Critical Systems

  • 874 views
Uploaded on

This talk was given by Cyrille Comar at the recent SPARK User Group. This talk reviews the prominent place and role testing holds in Safety Standards. It compares the strengths and weaknesses of …

This talk was given by Cyrille Comar at the recent SPARK User Group. This talk reviews the prominent place and role testing holds in Safety Standards. It compares the strengths and weaknesses of testing with an alternative verification technique based on formal methods. It then explores specific instances where a combination of both approaches makes sense and can bring significant cost savings, without forcing dramatic changes in internal development procedures.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
874
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
6
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Integrating Proof &Testing in Verification Strategies Cyrille Comar
  • 2. Agenda•  Place of Testing in Safety Standards•  Programming with executable Contracts•  Executable Contracts vs Formal Contracts•  Reducing Testing with Formal Proofs
  • 3. Place of Testing in DO-178DO-178 define mainly 6 sets of objectives related to•  Planning (7)•  Development (7)•  Verification (43)•  Configuration Management (6) ü  Various reviews & analysis•  Quality Assurance (5) ü  Requirements-Based Testing•  Certification Liaison (3) SLR HLR LLR
  • 4. Requirements in DO-178 •  There are 3 levels requirements •  System Requirements (SLR) •  High-Level Requirements (HLR) •  Low-Level Requirements (LLR) LLRs SLRs HLRs Allocated to Software Software Architecture
  • 5. Requirements- Based (RB) Testing in DO-178 RB Test Generation LLR HLR Hardware / Low-Level Software Software Tests Integration Integration Tests Tests RBTest Coverage Analysis Statements Structural Decisions Coverage Analysis MCDC
  • 6. Robustness vs functional test cases in DO-178 •  2 kinds of test cases •  Normal Range Test Cases •  Robustness Test Cases normal    “  …  ability  of  the  software  to  respond  to                              inputs  and  conditions  …  ”   abnormal   From outside From inside (response should be specified in SLR/HLR) (Software error) Integrated software comp1 comp2 comp3
  • 7. Low-Level Testing is long & difficult•  Is it as relevant as HLR based testing ?•  LLR verification is important but is “testing” the most appropriate one? subp2   subp1   subp5   subp3   subp4  Test subp6  harness Subp3 Subp4 stub
  • 8. Programming by Contract •  Precondition     •  Postcondition    •  A CONTRACT is •  Invariant     •  Predicate     •  A logical property •  Assertion   •  subprogram   •  type   •  Associated with a program entity •  statement   •  It can be verified •  It can be relied upon
  • 9. Programming By Contract ( Ada examples) procedure Square_Root (Input : Integer; Res : out Natural) with pre => Input >= 0, post => Res * Res <= Input and (Res+1)*(Res+1) > Input; Implicit contract Explicit contract type Positive_Stack is private with Invariant => Is_Empty (Stack) or else Top (Stack) > 0; subtype Odd_Positive is Natural with Predicate => (Odd_Positive/2)*2 /= Odd_Positive;
  • 10. What can Contracts be used for?•  Adding user-defined dynamic checks at runtime•  A way of expressing constraints, properties LLRs “When a low level requirement can be expressed as a contract, it can be tested much more easily”
  • 11. Robustness Testing & Contracts• Internal Robustness issues are related to Run-Time Errors• Ada guarantees (most cases of) RTEs è run-time exception• Preconditions are better than “defensive coding”• The more runtime checks the better Robustness Testing is
  • 12. Requirement Testing & Contracts•  LLR is expressed as a subprogram contract•  Successful execution of Postcondition è Test successful•  No need for collecting & verifying output
  • 13. Executable Contracts vs Formal Contracts procedure  Square_Root     procedure  Square_Root        (Input  :  Integer;  Res  :  out  Natural)      (Input  :  Integer;  Res  :  out  Natural)   with     -­‐-­‐#  pre    Input  >=  0;      pre    =>  Input  >=  0,   -­‐-­‐#  post        Res      *  Res        <=  Input        post  =>    Res      *  Res        <=  Input     -­‐-­‐#        and  (Res+1)*(Res+1)  >    Input;                and  (Res+1)*(Res+1)  >    Input;           (Ada  2012)   (SPARK  2005)  •  Execution of “executable” contracts can be disabled•  The syntactic difference is not relevant•  The same contracts can be interpreted in 2 different world Executable boolean expression First order logic formula
  • 14. Requirement based Verification & Formal Contracts•  LLR is expressed as a subprogram formal contract•  Successful proof of Postcondition è LLR verified for any input•  Approach allowed by DO-333•  Proof of absence of RTEs provides for some Robustness verification
  • 15. Hybrid Verification (1)•  Some LLRs are tested•  Some LLRs are proved•  Is the combination as “strong” as all LLRs are tested?
  • 16. Hybrid Verification (2)Modular Formal verification is based on assumptions: •  My Precondition is never violated by Callers •  Postconditions of callees are true after the call •  all outputs are known •  Direct or Indirect •  Explicit (out parameters) or Implicit (globals) •  No surprises (e.g. unexpected aliasing)
  • 17. Hybrid Verification (3) Tested_Function   calls Proved_Function   Precondition must be executed Proved_Function   calls Tested_Function   Postcondition must be executed•  Executable contracts “protect” assumptions of formal verification•  Globals can be computed by static analysis•  Surprises can be eliminated by subsetting
  • 18. Conclusion: Hybrid Verification Benefits•  Helps with gradual introduction of Formal proofs•  The traditional 80 / 20 % rule is true for both formal verification and testing 80% remaining easily tested 80% easily proved
  • 19. Extra information•  Open-DO Initiative •  www.open-do.org •  http://www.open-do.org/projects/hi-lite/
  • 20. Cyrille Comar +33-1-49-70-67-16comar@adacore.com46 Rue d’Amsterdam Paris 75009