Integrating Proof and Testing in Verification Strategies for Safety Critical Systems


Published on

This talk was given by Cyrille Comar at the recent SPARK User Group. This talk reviews the prominent place and role testing holds in Safety Standards. It compares the strengths and weaknesses of testing with an alternative verification technique based on formal methods. It then explores specific instances where a combination of both approaches makes sense and can bring significant cost savings, without forcing dramatic changes in internal development procedures.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Integrating Proof and Testing in Verification Strategies for Safety Critical Systems

  1. 1. Integrating Proof &Testing in Verification Strategies Cyrille Comar
  2. 2. Agenda•  Place of Testing in Safety Standards•  Programming with executable Contracts•  Executable Contracts vs Formal Contracts•  Reducing Testing with Formal Proofs
  3. 3. Place of Testing in DO-178DO-178 define mainly 6 sets of objectives related to•  Planning (7)•  Development (7)•  Verification (43)•  Configuration Management (6) ü  Various reviews & analysis•  Quality Assurance (5) ü  Requirements-Based Testing•  Certification Liaison (3) SLR HLR LLR
  4. 4. Requirements in DO-178 •  There are 3 levels requirements •  System Requirements (SLR) •  High-Level Requirements (HLR) •  Low-Level Requirements (LLR) LLRs SLRs HLRs Allocated to Software Software Architecture
  5. 5. Requirements- Based (RB) Testing in DO-178 RB Test Generation LLR HLR Hardware / Low-Level Software Software Tests Integration Integration Tests Tests RBTest Coverage Analysis Statements Structural Decisions Coverage Analysis MCDC
  6. 6. Robustness vs functional test cases in DO-178 •  2 kinds of test cases •  Normal Range Test Cases •  Robustness Test Cases normal    “  …  ability  of  the  software  to  respond  to                              inputs  and  conditions  …  ”   abnormal   From outside From inside (response should be specified in SLR/HLR) (Software error) Integrated software comp1 comp2 comp3
  7. 7. Low-Level Testing is long & difficult•  Is it as relevant as HLR based testing ?•  LLR verification is important but is “testing” the most appropriate one? subp2   subp1   subp5   subp3   subp4  Test subp6  harness Subp3 Subp4 stub
  8. 8. Programming by Contract •  Precondition     •  Postcondition    •  A CONTRACT is •  Invariant     •  Predicate     •  A logical property •  Assertion   •  subprogram   •  type   •  Associated with a program entity •  statement   •  It can be verified •  It can be relied upon
  9. 9. Programming By Contract ( Ada examples) procedure Square_Root (Input : Integer; Res : out Natural) with pre => Input >= 0, post => Res * Res <= Input and (Res+1)*(Res+1) > Input; Implicit contract Explicit contract type Positive_Stack is private with Invariant => Is_Empty (Stack) or else Top (Stack) > 0; subtype Odd_Positive is Natural with Predicate => (Odd_Positive/2)*2 /= Odd_Positive;
  10. 10. What can Contracts be used for?•  Adding user-defined dynamic checks at runtime•  A way of expressing constraints, properties LLRs “When a low level requirement can be expressed as a contract, it can be tested much more easily”
  11. 11. Robustness Testing & Contracts• Internal Robustness issues are related to Run-Time Errors• Ada guarantees (most cases of) RTEs è run-time exception• Preconditions are better than “defensive coding”• The more runtime checks the better Robustness Testing is
  12. 12. Requirement Testing & Contracts•  LLR is expressed as a subprogram contract•  Successful execution of Postcondition è Test successful•  No need for collecting & verifying output
  13. 13. Executable Contracts vs Formal Contracts procedure  Square_Root     procedure  Square_Root        (Input  :  Integer;  Res  :  out  Natural)      (Input  :  Integer;  Res  :  out  Natural)   with     -­‐-­‐#  pre    Input  >=  0;      pre    =>  Input  >=  0,   -­‐-­‐#  post        Res      *  Res        <=  Input        post  =>    Res      *  Res        <=  Input     -­‐-­‐#        and  (Res+1)*(Res+1)  >    Input;                and  (Res+1)*(Res+1)  >    Input;           (Ada  2012)   (SPARK  2005)  •  Execution of “executable” contracts can be disabled•  The syntactic difference is not relevant•  The same contracts can be interpreted in 2 different world Executable boolean expression First order logic formula
  14. 14. Requirement based Verification & Formal Contracts•  LLR is expressed as a subprogram formal contract•  Successful proof of Postcondition è LLR verified for any input•  Approach allowed by DO-333•  Proof of absence of RTEs provides for some Robustness verification
  15. 15. Hybrid Verification (1)•  Some LLRs are tested•  Some LLRs are proved•  Is the combination as “strong” as all LLRs are tested?
  16. 16. Hybrid Verification (2)Modular Formal verification is based on assumptions: •  My Precondition is never violated by Callers •  Postconditions of callees are true after the call •  all outputs are known •  Direct or Indirect •  Explicit (out parameters) or Implicit (globals) •  No surprises (e.g. unexpected aliasing)
  17. 17. Hybrid Verification (3) Tested_Function   calls Proved_Function   Precondition must be executed Proved_Function   calls Tested_Function   Postcondition must be executed•  Executable contracts “protect” assumptions of formal verification•  Globals can be computed by static analysis•  Surprises can be eliminated by subsetting
  18. 18. Conclusion: Hybrid Verification Benefits•  Helps with gradual introduction of Formal proofs•  The traditional 80 / 20 % rule is true for both formal verification and testing 80% remaining easily tested 80% easily proved
  19. 19. Extra information•  Open-DO Initiative • •
  20. 20. Cyrille Comar +33-1-49-70-67-16comar@adacore.com46 Rue d’Amsterdam Paris 75009