Coverage analysis from execution traces


Published on

Theoretical foundation of source coverage analysis from execution traces!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Coverage analysis from execution traces

  1. 1. Theoretical foundation of sourcecoverage analysis from execution traces! Thomas Quinot!
  2. 2. Summary!•  Original Needs & Goals!•  Challenges along the way!•  Main Results!
  3. 3. Original Needs!•  Structural Coverage Analysis is required by certification standards:! •  Open source Coverage Tools exist but are not usable in a HI context! •  Proprietary Tools exist but do not support all versions of Ada!•  Complete the GNAT Pro Toolset for the High Integrity Market!•  Better support for the rapidly evolving versions of Ada (83 … 95 … 2005 … 2012 …)!
  4. 4. Original Goals!•  Provide an High Quality Open Source alternative to existing proprietary tools!•  Provide Support for Agile/Lean Development! •  In particular: Continuous Integration/Certification! •  Open-DO initiative!•  Find the best compromise between Source and Object Coverage!
  5. 5. The Couverture Project (2008-2010)!•  One of the first FUI projects from the GTLL at System@atic!•  4 partners (AdaCore, Openwide, Telecom PT, Paris 6)!•  Effort of 160 man-month (2,23 M€) over 2 years •  45% Financed by the city of Paris, IdF region, DGE ! This project gave us the capability to meet the unexpected challenges we were facing.
  6. 6. Object Coverage vs Source Coverage! •  Big debate in the Certification Community! •  Which one is the most Accurate / Appropriate ?! •  Which one is the most efficient ?! Source Object -  Statement/Decision are source concepts -  on final code (no instrumentation) -  usually works by instrumenting the code -  on final hardware -  can be done on fast native platforms -  not language specific -  requires double testing strategy -  more precise
  7. 7. Object Coverage vs Source Coverage!•  Object coverage metrics:! •  Instruction Coverage! •  Object Branch Coverage (OBC)!•  Source coverage metrics:! •  Statement Coverage! •  Decision Coverage (DC)! •  Modified Condition/Decision Coverage (MC/DC)
 Independent influence of each condition within a decision!
  8. 8. Challenge 1!!It is difficult to provide accurate source coverage info from execution traces:!! !- no trace of “statement” / “condition” / “decision” at ! binary level!! !- optimization can change significantly the control flow
  9. 9. Accurate Source Coverage Info!Sources Sources Sources Sources GNAT Pro Executable Exec decorated Exec decorated traces Exec sources decorated traces sources GNATcoverage traces sources
  10. 10. Accurate Source Coverage Info! Not sufficient to locate precise statements, decisions, or conditions boundaries Sources Sources Sources Sources GNAT Pro Executable Debug info Exec decorated Exec decorated traces Exec sources decorated traces sources GNATcoverage traces sources
  11. 11. Accurate Source Coverage Info! Source Coverage Information (Static analysis) -fpreserve-control-flow Executable Sources Sources Debug info Sources Enhanced Sources GNAT Pro SCOs Exec decorated Exec decorated traces Exec sources decorated traces sources GNATcoverage traces sources
  12. 12. Challenge 2! OBC does not imply MC/DC!We need better theoretical foundations !
  13. 13. Initial ideas!•  General Belief at beginning of project :!  Object Coverage => Statement Coverage!  Object Branch Coverage => Decision Coverage!  Object Branch Coverage => MC/DC (when using short circuit operators)!•  But a FAA study arrived after the beginning showing unexplainable differences between OBC and MC/DC DOT/FAA/AR-07/17!
  14. 14. Elementary counter-example!function P (A, B, C : Boolean) return Boolean isbegin Conditions if ( A and then B ) or else C then return True; end if;end P; Decision OBC MC / DC A B C if statement A B C if statement T T F T T T x T A F T F F T F T T C B F T T T F x F F T F F F 3 tests are sufficient At least n+1 tests n = number of conditions
  15. 15. Counter-measures!•  Definition of a formal model to express coverage metrics based on BDD (Binary Decision Diagram)!•  Express OBC and MC/DC in this model! Use •  Find counter-examples! Open Source Model Checker •  Find precise perimeter where the Alloy equivalence can be proven!•  Formally prove this result!
  16. 16. Evaluation of short circuit boolean expressions! •  Evaluating a short circuit decision is a traversal
 of its Reduced Ordered Binary Decision Diagram! •  Each ROBDD node is a test for a condition! •  Evaluate conditions left to right! •  Do not evaluate RHS if LHS is sufficient! •  A condition vector denotes a path trough the ROBDD! A T F ( A and then B ) or else C F B C T T F T T F
  17. 17. More counter-examples (Alloy)! A function P (A, B, C : Boolean) return Boolean is begin T F if ( A and then B ) or else C then return True; B F end if; end P; T C T F T F BDD C0 T F function P (C0, C1, C2, C3, C4 … : Boolean) return Boolean is C1 begin C2 F if ((((…(C0 and then C1) or else C2) and then C3) or else C4 … T T F then return True; C3 end if; F C4 end P; T F T
  18. 18. Pathological case!(((C0 AND THEN C1) OR ELSE C2) AND THEN C3) OR ELSE C4... •  N conditions •  MC/DC requires at least N + 1 tests •  OBC can be achieved in 3 tests! C0 T C1 F T F C2 T C3 F T F C4 T F T T F 3 tests sufficient instead of N+1, for any N
  19. 19. What does that mean?!•  For a given test campaign! •  OBC (BDDBC) are local properties of each BDD node: stateless (union of all paths are covering the BDD)! •  MC/DC is a property of trajectories taken through the ROBDD: stateful (all paths through the BDD are taken)!•  In general MC/DC requires complete history of each conditional branch instruction (each BDD node)!•  Are there specific cases where we can do better?!
  20. 20. Equivalence can be proven when !•  There are no diamonds in the BDD (nodes that can be reached through multiple paths)!•  How does this translate in “User Terms” ?!•  No easy formulation… the best we found is! •  Transform Boolean expression in “Negative Normal Form”! •  No “and then” in left operand of a “or else”! •  No “or else” in left operand of a “and then”!
  21. 21. Proof sketch!•  In the no-diamond case, each path covers a distinct terminal edge
 ⇒ all terminal edges covered implies all paths covered, MC/DC is achieved!•  If thereʼs a diamond, we construct a covering path set that fails to show independent influence of one condition (all paths through that condition have the same outcome)! A T F B F T C T F T F
  22. 22. Main Results!•  Emulation is key to Agile cross development!•  GNATcoverage takes advantage of the theoretical results to:! •  Implement properly MC/DC in the complex case! •  Optimize the simple case by using OBC!•  Definition of specific compilation artefacts (SCOs) and of a certification-friendly code generation mode in GCC (-fpreserve-control-flow)!•  Creation of “open source” qualification material as part of Open-DO!
  23. 23. Conclusion!•  The Couverture project allowed us to concentrate on solving properly the unexpected challenges!•  Existing Open-Source technologies have played a key role:! •  Qemu is the base of GNATemulator! •  Alloy helped a lot for the mathematical proofs!•  As a result, new industrial-ready Open Source tools are now available for the HI developersʼ community !