Theoretical foundation of sourcecoverage analysis from execution traces! Thomas Quinot!
Summary!• Original Needs & Goals!• Challenges along the way!• Main Results!
Original Needs!• Structural Coverage Analysis is required by certiﬁcation standards:! • Open source Coverage Tools exist but are not usable in a HI context! • Proprietary Tools exist but do not support all versions of Ada!• Complete the GNAT Pro Toolset for the High Integrity Market!• Better support for the rapidly evolving versions of Ada (83 … 95 … 2005 … 2012 …)!
Original Goals!• Provide an High Quality Open Source alternative to existing proprietary tools!• Provide Support for Agile/Lean Development! • In particular: Continuous Integration/Certiﬁcation! • Open-DO initiative!• Find the best compromise between Source and Object Coverage!
The Couverture Project (2008-2010)!• One of the ﬁrst FUI projects from the GTLL at System@atic!• 4 partners (AdaCore, Openwide, Telecom PT, Paris 6)!• Effort of 160 man-month (2,23 M€) over 2 years • 45% Financed by the city of Paris, IdF region, DGE ! This project gave us the capability to meet the unexpected challenges we were facing.
Object Coverage vs Source Coverage! • Big debate in the Certiﬁcation Community! • Which one is the most Accurate / Appropriate ?! • Which one is the most efﬁcient ?! Source Object - Statement/Decision are source concepts - on ﬁnal code (no instrumentation) - usually works by instrumenting the code - on ﬁnal hardware - can be done on fast native platforms - not language speciﬁc - requires double testing strategy - more precise
Object Coverage vs Source Coverage!• Object coverage metrics:! • Instruction Coverage! • Object Branch Coverage (OBC)!• Source coverage metrics:! • Statement Coverage! • Decision Coverage (DC)! • Modiﬁed Condition/Decision Coverage (MC/DC) Independent inﬂuence of each condition within a decision!
Challenge 1!!It is difﬁcult to provide accurate source coverage info from execution traces:!! !- no trace of “statement” / “condition” / “decision” at ! binary level!! !- optimization can change signiﬁcantly the control ﬂow
Accurate Source Coverage Info! Not sufﬁcient to locate precise statements, decisions, or conditions boundaries Sources Sources Sources Sources GNAT Pro Executable Debug info Exec decorated Exec decorated traces Exec sources decorated traces sources GNATcoverage traces sources
Accurate Source Coverage Info! Source Coverage Information (Static analysis) -fpreserve-control-ﬂow Executable Sources Sources Debug info Sources Enhanced Sources GNAT Pro SCOs Exec decorated Exec decorated traces Exec sources decorated traces sources GNATcoverage traces sources
Challenge 2! OBC does not imply MC/DC!We need better theoretical foundations !
Initial ideas!• General Belief at beginning of project :! Object Coverage => Statement Coverage! Object Branch Coverage => Decision Coverage! Object Branch Coverage => MC/DC (when using short circuit operators)!• But a FAA study arrived after the beginning showing unexplainable differences between OBC and MC/DC DOT/FAA/AR-07/17!
Elementary counter-example!function P (A, B, C : Boolean) return Boolean isbegin Conditions if ( A and then B ) or else C then return True; end if;end P; Decision OBC MC / DC A B C if statement A B C if statement T T F T T T x T A F T F F T F T T C B F T T T F x F F T F F F 3 tests are sufﬁcient At least n+1 tests n = number of conditions
Counter-measures!• Deﬁnition of a formal model to express coverage metrics based on BDD (Binary Decision Diagram)!• Express OBC and MC/DC in this model! Use • Find counter-examples! Open Source Model Checker • Find precise perimeter where the Alloy equivalence can be proven!• Formally prove this result!
Evaluation of short circuit boolean expressions! • Evaluating a short circuit decision is a traversal of its Reduced Ordered Binary Decision Diagram! • Each ROBDD node is a test for a condition! • Evaluate conditions left to right! • Do not evaluate RHS if LHS is sufﬁcient! • A condition vector denotes a path trough the ROBDD! A T F ( A and then B ) or else C F B C T T F T T F
More counter-examples (Alloy)! A function P (A, B, C : Boolean) return Boolean is begin T F if ( A and then B ) or else C then return True; B F end if; end P; T C T F T F BDD C0 T F function P (C0, C1, C2, C3, C4 … : Boolean) return Boolean is C1 begin C2 F if ((((…(C0 and then C1) or else C2) and then C3) or else C4 … T T F then return True; C3 end if; F C4 end P; T F T
Pathological case!(((C0 AND THEN C1) OR ELSE C2) AND THEN C3) OR ELSE C4... • N conditions • MC/DC requires at least N + 1 tests • OBC can be achieved in 3 tests! C0 T C1 F T F C2 T C3 F T F C4 T F T T F 3 tests sufﬁcient instead of N+1, for any N
What does that mean?!• For a given test campaign! • OBC (BDDBC) are local properties of each BDD node: stateless (union of all paths are covering the BDD)! • MC/DC is a property of trajectories taken through the ROBDD: stateful (all paths through the BDD are taken)!• In general MC/DC requires complete history of each conditional branch instruction (each BDD node)!• Are there speciﬁc cases where we can do better?!
Equivalence can be proven when !• There are no diamonds in the BDD (nodes that can be reached through multiple paths)!• How does this translate in “User Terms” ?!• No easy formulation… the best we found is! • Transform Boolean expression in “Negative Normal Form”! • No “and then” in left operand of a “or else”! • No “or else” in left operand of a “and then”!
Proof sketch!• In the no-diamond case, each path covers a distinct terminal edge ⇒ all terminal edges covered implies all paths covered, MC/DC is achieved!• If thereʼs a diamond, we construct a covering path set that fails to show independent inﬂuence of one condition (all paths through that condition have the same outcome)! A T F B F T C T F T F
Main Results!• Emulation is key to Agile cross development!• GNATcoverage takes advantage of the theoretical results to:! • Implement properly MC/DC in the complex case! • Optimize the simple case by using OBC!• Deﬁnition of speciﬁc compilation artefacts (SCOs) and of a certiﬁcation-friendly code generation mode in GCC (-fpreserve-control-ﬂow)!• Creation of “open source” qualiﬁcation material as part of Open-DO!
Conclusion!• The Couverture project allowed us to concentrate on solving properly the unexpected challenges!• Existing Open-Source technologies have played a key role:! • Qemu is the base of GNATemulator! • Alloy helped a lot for the mathematical proofs!• As a result, new industrial-ready Open Source tools are now available for the HI developersʼ community !