Couverture erts2012

Formalization and Comparison of mcdc and Object Branch Coverage CriteriaCyrille Comar, Jerome Guitton, Olivier Hainque, Thomas Quinot 1 / 17

Coverage for Level A Considering the highest level of certiﬁcation for aircraft (A in DO-178B): Test coverage goal: mcdc 2 / 17

Coverage for Level A Considering the highest level of certiﬁcation for aircraft (A in DO-178B): Test coverage goal: mcdc Structural (source) coverage 2 / 17

Coverage for Level A Considering the highest level of certiﬁcation for aircraft (A in DO-178B): Test coverage goal: mcdc Structural (source) coverage More than statement coverage... 2 / 17

Coverage for Level A Considering the highest level of certiﬁcation for aircraft (A in DO-178B): Test coverage goal: mcdc Structural (source) coverage More than statement coverage... ...but not all execution paths (too costly) 2 / 17

Coverage for Level A Considering the highest level of certiﬁcation for aircraft (A in DO-178B): Test coverage goal: mcdc Structural (source) coverage More than statement coverage... ...but not all execution paths (too costly) e.g. for a decision C1 and C2 and . . . and CN : 2 / 17

Coverage for Level A Considering the highest level of certiﬁcation for aircraft (A in DO-178B): Test coverage goal: mcdc Structural (source) coverage More than statement coverage... ...but not all execution paths (too costly) e.g. for a decision C1 and C2 and . . . and CN : 2N execution paths 2 / 17

Coverage for Level A Considering the highest level of certiﬁcation for aircraft (A in DO-178B): Test coverage goal: mcdc Structural (source) coverage More than statement coverage... ...but not all execution paths (too costly) e.g. for a decision C1 and C2 and . . . and CN : 2N execution paths N + 1 tests in mcdc 2 / 17

Some issues with mcdc source coverage criteria means language-dependent 3 / 17

Some issues with mcdc source coverage criteria means language-dependent no tool for Ada 2005 when Couverture started... 3 / 17

Some issues with mcdc source coverage criteria means language-dependent no tool for Ada 2005 when Couverture started... instrumenting source code? intrusive... 3 / 17

Some issues with mcdc source coverage criteria means language-dependent no tool for Ada 2005 when Couverture started... instrumenting source code? intrusive... unbounded execution traces 3 / 17

Some issues with mcdc source coverage criteria means language-dependent no tool for Ada 2005 when Couverture started... instrumenting source code? intrusive... unbounded execution traces Use object coverage instead? 3 / 17

Object coverage to assess mcdc Assumption that object branch coverage (obc) is stronger than mcdc 4 / 17

Object coverage to assess mcdc Assumption that object branch coverage (obc) is stronger than mcdc widespread industrial practise 4 / 17

Object coverage to assess mcdc Assumption that object branch coverage (obc) is stronger than mcdc widespread industrial practise language-independent 4 / 17

Object coverage to assess mcdc Assumption that object branch coverage (obc) is stronger than mcdc widespread industrial practise language-independent bounded traces 4 / 17

Using obc to achieve mcdc ? short-circuit operators... (A mod B = 0) and then (C = 0)

Using obc to achieve mcdc ? short-circuit operators... (A mod B = 0) and then (C = 0) A mod B = 0 F T F C =0 F T F T 5 / 17

Using obc to achieve mcdc ? short-circuit operators... A mod B = 0 B = −1 F (A mod B = 0) and then (C = 0) A 0 F A mod B = 0 B<0 T T F T F T F C =0 F T R=0 F T F T F C =0 F T F T 5 / 17

What do certiﬁcation standards say about that? DO-248C FAQ #42: Object coverage can be used as long as analysis can be provided which demonstrates that the coverage analysis conducted at the Object Code will achieve a comparable level of coverage assurance as that conducted at the Source Code level. 6 / 17

The sad truth... The assumption is wrong: obc is not stronger than mcdc! 7 / 17

The sad truth... The assumption is wrong: obc is not stronger than mcdc! A counterexample in DOT/FAA/AR-07/20, Jun 2007: Object Oriented Technology Veriﬁcation Phase 3 Report - Structural Coverage at the Source Code and Object Code Levels 7 / 17

The sad truth... The assumption is wrong: obc is not stronger than mcdc! A counterexample in DOT/FAA/AR-07/20, Jun 2007: Object Oriented Technology Veriﬁcation Phase 3 Report - Structural Coverage at the Source Code and Object Code Levels (A and then B) or else C can be covered for obc without achieving mcdc 7 / 17

Having a closer look... Alloy model: check conjectures, generate counterexamples... 8 / 17

Pathological case Alloy helped ﬁnding a impressive counterexample: 9 / 17

Pathological case Alloy helped ﬁnding a impressive counterexample: Decision with an arbitrary high number of conditions N... 9 / 17

Pathological case Alloy helped ﬁnding a impressive counterexample: Decision with an arbitrary high number of conditions N... ...that needs N+1 tests to be mcdc-covered... 9 / 17

Pathological case Alloy helped ﬁnding a impressive counterexample: Decision with an arbitrary high number of conditions N... ...that needs N+1 tests to be mcdc-covered... ...and only 3 tests to be obc-covered! 9 / 17

Pathological case Alloy helped ﬁnding a impressive counterexample: Decision with an arbitrary high number of conditions N... ...that needs N+1 tests to be mcdc-covered... ...and only 3 tests to be obc-covered! Here obc is much weaker than mcdc! 9 / 17

Now what? Sure, in some cases, obc does not imply mcdc... 10 / 17

Now what? Sure, in some cases, obc does not imply mcdc... ...and in some pathological cases, the two criteria diverges quite badly... 10 / 17

Now what? Sure, in some cases, obc does not imply mcdc... ...and in some pathological cases, the two criteria diverges quite badly... ...but past experience has shown that it works in many cases 10 / 17

Now what? Sure, in some cases, obc does not imply mcdc... ...and in some pathological cases, the two criteria diverges quite badly... ...but past experience has shown that it works in many cases ...so are there conditions that would allow this implication? 10 / 17

Now what? Sure, in some cases, obc does not imply mcdc... ...and in some pathological cases, the two criteria diverges quite badly... ...but past experience has shown that it works in many cases ...so are there conditions that would allow this implication? could be enforced by a coding standard 10 / 17

Now what? Sure, in some cases, obc does not imply mcdc... ...and in some pathological cases, the two criteria diverges quite badly... ...but past experience has shown that it works in many cases ...so are there conditions that would allow this implication? could be enforced by a coding standard could be an optimization for coverage tools 10 / 17

Cases where obc implies mcdc Theorem If there is only one execution path to each condition, then obc implies mcdc. 11 / 17

Cases where obc implies mcdc Theorem If there is only one execution path to each condition, then obc implies mcdc. C1 and then C2 11 / 17

Cases where obc implies mcdc Theorem If there is only one execution path to each condition, then obc implies mcdc. C1 and then C2 C1 F T F C2 F T F T

Cases where obc implies mcdc Theorem If there is only one execution path to each condition, then obc implies mcdc. C1 and then C2 C1 F T F C2 F T F T 11 / 17

Cases where obc does not imply mcdc Theorem On the contrary, if there exists a condition that can be reached by more than one execution path, obc does not always imply mcdc. 12 / 17

Cases where obc does not imply mcdc Theorem On the contrary, if there exists a condition that can be reached by more than one execution path, obc does not always imply mcdc. (A and then B) or else C 12 / 17

Cases where obc does not imply mcdc Theorem On the contrary, if there exists a condition that can be reached by more than one execution path, obc does not always imply mcdc. (A and then B) or else C A B T C T F

Cases where obc does not imply mcdc Theorem On the contrary, if there exists a condition that can be reached by more than one execution path, obc does not always imply mcdc. (A and then B) or else C A B T C T F 12 / 17

human-readable characteristic Dec1 or else (Dec2 and then . . . ) 13 / 17

human-readable characteristic Dec1 or else (Dec2 and then . . . ) OR ELSE c1 AND THEN De [. c2 .. De ] 13 / 17

human-readable characteristic Dec1 or else (Dec2 and then . . . ) OR ELSE c1 AND THEN De no and then [. c2 .. De ] 13 / 17

human-readable characteristic Dec1 or else (Dec2 and then . . . ) OR ELSE c1 AND THEN De [. c2 .. De ] no or else 13 / 17

Experimental results In the industrial applications that we looked at, 99 % of the decisions are such that obc implies mcdc 14 / 17

Experimental results In the industrial applications that we looked at, 99 % of the decisions are such that obc implies mcdc conﬁguration App. 1 App. 2 GNATcoverage #decisions 869 37324 1026 #non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %) 14 / 17

Experimental results Evaluating the impact of this optimization on the qualiﬁcation testsuite of GNATcoverage; 15 / 17

Experimental results Evaluating the impact of this optimization on the qualification testsuite of GNATcoverage; compute the coverage of GNATcoverage in 3 different configurations: obc: as a baseline; mcdc 1: historical traces on branches of all decision; mcdc 2: historical traces only when there are conditions reachable by several paths... 15 / 17

Experimental results Evaluating the impact of this optimization on the qualification testsuite of GNATcoverage; compute the coverage of GNATcoverage in 3 different configurations: obc: as a baseline; mcdc 1: historical traces on branches of all decision; mcdc 2: historical traces only when there are conditions reachable by several paths... ...and compare the size of the generated traces. 15 / 17

Experimental results conﬁguration obc mcdc 1 mcdc 2 #branches to trace 0 1788 22 size of traces 1.33G 5.06G 1.37G 16 / 17

Experimental results conﬁguration obc mcdc 1 mcdc 2 #branches to trace 0 1788 22 size of traces 1.33G 5.06G 1.37G The optimization removes 99 % of historical traces 16 / 17

Experimental results conﬁguration obc mcdc 1 mcdc 2 #branches to trace 0 1788 22 size of traces 1.33G 5.06G 1.37G The optimization removes 99 % of historical traces The overead compared to obc is marginal 16 / 17

More information... Resources: Couverture public repository on the Open-Do forge (https://forge.open-do.org/projects/couverture/) 17 / 17

More information... Resources: Couverture public repository on the Open-Do forge (https://forge.open-do.org/projects/couverture/) Alloy models 17 / 17

More information... Resources: Couverture public repository on the Open-Do forge (https://forge.open-do.org/projects/couverture/) Alloy models Proofs of theorems comparing obc and mcdc 17 / 17

More information... Resources: Couverture public repository on the Open-Do forge (https://forge.open-do.org/projects/couverture/) Alloy models Proofs of theorems comparing obc and mcdc other results about mcdc 17 / 17

More information... Resources: Couverture public repository on the Open-Do forge (https://forge.open-do.org/projects/couverture/) Alloy models Proofs of theorems comparing obc and mcdc other results about mcdc Couverture project gave birth to an industrial tool: GNATcoverage 17 / 17

http://www.open-do.org	4113
http://libre.adacore.com	565
http://www.adacore.com	489
http://www2.adacore.com	5
http://translate.googleusercontent.com	4
http://www1.adacore.com	4
http://abtasty.com	4
http://corp3.adacore.com	2
http://127.0.0.1	1
http://new.open-do.org	1
http://homi.abakus-systemhaus.de	1
http://paris.act-europe.fr	1

Couverture erts2012

by AdaCore on Mar 13, 2012

Accessibility

Categories

Upload Details

Usage Rights

12 Embeds 5,190

Statistics

Couverture erts2012 Presentation Transcript