Jerome Guitton's presentation on the formal results derived from the COUVERTURE project, whose goal was to develop tools to support structural coverage analysis of unin- strumented safety-critical software. After briefly intro- ducing the project context and explaining the need for formal foundations, we focus on the relationships be- tween machine branch coverage and the DO-178B Modified Condition/Decision Coverage (MCDC) criterion. The full paper can be found at http://www.erts2012.org/Site/0P2RUC89/7A-3.pdf

1. Formalization and Comparison
of mcdc and Object Branch
Coverage Criteria
Cyrille Comar, Jerome Guitton, Olivier Hainque, Thomas Quinot
1 / 17

2. Coverage for Level A
Considering the highest level of certification for aircraft (A in
DO-178B):
Test coverage goal: mcdc
2 / 17

3. Coverage for Level A
Considering the highest level of certification for aircraft (A in
DO-178B):
Test coverage goal: mcdc
Structural (source) coverage
2 / 17

4. Coverage for Level A
Considering the highest level of certification for aircraft (A in
DO-178B):
Test coverage goal: mcdc
Structural (source) coverage
More than statement coverage...
2 / 17

5. Coverage for Level A
Considering the highest level of certification for aircraft (A in
DO-178B):
Test coverage goal: mcdc
Structural (source) coverage
More than statement coverage...
...but not all execution paths (too costly)
2 / 17

6. Coverage for Level A
Considering the highest level of certification for aircraft (A in
DO-178B):
Test coverage goal: mcdc
Structural (source) coverage
More than statement coverage...
...but not all execution paths (too costly)
e.g. for a decision C1 and C2 and . . . and CN :
2 / 17

7. Coverage for Level A
Considering the highest level of certification for aircraft (A in
DO-178B):
Test coverage goal: mcdc
Structural (source) coverage
More than statement coverage...
...but not all execution paths (too costly)
e.g. for a decision C1 and C2 and . . . and CN :
2N execution paths
2 / 17

8. Coverage for Level A
Considering the highest level of certification for aircraft (A in
DO-178B):
Test coverage goal: mcdc
Structural (source) coverage
More than statement coverage...
...but not all execution paths (too costly)
e.g. for a decision C1 and C2 and . . . and CN :
2N execution paths
N + 1 tests in mcdc
2 / 17

9. Some issues with mcdc
source coverage criteria means language-dependent
3 / 17

10. Some issues with mcdc
source coverage criteria means language-dependent
no tool for Ada 2005 when Couverture started...
3 / 17

11. Some issues with mcdc
source coverage criteria means language-dependent
no tool for Ada 2005 when Couverture started...
instrumenting source code? intrusive...
3 / 17

12. Some issues with mcdc
source coverage criteria means language-dependent
no tool for Ada 2005 when Couverture started...
instrumenting source code? intrusive...
unbounded execution traces
3 / 17

13. Some issues with mcdc
source coverage criteria means language-dependent
no tool for Ada 2005 when Couverture started...
instrumenting source code? intrusive...
unbounded execution traces
Use object coverage instead?
3 / 17

14. Object coverage to assess mcdc
Assumption that object branch coverage (obc) is stronger
than mcdc
4 / 17

15. Object coverage to assess mcdc
Assumption that object branch coverage (obc) is stronger
than mcdc
widespread industrial practise
4 / 17

16. Object coverage to assess mcdc
Assumption that object branch coverage (obc) is stronger
than mcdc
widespread industrial practise
language-independent
4 / 17

17. Object coverage to assess mcdc
Assumption that object branch coverage (obc) is stronger
than mcdc
widespread industrial practise
language-independent
bounded traces
4 / 17

18. Using obc to achieve mcdc ? short-circuit operators...
(A mod B = 0) and then (C = 0)

19. Using obc to achieve mcdc ? short-circuit operators...
(A mod B = 0) and then (C = 0)
A mod B = 0
F T
F C =0
F T
F T
5 / 17

20. Using obc to achieve mcdc ? short-circuit operators...
A mod B = 0
B = −1
F
(A mod B = 0) and then (C = 0) A 0
F
A mod B = 0 B<0
T T
F T
F T
F C =0
F T R=0
F T F T
F C =0
F T
F T
5 / 17

21. What do certification standards say about that?
DO-248C FAQ #42: Object coverage can be used as long as
analysis can be provided which demonstrates that the coverage
analysis conducted at the Object Code will achieve a comparable
level of coverage assurance as that conducted at the Source Code
level.
6 / 17

22. The sad truth...
The assumption is wrong: obc is not stronger than mcdc!
7 / 17

23. The sad truth...
The assumption is wrong: obc is not stronger than mcdc!
A counterexample in DOT/FAA/AR-07/20, Jun 2007: Object
Oriented Technology Verification Phase 3 Report - Structural
Coverage at the Source Code and Object Code Levels
7 / 17

24. The sad truth...
The assumption is wrong: obc is not stronger than mcdc!
A counterexample in DOT/FAA/AR-07/20, Jun 2007: Object
Oriented Technology Verification Phase 3 Report - Structural
Coverage at the Source Code and Object Code Levels
(A and then B) or else C can be covered for obc without
achieving mcdc
7 / 17

25. Having a closer look...
Alloy model: check conjectures, generate counterexamples...
8 / 17

26. Pathological case
Alloy helped finding a impressive counterexample:
9 / 17

27. Pathological case
Alloy helped finding a impressive counterexample:
Decision with an arbitrary high number of conditions N...
9 / 17

28. Pathological case
Alloy helped finding a impressive counterexample:
Decision with an arbitrary high number of conditions N...
...that needs N+1 tests to be mcdc-covered...
9 / 17

29. Pathological case
Alloy helped finding a impressive counterexample:
Decision with an arbitrary high number of conditions N...
...that needs N+1 tests to be mcdc-covered...
...and only 3 tests to be obc-covered!
9 / 17

30. Pathological case
Alloy helped finding a impressive counterexample:
Decision with an arbitrary high number of conditions N...
...that needs N+1 tests to be mcdc-covered...
...and only 3 tests to be obc-covered!
Here obc is much weaker than mcdc!
9 / 17

31. Now what?
Sure, in some cases, obc does not imply mcdc...
10 / 17

32. Now what?
Sure, in some cases, obc does not imply mcdc...
...and in some pathological cases, the two criteria diverges
quite badly...
10 / 17

33. Now what?
Sure, in some cases, obc does not imply mcdc...
...and in some pathological cases, the two criteria diverges
quite badly...
...but past experience has shown that it works in many cases
10 / 17

34. Now what?
Sure, in some cases, obc does not imply mcdc...
...and in some pathological cases, the two criteria diverges
quite badly...
...but past experience has shown that it works in many cases
...so are there conditions that would allow this implication?
10 / 17

35. Now what?
Sure, in some cases, obc does not imply mcdc...
...and in some pathological cases, the two criteria diverges
quite badly...
...but past experience has shown that it works in many cases
...so are there conditions that would allow this implication?
could be enforced by a coding standard
10 / 17

36. Now what?
Sure, in some cases, obc does not imply mcdc...
...and in some pathological cases, the two criteria diverges
quite badly...
...but past experience has shown that it works in many cases
...so are there conditions that would allow this implication?
could be enforced by a coding standard
could be an optimization for coverage tools
10 / 17

37. Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obc
implies mcdc.
11 / 17

38. Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obc
implies mcdc.
C1 and then C2
11 / 17

39. Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obc
implies mcdc.
C1 and then C2
C1
F T
F C2
F T
F T

40. Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obc
implies mcdc.
C1 and then C2
C1
F T
F C2
F T
F T
11 / 17

41. Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached by
more than one execution path, obc does not always imply mcdc.
12 / 17

42. Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached by
more than one execution path, obc does not always imply mcdc.
(A and then B) or else C
12 / 17

43. Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached by
more than one execution path, obc does not always imply mcdc.
(A and then B) or else C
A
B
T C
T F

44. Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached by
more than one execution path, obc does not always imply mcdc.
(A and then B) or else C
A
B
T C
T F
12 / 17

45. human-readable characteristic
Dec1 or else (Dec2 and then . . . )
13 / 17

46. human-readable characteristic
Dec1 or else (Dec2 and then . . . )
OR ELSE
c1
AND THEN
De
[.
c2
..
De
]
13 / 17

47. human-readable characteristic
Dec1 or else (Dec2 and then . . . )
OR ELSE
c1
AND THEN
De
no and then
[.
c2
..
De
]
13 / 17

48. human-readable characteristic
Dec1 or else (Dec2 and then . . . )
OR ELSE
c1
AND THEN
De
[.
c2
..
De
]
no or else
13 / 17

49. Experimental results
In the industrial applications that we looked at, 99 % of the
decisions are such that obc implies mcdc
14 / 17

50. Experimental results
In the industrial applications that we looked at, 99 % of the
decisions are such that obc implies mcdc
configuration App. 1 App. 2 GNATcoverage
#decisions 869 37324 1026
#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)
14 / 17

51. Experimental results
In the industrial applications that we looked at, 99 % of the
decisions are such that obc implies mcdc
configuration App. 1 App. 2 GNATcoverage
#decisions 869 37324 1026
#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)
14 / 17

52. Experimental results
Evaluating the impact of this optimization on the qualification
testsuite of GNATcoverage;
15 / 17

53. Experimental results
Evaluating the impact of this optimization on the qualification
testsuite of GNATcoverage;
compute the coverage of GNATcoverage in 3 different
configurations:
obc: as a baseline;
mcdc 1: historical traces on branches of all decision;
mcdc 2: historical traces only when there are conditions
reachable by several paths...
15 / 17

54. Experimental results
Evaluating the impact of this optimization on the qualification
testsuite of GNATcoverage;
compute the coverage of GNATcoverage in 3 different
configurations:
obc: as a baseline;
mcdc 1: historical traces on branches of all decision;
mcdc 2: historical traces only when there are conditions
reachable by several paths...
...and compare the size of the generated traces.
15 / 17

55. Experimental results
configuration obc mcdc 1 mcdc 2
#branches to trace 0 1788 22
size of traces 1.33G 5.06G 1.37G
16 / 17

56. Experimental results
configuration obc mcdc 1 mcdc 2
#branches to trace 0 1788 22
size of traces 1.33G 5.06G 1.37G
The optimization removes 99 % of historical traces
16 / 17

57. Experimental results
configuration obc mcdc 1 mcdc 2
#branches to trace 0 1788 22
size of traces 1.33G 5.06G 1.37G
The optimization removes 99 % of historical traces
The overead compared to obc is marginal
16 / 17

58. More information...
Resources: Couverture public repository on the Open-Do forge
(https://forge.open-do.org/projects/couverture/)
17 / 17

59. More information...
Resources: Couverture public repository on the Open-Do forge
(https://forge.open-do.org/projects/couverture/)
Alloy models
17 / 17

60. More information...
Resources: Couverture public repository on the Open-Do forge
(https://forge.open-do.org/projects/couverture/)
Alloy models
Proofs of theorems comparing obc and mcdc
17 / 17

61. More information...
Resources: Couverture public repository on the Open-Do forge
(https://forge.open-do.org/projects/couverture/)
Alloy models
Proofs of theorems comparing obc and mcdc
other results about mcdc
17 / 17

62. More information...
Resources: Couverture public repository on the Open-Do forge
(https://forge.open-do.org/projects/couverture/)
Alloy models
Proofs of theorems comparing obc and mcdc
other results about mcdc
Couverture project gave birth to an industrial tool:
GNATcoverage
17 / 17