GNATcoverage/GNATemulator launch

From Couverture to GNATcoverage
The path from the original Idea to the final Product(s)
CyrilleComar

Summary
Original Needs & Goals
Challenges along the way
Main Results

Original Needs
Structural Coverage Analysis is required by certification standards:
Open source Coverage Tools exist but are not usable in a HI context
Proprietary Tools exist but do not support all versions of Ada
Complete the GNAT Pro Toolset for the High Integrity Market
Better support for the rapidly evolving versions of Ada (83 … 95 … 2005 … 2012 …)

Original Goals
Provide an High Quality Open Source alternative to existing proprietary tools
Provide Support for Agile/Lean Development
In particular: Continuous Integration/Certification
Open-DO initiative
Find the best compromise between Source and Object Coverage

The Couverture Project (2008-2010)
One of the first FUI projects from the GTLL at System@atic
4 partners (AdaCore, Openwide, Telecom PT, Paris 6)
Effort of 160 man-month (2,23 M€) over 2 years
45% Financed by the city of Paris, IdFregion, DGE
This project gave us the capability to meet the unexpected challenges we were faced with

Object Coverage vs Source Coverage
Big debate in the Certification Community
Which one is the most Accurate / Appropriate ?
Which one is the most efficient ?
Source
Statement/Decision are Source concepts
usually works by instrumenting the code
can be done on fast native platforms
requires double testing strategy
Object
on final code (no instrumentation)
on final hardware
not language specific
more precise

Object Coverage vs Source Coverage
General Belief at beginning of project :
Object Coverage Statement Coverage
Object Branch Coverage  Decision Coverage
Object Branch Coverage  MC/DC
(when using short-circuit operators instead of and/or)
But a FAA study arrived after the beginning showing unexplainable differences between OBC and MC/DC DOT/FAA/AR-07/17

Challenge 1
It is difficult to provide accurate source coverage info from execution traces:
- no trace of “statement” / “condition” / “decision” at binary level
- optimization can change significantly the control flow

Accurate Source Coverage Info
Sources
Sources
Executable
Sources
Sources
GNAT Pro
GNATemulator
Exec
traces
Exec
traces
decorated sources
Exec
traces
decorated sources
decorated sources
GNATcoverage

Not accurate enough to locate precise statements, decisions, or conditions boundaries
Sources
Sources
Executable
Sources
Sources
GNAT Pro
Debug info
GNATemulator
Exec
traces
Exec
traces
decorated sources
Exec
traces
decorated sources
decorated sources
GNATcoverage

Source Coverage Information
(Static analysis)
-fpreserve-control-flow
Executable
Sources
Sources
Sources
Enhanced
GNAT Pro
Sources
Debug info
SCOs
GNATemulator
Exec
traces
Exec
traces
decorated sources
Exec
traces
decorated sources
decorated sources
GNATcoverage

Challenge 2
OBC does not imply MC/DC
We Need better theoretical foundations

Object Branch Coverage ≠ MC/DC
function P (A, B, C : Boolean) return Boolean is
begin
if( A and then B ) or else C then
return True;
end if;
end P;
Conditions
A
Decision
C
B
MC / DC
OBC
3 tests are sufficient
At least n+1 tests
n = number of conditions

Object Branch Coverage ≠ MC/DC
Definition of a formal model to express coverage metrics based on BDD (Binary Decision Tree)
Express OBC and MC/DC in this model
Find counter-examples
Find precise perimeter where the equivalence can be proven
Formally prove this result
Use
Open Source Model Checker
Alloy

Counter-Examples
A
function P (A, B, C : Boolean) return Boolean is
begin
if( A and then B ) or else C then
return True;
end if;
end P;
T
F
B
F
C
T
F
T
BDD
OBC = covering paths in the BDD
C0
T
F
C1
function P (C0, C1, C2, C3, C4 … : Boolean) return Boolean is
begin
if ((((…(C0 and thenC1) or else C2) and then C3) or else C4 … then
return True;
end if;
end P;
C2
F
T
F
T
C3
C4
F
T
F
T
3 tests sufficient instead of N+1, for any N

Equivalence can be proven when
There are no diamond in the BDD
How to translate this in “User Terms” ?
No easy formulation… the best we found is
Transform Boolean expression in “Negative Normal Form”
No “and then” in left operand of a “or else”
No “or else” in left operand of a “and then”

Main Results
Emulation is key to Agile cross development
GNATcoverage takes advantage of the theoretical results to:
Implement properly MC/DC in the complex case
Optimize the simple case by using OBC
Creation of “open source” qualif material as part of Open-DO
-fpreserve-control-flow: a certification friendly mode for GCC

Conclusion
The Couverture project allowed us to concentrate on solving properly the unexpected challenges
Existing Open-Source technologies have played a key role:
Qemu is the base of GNATemulator
Alloy helped a lot for the mathematical proofs
As a result, new industrial-ready Open Source tools are now available for the HI developers’ community

http://www.open-do.org	399
https://bentobako.org	1
http://webcache.googleusercontent.com	1
http://translate.googleusercontent.com	1

GNATcoverage/GNATemulator launch

by AdaCore on Feb 23, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 402

Statistics

GNATcoverage/GNATemulator launch — Presentation Transcript