From Couverture to GNATcoverage<br />The path from the original Idea to the final Product(s)<br />CyrilleComar<br />
Summary<br /><ul><li>Original Needs & Goals
Challenges along the way
Main Results</li></li></ul><li>Original Needs<br /><ul><li>Structural Coverage Analysis is required by certification stand...
Open  source Coverage Tools exist but are not usable in a HI context
Proprietary Tools exist but do not support all versions of Ada
Complete the GNAT Pro Toolset for the High Integrity Market
Better support for the rapidly evolving versions of Ada (83 … 95 … 2005 … 2012 …)</li></li></ul><li>Original Goals<br /><u...
Provide Support for Agile/Lean Development
In particular: Continuous Integration/Certification
Open-DO initiative
Find the best compromise between Source and Object Coverage</li></li></ul><li>The Couverture Project (2008-2010)<br /><ul>...
4 partners (AdaCore, Openwide,  Telecom PT, Paris 6)
Effort  of 160 man-month (2,23 M€) over 2 years
45% Financed by the city of Paris, IdFregion, DGE </li></ul>This project gave us the capability to meet the unexpected cha...
Object Coverage vs Source Coverage<br /><ul><li>Big debate in the  Certification Community
Which one is the most Accurate / Appropriate ?
Which one is the most efficient ?</li></ul>Source<br /><ul><li> Statement/Decision are Source concepts
Upcoming SlideShare
Loading in …5
×

GNATcoverage/GNATemulator launch

2,219
-1

Published on

Cyrille Comar slides show the objectives and results of the Couverture project (http://www.open-do.org/projects/couverture/) and the corresponding products that have been developed.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,219
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

GNATcoverage/GNATemulator launch

  1. 1. From Couverture to GNATcoverage<br />The path from the original Idea to the final Product(s)<br />CyrilleComar<br />
  2. 2. Summary<br /><ul><li>Original Needs & Goals
  3. 3. Challenges along the way
  4. 4. Main Results</li></li></ul><li>Original Needs<br /><ul><li>Structural Coverage Analysis is required by certification standards:
  5. 5. Open source Coverage Tools exist but are not usable in a HI context
  6. 6. Proprietary Tools exist but do not support all versions of Ada
  7. 7. Complete the GNAT Pro Toolset for the High Integrity Market
  8. 8. Better support for the rapidly evolving versions of Ada (83 … 95 … 2005 … 2012 …)</li></li></ul><li>Original Goals<br /><ul><li>Provide an High Quality Open Source alternative to existing proprietary tools
  9. 9. Provide Support for Agile/Lean Development
  10. 10. In particular: Continuous Integration/Certification
  11. 11. Open-DO initiative
  12. 12. Find the best compromise between Source and Object Coverage</li></li></ul><li>The Couverture Project (2008-2010)<br /><ul><li>One of the first FUI projects from the GTLL at System@atic
  13. 13. 4 partners (AdaCore, Openwide, Telecom PT, Paris 6)
  14. 14. Effort of 160 man-month (2,23 M€) over 2 years
  15. 15. 45% Financed by the city of Paris, IdFregion, DGE </li></ul>This project gave us the capability to meet the unexpected challenges we were faced with<br />
  16. 16. Object Coverage vs Source Coverage<br /><ul><li>Big debate in the Certification Community
  17. 17. Which one is the most Accurate / Appropriate ?
  18. 18. Which one is the most efficient ?</li></ul>Source<br /><ul><li> Statement/Decision are Source concepts
  19. 19. usually works by instrumenting the code
  20. 20. can be done on fast native platforms
  21. 21. requires double testing strategy </li></ul>Object<br /><ul><li> on final code (no instrumentation)
  22. 22. on final hardware
  23. 23. not language specific
  24. 24. more precise</li></li></ul><li>Object Coverage vs Source Coverage<br /><ul><li>General Belief at beginning of project :
  25. 25. Object Coverage Statement Coverage
  26. 26. Object Branch Coverage  Decision Coverage
  27. 27. Object Branch Coverage  MC/DC</li></ul> (when using short-circuit operators instead of and/or) <br /><ul><li>But a FAA study arrived after the beginning showing unexplainable differences between OBC and MC/DC DOT/FAA/AR-07/17</li></li></ul><li>Challenge 1<br /> It is difficult to provide accurate source coverage info from execution traces:<br /> - no trace of “statement” / “condition” / “decision” at binary level<br /> - optimization can change significantly the control flow<br />
  28. 28. Accurate Source Coverage Info<br />Sources<br />Sources<br />Executable<br />Sources<br />Sources<br />GNAT Pro<br />GNATemulator<br />Exec<br />traces<br />Exec<br />traces<br />decorated sources<br />Exec<br />traces<br />decorated sources<br />decorated sources<br />GNATcoverage<br />
  29. 29. Accurate Source Coverage Info<br />Not accurate enough to locate precise statements, decisions, or conditions boundaries <br />Sources<br />Sources<br />Executable<br />Sources<br />Sources<br />GNAT Pro<br />Debug info<br />GNATemulator<br />Exec<br />traces<br />Exec<br />traces<br />decorated sources<br />Exec<br />traces<br />decorated sources<br />decorated sources<br />GNATcoverage<br />
  30. 30. Accurate Source Coverage Info<br />Source Coverage Information<br /> (Static analysis)<br />-fpreserve-control-flow<br />Executable<br />Sources<br />Sources<br />Sources<br />Enhanced<br />GNAT Pro<br />Sources<br />Debug info<br />SCOs<br />GNATemulator<br />Exec<br />traces<br />Exec<br />traces<br />decorated sources<br />Exec<br />traces<br />decorated sources<br />decorated sources<br />GNATcoverage<br />
  31. 31. Challenge 2<br />OBC does not imply MC/DC<br />We Need better theoretical foundations <br />
  32. 32. Object Branch Coverage ≠ MC/DC<br />function P (A, B, C : Boolean) return Boolean is<br />begin<br />if( A and then B ) or else C then<br /> return True;<br />end if;<br />end P; <br />Conditions<br />A<br />Decision<br />C<br />B<br />MC / DC<br />OBC<br />3 tests are sufficient<br />At least n+1 tests<br />n = number of conditions<br />
  33. 33. Object Branch Coverage ≠ MC/DC<br /><ul><li>Definition of a formal model to express coverage metrics based on BDD (Binary Decision Tree)
  34. 34. Express OBC and MC/DC in this model
  35. 35. Find counter-examples
  36. 36. Find precise perimeter where the equivalence can be proven
  37. 37. Formally prove this result</li></ul>Use<br /> Open Source Model Checker <br />Alloy <br />
  38. 38. Counter-Examples<br />A<br />function P (A, B, C : Boolean) return Boolean is<br />begin<br />if( A and then B ) or else C then<br /> return True;<br />end if;<br />end P; <br />T<br />F<br />B<br />F<br />C<br />T<br />F<br />T<br />BDD<br />OBC = covering paths in the BDD <br />C0<br />T<br />F<br />C1<br />function P (C0, C1, C2, C3, C4 … : Boolean) return Boolean is<br />begin<br />if ((((…(C0 and thenC1) or else C2) and then C3) or else C4 … then<br /> return True;<br />end if;<br />end P; <br />C2<br />F<br />T<br />F<br />T<br />C3<br />C4<br />F<br />T<br />F<br />T<br />3 tests sufficient instead of N+1, for any N<br />
  39. 39. Equivalence can be proven when <br /><ul><li>There are no diamond in the BDD
  40. 40. How to translate this in “User Terms” ?
  41. 41. No easy formulation… the best we found is
  42. 42. Transform Boolean expression in “Negative Normal Form”
  43. 43. No “and then” in left operand of a “or else”
  44. 44. No “or else” in left operand of a “and then”</li></li></ul><li>Main Results<br /><ul><li>Emulation is key to Agile cross development
  45. 45. GNATcoverage takes advantage of the theoretical results to:
  46. 46. Implement properly MC/DC in the complex case
  47. 47. Optimize the simple case by using OBC
  48. 48. Creation of “open source” qualif material as part of Open-DO
  49. 49. -fpreserve-control-flow: a certification friendly mode for GCC</li></li></ul><li>Conclusion<br /><ul><li>The Couverture project allowed us to concentrate on solving properly the unexpected challenges
  50. 50. Existing Open-Source technologies have played a key role:
  51. 51. Qemu is the base of GNATemulator
  52. 52. Alloy helped a lot for the mathematical proofs
  53. 53. As a result, new industrial-ready Open Source tools are now available for the HI developers’ community </li>

×