1. Integrating Formal Program
Verification with Testing
Cyrille Comar, Johannes Kanig and Yannick Moy

2. Integrating Formal Program
with Testing
Verification
Cyrille Comar, Johannes Kanig and Yannick Moy

3. Integrating Formal Program
with
Verification
Cyrille Comar, Johannes Kanig and Yannick Moy

4. Motivation

5. Cost of testing
• Cost of testing greater than cost of development
• 10% increase each year for avionics software (Boeing META Project)
• Uneven repartition:
20%  80% of effort!
80%
• Uneven quality: 80% of errors traced to 20% of code
(NASA Software Safety Guidebook)
• Need to reduce and focus the cost of testing

6. DO-178C: formal methods can replace testing
Formal methods […] might be the
primary source of evidence for
the satisfaction of many of the
objectives concerned with
development and verification.
2011: Formal Methods Supplement (DO-333)

7. Myths of formal methods
• Myth 4: Formal methods require highly trained mathematicians
• Myth 5: Formal methods increase the cost of development
• Myth 6: Formal methods are unacceptable to users
• Myth 7: Formal methods are not used on real, large-scale software
(Anthony Hall, Praxis Systems, 1990)

8. Practice of formal methods
Since 2001, Airbus has been
integrating several tool supported
formal verification techniques into
the development process of
avionics software products.
2009: Formal Verification of Avionics Software Products
(Souyris, Wiels, Delmas, Delseny)

9. Cost of verification
20%  80% of 20%  80% of
80% testing effort 80% formal effort
Hi-Lite goal: using formal verification first, then testing…
4%
16%
testing
80%
formal
… to reduce and focus the cost of verification

10. Proof + Test

11. Programming Contracts
{P}C{Q} Hoare logic (1969)
logic contracts executable contracts
for proofs for tests
SPARK (1987) Eiffel DbC (1986)
Hi-Lite: executable annotation language???

12. Project

13. Ada 2012

14. Testing vs. Formal Verification
prove pre of Q
use Q code
assume post of Q
cover P constructs P calls Q
P calls Q
P P
Q Q
assume pre of Q
actual body of Q
prove post of Q
or stub…
local exhaustivity argument: global soundness argument:
each function covered P all functions proved
 enough behaviors  all assumptions justified
explored
Q
R

15. Combining tests and proofs
P is tested
P calls Q
How so we justify
assumptions made
P during proof?
Q
Q calls P Q is proved
verification combining tests and proofs should be
AT LEAST AS GOOD AS
verification based on tests only

16. Caution: contracts are not only pre/post!
strong typing parameters
not aliased
)…
parameters
initialized
data dependences

17. Combination 1: tested calls proved
P is tested
P calls Q
during testing:
check that P
precondition of Q Q
is respected Q is proved
assumption for proof:
precondition of Q
is respected

18. Combination 2: proved calls tested
P is tested
during testing:
check that P
postcondition of P Q
is respected Q calls P Q is proved
assumption for proof:
postcondition of P
is respected

19. Testing + Formal Verification
tested
P proved
Q
R
proved
local exhaustivity argument: global soundness argument:
- test: function covered - proof: assumptions proved
- proof: by nature of proof - test: assumptions tested
Testing must check additional properties
Done by compiler instrumentation

20. GNAT toolsuite
executable
GNAT GNATtest
compiler unit testing aggregated
verification
results
GNATprove
unit proof

21. Conclusion

22. Airbus 5 “must-have” of formal methods
• Soundness
• Applicability to the code
• Usability by normal engineers on normal computers
• Improve on classical methods
current work
• Certifiability

23. Benefits of openness
.org
• announcements • public: • all code
• meeting slides  meeting minutes • dev docs
• articles / docs  technical work • user docs
 69 members
• private:
 management
 partner code
 external collaborations with industry and academia

24. Project Partners

25. www.open-do.org/projects/hi-lite