Software Testing for Critical Systems

446 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
446
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Software Testing for Critical Systems

  1. 1. Software Testing for Critical Systems Julien Fayolle and Sandrine-Dominique Gouraud1 1 G´nie e logiciel, LRI, Universit´ d’Orsay. e {fayolle, gouraud}@lri.fr Web pages: www.lri.fr/∼fayolle and www.lri.fr/∼gouraud University of York, CS seminar. Fayolle & Gouraud Software Testing for Critical Systems 1
  2. 2. Overview of Testing How does one test a software? Á Testing all entries is not possible. Á We don’t prove the software right. Ä Create a test set. Functional testing: specifications, black box Structural testing: structure and code, glass box Statistical testing: draw at random from the input domain Ä Run the program on the test set. Ä Compare the outputs to the expected outputs (oracle). Fayolle & Gouraud Software Testing for Critical Systems 2
  3. 3. Statistical Testing Software for critical systems (planes’ motors, nuclear plants’ safety, missiles, . . . ); Intensive testing; Provide quantitative figures on the test set; Provide adequacy figures on the test methods; Drawback: coverage of rare cases. factorial(integer n) if(n!=0) fact:=1; for i from n to 2 fact:=fact*i; else fact:=0; Fayolle & Gouraud Software Testing for Critical Systems 3
  4. 4. Statistical Structural Testing First answer: [Th´venod-Fosse & Waeselynck’91, LAAS] e « Combining statistical and structural testing. « Compute a distribution on the input domain and draw tests inputs accordingly. « How good does the test set cover the possible software executions? Fayolle & Gouraud Software Testing for Critical Systems 4
  5. 5. Test quality 1 6 j a c g e h 0 3 4 7 b f k 2 5 d Test quality qn = 1 − (1 − q)n , where n is the number of tests and q the minimal probability to reach one element. Fayolle & Gouraud Software Testing for Critical Systems 5
  6. 6. AuGuSTe Drawback: the input distribution has to be computed by hand. AuGuSTe.   Automated Generation of Statistical Tests, [Gouraud et alii ’04] First programmed for C code. Coded in Objective Caml; uses MuPAD, Eclipse, Prolog, C, Java and a GATEL module. Statistical structural testing. Drawing uniformly at random paths of a given length in the control graph. AuGuSTe web site: http://www.lri.fr/∼fayolle/Auguste.html Fayolle & Gouraud Software Testing for Critical Systems 6
  7. 7. Structural coverage criteria Given a control flow graph All paths All i-paths Branch coverage: all edges Block coverage: all instructions Predicate coverages Test quality for statistical structural tests qn minimal probability to reach one element with a test set of size n. qn = 1 − (1 − q)n . Fayolle & Gouraud Software Testing for Critical Systems 7
  8. 8. Random generation of paths Goal: optimize the test quality i.e. maximize the smallest probability to reach an element. Combinatorial description of the control graph Uniform drawing of path (criteria consideration) 1 6 j a c g e 0 3 4 7 f h b k 2 5 d Atoms are edges G = ac(egj + fG1 ) + bdG1 , and G1 = hj + k. Fayolle & Gouraud Software Testing for Critical Systems 8
  9. 9. How does AuGuSTe work? Inputs Program Coverage criteria Number of tests (test quality) Maximal length of the paths (elementary path) 1. Analysis: building the combinatorial structure of the control graph 2. Drawing paths uniformly at random in the graph 3. Constraint resolution (uses a GATEL module) Output Test set. Fayolle & Gouraud Software Testing for Critical Systems 9
  10. 10. Testing methods comparison using mutants Creating mutants : seed faults in the original program. A large repertoire of mutation operators: changing ≥ in ≤, > or <; changing 0 in 2; performance criteria: mutation score (kill count). stability of the fault detection ability ¸ average. Softwares for creating mutants Mothra [Offutt, DeMillo et alii’88]; SESAME [Crouzet, Th´venod-Fosse & Waeselynck’98]; e µJava [Offutt, Ma & Kwon’03]. Fayolle & Gouraud Software Testing for Critical Systems 10
  11. 11. Experimental results Function from industrial software (77 lines) 605 mutants (for some mutations). Test quality .9999 5 runs of test sets of size 850. min avg max Uniform testing 0.8950 NA 0.9150 Structural statistical testing 0.9898 0.9901 0.9915 AuGuSTe 0.9854 0.9854 0.9854 Table: Experimental mutation scores over 5 runs Fayolle & Gouraud Software Testing for Critical Systems 11
  12. 12. GATEL GATEL   Test Suites Generation from Lustre Descriptions [Marre et alii, CEA and LRI] Uses Lustre descriptions. Lustre code or Lustre spec. (test) Uses constraint solving techniques, non-deterministic heuristics. Structural or functional software testing. GATEL web site : http://www-list.cea.fr/labos/gb/LSL/test/gatel/index.html Fayolle & Gouraud Software Testing for Critical Systems 12
  13. 13. Lustre code node Alarm(stream0, s up, s down: real) returns (alarm s: bool); var upstream, downstream: bool let upstream = stream0 > s up; downstream = stream0 < s down ; alarm s = Automaton(upstream, upstream, downstream); tel; node Automaton(init, upstream, downstream: bool) returns (state: bool); let state = init -> if upstream and not pre(state) then true else if downstream and pre(state) then false else pre(state); let; Fayolle & Gouraud Software Testing for Critical Systems 13
  14. 14. Lustre Declarative language for specification and/or programming. Reactive synchronous systems. Data-flow. The specification allows the automatic generation of code. Fayolle & Gouraud Software Testing for Critical Systems 14
  15. 15. Perspectives ( Extend the idea of uniform drawing to the synchronous data-flow languages. ( Application to Lustre ( Enrich GATEL with random generation of combinatorial structures. ( Provide precise bounds on the fault detection ability of AuGuSTe-like methods for imperative and declarative languages. Fayolle & Gouraud Software Testing for Critical Systems 15

×