FASTEST: Test Case Generation from Z Specifications

1,753 views

Published on

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,753
On SlideShare
0
From Embeds
0
Number of Embeds
193
Actions
Shares
0
Downloads
51
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

FASTEST: Test Case Generation from Z Specifications

  1. 1. FastestTest Case Generation from Z Specifications Maximiliano Cristi´ a CIFASIS Universidad Nacional de Rosario Rosario – Argentina cristia@cifasis-conicet.gov.ar Alpen-Adria Univesit¨t – Klagenfurt – March, 2012 a
  2. 2. Software Testing Basics Program Correctness A program is correct if it behaves according to its specification Edsger Dijkstra “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” Donald Knuth “Beware of bugs in the above code; I have only proved it correct, not tried it.” 2 / 34
  3. 3. Model-Based Testing (MBT) MBT analyses a model of the MBT Process program to derive test cases which requeriments later are provided as input. At the beginning, the model is used model error? to generate test cases. verify generate At end, the model becomes the abstract abstract oracle. tests results If a model is provided, the process abstract is almost automatic. refine execute tests results 3 / 34
  4. 4. The Test Template Framework and Fastest The Test Template Framework (TTF) A particular method of MBT which uses Z specifications. Unit testing. Stocks and Carrington, A Framework for Specification-Based Testing, IEEE TOSE 1996. Fastest It is the only existing implementation of the TTF. Available at www.fceia.unr.edu.ar/~mcristia. Cristi´ and Rodr´ a ıguez Monetti, Implementing and Applying the Stocks-Carrington Framework for MBT, ICFEM 2009. 4 / 34
  5. 5. TTF AutomationThe TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors. 5 / 34
  6. 6. An Example The Requirements A bank has a certain number of savings accounts. Customers can deposit and withdraw money. Each account is identified with an account number. An amount of money is considered to be a natural number. Only the deposit operation is going to be shown. 6 / 34
  7. 7. The Z SpecificationTypes and State Space [ACCNUM] A type for account numbers BALANCE == N A type for balances AMOUNT == N A type for amounts of money Bank sa : ACCNUM → BALANCE 7 / 34
  8. 8. The Z SpecificationThe Deposit Operation DepositOk ∆Bank n? : ACCNUM a? : AMOUNT n? ∈ dom sa a? > 0 sa = sa ⊕ {n? → sa n? + a?} DepositE 1 == [ΞBank; n? : ACCNUM | n? ∈ dom sa] / DepositE 2 == [ΞBank; a? : Z | a? ≤ 0] Deposit == DepositOk ∨ DepositE 1 ∨ DepositE 2 loadspec bank.tex selop Deposit 8 / 34
  9. 9. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree. 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  10. 10. Domain Partition Given a Z operation, take its input space and get a partition of it. input testing input testing input testing space tactic space tactic space tactic P Q P P1 Q Q1 atomic predicates Each member of a partition is given by a Z predicate (i.e., first-order logic and Zermelo-Fraenkel set theory). Partitioning can be represented as a tree. Test cases are derived only from the “leaves” Leaves are called test conditions. 10 / 34
  11. 11. Testing Tactics The TTF performs domain partition by applying testing tactics. Disjunctive Normal Form (DNF) Write the operation predicate in DNF and take the precondition of each disjunct. Standard Partition (SP) for R ⊕ G R = {}, G = {} R = {}, G = {} R = {}, G = {} R = {}, G = {}, dom R = dom G R = {}, G = {}, dom G ⊂ dom R R = {}, G = {}, (dom R ∩ dom G ) = {} R = {}, G = {}, dom R ⊂ dom G R = {}, G = {}, (dom R ∩ dom G ) = {}, ¬ (dom G ⊆ dom R), ¬ (dom R ⊆ dom G ) 11 / 34
  12. 12. Applying DNF and SP to Deposit First DNF DNF == [Deposit VIS | n? ∈ dom sa ∧ a? > 0] Deposit1 DNF == [Deposit VIS | n? ∈ dom sa] Deposit2 / DNF == [Deposit VIS | a? ≤ 0] Deposit3 DNF Second SP, but only to Deposit1 SP Deposit1 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit2 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit3 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit4 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {} ∧ dom sa = dom{n? → (sa n? + a?)}] SP DNF Deposit5 == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {} ∧ dom{n? → (sa n? + a?)} ⊂ dom sa] (three more test conditions not shown here) genalltt addtactic Deposit_DNF_1 SP oplus sa oplus {(n? , sa~n? + a?)} 12 / 34
  13. 13. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  14. 14. Pruning the Testing Tree Pruning involves eliminating unsatisfiable test conditions. Before pruning After pruning Deposit VIS Deposit VIS Deposit DNF 1 Deposit DNF 1 Deposit SP 1 Deposit SP 4 Deposit SP 2 Deposit SP 5 Deposit SP 3 Deposit DNF 2 Deposit SP 4 Deposit DNF 3 Deposit SP 5 Deposit SP 6 prunett Deposit SP 7 Deposit SP 8 Deposit DNF 2 Deposit DNF 3 14 / 34
  15. 15. Pruning in Detail The problem – I Many test conditions are unsatisfiable. Most of them are mathematical contradictions. Then, no test cases can be derived from them. Therefore, unsatisfiable test conditions should be pruned from testing trees. In general, this problem is undecidable. The problem – II Z includes many mathematical theories (sets, relations, partial functions, sequences, etc.), making the problem harder. 15 / 34
  16. 16. Detecting Contradictions Detecting logical contradictions Since every test condition is a conjunction of atomic predicates, then the only possible logical contradiction is of the form: ··· ∧ p ∧ ··· ∧ ¬ p ∧ ... This is the easy part. Detecting mathematical contradictions Most of the contradictions are of a mathematical nature. Some depend on the particular semantics of the Z mathematical theories. 16 / 34
  17. 17. A Library of Mathematical Contradictions The core of our method is a library of elimination theorems. Each elimination theorem should represent a contradiction. ETheorem ETheoremName [parameters] atomicPred ............... atomicPred parameters is roughly any Z declaration. atomicPred is any Z atomic predicate and some keywords. Subtyping and equivalence rules. A User extensible (LTEX). No deduction, only pattern-matching. 17 / 34
  18. 18. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  19. 19. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  20. 20. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  21. 21. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  22. 22. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  23. 23. Generating Abstract Test Cases Find constants satisfying test conditions. TC Deposit1 == SP [Deposit4 | n? = accnum2 ∧ sa = {(accnum2 , 1)} ∧ a? = 1] TC Deposit2 == SP [Deposit5 | n? = accnum2 ∧ sa = {(accnum2 , 1), (accnum1 , 1)} ∧ a? = 1] TC DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0] Deposit3 == [Deposit2 2 TC DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0] Deposit4 == [Deposit3 2 genalltca 20 / 34
  24. 24. Test Case Generation in Detail Currently, Fastest implements a rough satisfiability algorithm: 1 Finite sets are bound to variables and types in a test template. ACCNUM, n? → {accnum0 , accnum1 , accnum2 } AMOUNT , BALANCE , a? → {0, 1, 2} ACCNUM → BALANCE , sa → {∅, {accnum2 → 1}, {accnum2 → 1, accnum1 → 1}, ...} 2 The predicate is evaluated on some elements of the Cartesian product of those finite sets. ZLive (from CZT) is used as evaluator. Some simple heuristics are applied. The mathematical structure of the predicate is not considered. 21 / 34
  25. 25. Is it good enough? Fastest has been able to find test cases for an average of 80 % of the satisfiable test conditions from eleven case studies. For the remaining 20 %, Fastest provides a command to help it to find a test case. setfinitemodel Deposit_SP_4 -fm "sa=={{accnum_2 mapsto 1}}" Nevertheless, Fastest’s algorithm needs to be improved or replaced by more sophisticated ones. 22 / 34
  26. 26. Difficulties with Test Case Generation in the TTF The ideas of test templates and of automatic partition of Z schema have been proposed several years ago. Their actual application to systems of industrial size was problematic due to the lack of efficient and adequate constraint solvers and theorem provers. Given the progresses recently achieved for these kinds of tools, these techniques are worth revisiting . . . Ana Cavalcanti and Marie-Claude Gaudel (2011): Testing for refinement in Circus. Acta Informatica, 48(2). 23 / 34
  27. 27. SMT Solvers for the TTF Satisfiability Modulo Theory (SMT) solvers SMT solvers are tools that solve the problem of satisfiability for a range of mathematical and logical theories. No SMT solver supports the Zermelo-Fraenkel set theory. Must define a shallow embedding. Many questions: Is the language of SMT solvers expressive enough? Is there just one shallow embedding? Will the chosen embedding solve all the satisfiable test specifications appearing in the TTF and real Z specifications? Which SMT solver and which embedding will be the best in satisfying more test specifications in less time? Will it be better than Fastest in finding test cases? Or should the SMT solver complement Fastest in this task? 24 / 34
  28. 28. First Results Two shallow embeddings We have defined a shallow embedding of the Zermelo-Franekel set theory for two mainstream SMT solvers: Yices and CVC3. Initial empirical assessment We manually translated 69 satisfiable test templates, for which Fastest cannot find a test case, into these shallow embeddings and ran the SMT solvers over them. 25 / 34
  29. 29. Empirical assessment Main embeddings Variant embeddings Case study Yices CVC3 Yices CVC3 Sat Unk Sat Unk Sat Unk Sat Unk Savings accounts (3) 8 8 8 8 Savings accounts (1) 2 2 2 2 Launching vehicle 8 8 8 8 Plavis 29 29 29 29 SWPDC 13 13 13 13 Scheduler 4 4 4 4 Security class 4 4 4 4 Pool of sensors 1 1 1 1 Totals 9 60 17 52 9 60 17 52 Time spent in processing all the 69 test templates: Yices: 3 s CVC3: 420 s Fastest: 390 s (but no test case is discovered) 26 / 34
  30. 30. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  31. 31. Refining Abstract Test Cases The implementation (C programming language) Assume sa is implemented as follows: struct sanode {int an; float bal; struct sanode *n;} *sa; where sa is a global variable. Besides, Deposit is implemented by the following function: int deposit(int n, float a) Refinement rules We need to link each specification variable to their implementation counterparts. This process is called abstract test case refinement. Write refinement rules and then run the refine command. 28 / 34
  32. 32. Refining Abstract Test Cases (II) A refinement rule for Deposit @RRULE deposit @PREAMBLE #include <bank.h> @LAWS l1:n? ==> n l2:a? ==> a l3:sa ==> sa AS LIST[SLL,n] WITH[sa.@dom ==> sa.an, sa.@ran ==> sa.bal] @UUT deposit(n, a) loadrefrule deposit.txt refine Deposit to test /bank/src implemented in C with deposit 29 / 34
  33. 33. Concrete Test Cases TC Refinement of Deposit2 . #include <bank.h> int main() { int n = 345; n? = accnum2 float a = 1; a? = 1 sa = {(accnum2, 1), (accnum1, 1)} struct sanode sanode0 = {876,1,NULL}; struct sanode sanode1 = {345,1,NULL}; sa = &sanode0; sanode0.n = &sanode1; deposit(n,a); return 1; } 30 / 34
  34. 34. The Remaining Steps Before executing a concrete test case it is necessary to insert code to get the program output. Abstraction rules, similar to refinement rules. Once a test case has been executed its output must be abstracted. Same abstraction rules, different command. The final verification is performed with another command. (Optional) Translating abstract test cases into natural language involves writing translation rules. Quite different from refinement rules. Work in progress Fastest is still a prototype: many of these features are roughly implemented and there can be bugs. 31 / 34
  35. 35. Our Experience with Fastest We have applied Fastest in the following projects (only up to abstract test case generation): On-board flight control software of a satellite launcher vehicle Along with Instituto de Aeron´utica e Espa¸o (IAE), Brazil. a c 139 lines of Z. One operation, 36 abstract test cases. Satellite communication protocols Two protocols, along with Instituto Nacional de Pesquisas Espaciais (INPE), Brazil. Two specifications comprising a total of 1,800 lines of Z. 26 operations, 195 abstract test cases. 32 / 34
  36. 36. Our Experience with Fastest (II) The ECSS-E-70-41A aerospace standard Along with INVAP, Argentina. A single specification of 2,011 lines of Z. Abstract test case translation to English. 256 test conditions. Many challenges to solve. Tokeneer First specification not written by us, 2,057 lines of Z. 139 schemas selected as operations. More than 25,000 test conditions, only DNF was applied. We had to modify the DNF algorithm due to “term explosion”. 33 / 34
  37. 37. Current and Future Work Continue with the SMT solvers. Z3. More satisfiable test specifications. A decision procedure for the Z notation? Test case refinement: greatly improve the implementation. Z specifications as program annotations. Implement the remaining steps of the TTF. Validate with more case studies. 34 / 34

×