Advertisement

Report

May. 9, 2012•0 likes## 4 likes

•1,873 views## views

Be the first to like this

Show More

Total views

0

On Slideshare

0

From embeds

0

Number of embeds

0

May. 9, 2012•0 likes## 4 likes

•1,873 views## views

Be the first to like this

Show More

Total views

0

On Slideshare

0

From embeds

0

Number of embeds

0

Download to read offline

Report

Technology

Education

Advertisement

- Fastest Test Case Generation from Z Speciﬁcations Maximiliano Cristi´ a CIFASIS Universidad Nacional de Rosario Rosario – Argentina cristia@cifasis-conicet.gov.ar Alpen-Adria Univesit¨t – Klagenfurt – March, 2012 a
- Software Testing Basics Program Correctness A program is correct if it behaves according to its speciﬁcation Edsger Dijkstra “Program testing can be a very eﬀective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” Donald Knuth “Beware of bugs in the above code; I have only proved it correct, not tried it.” 2 / 34
- Model-Based Testing (MBT) MBT analyses a model of the MBT Process program to derive test cases which requeriments later are provided as input. At the beginning, the model is used model error? to generate test cases. verify generate At end, the model becomes the abstract abstract oracle. tests results If a model is provided, the process abstract is almost automatic. reﬁne execute tests results 3 / 34
- The Test Template Framework and Fastest The Test Template Framework (TTF) A particular method of MBT which uses Z speciﬁcations. Unit testing. Stocks and Carrington, A Framework for Speciﬁcation-Based Testing, IEEE TOSE 1996. Fastest It is the only existing implementation of the TTF. Available at www.fceia.unr.edu.ar/~mcristia. Cristi´ and Rodr´ a ıguez Monetti, Implementing and Applying the Stocks-Carrington Framework for MBT, ICFEM 2009. 4 / 34
- TTF Automation The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Reﬁne abstract test cases. reﬁne abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors. 5 / 34
- An Example The Requirements A bank has a certain number of savings accounts. Customers can deposit and withdraw money. Each account is identiﬁed with an account number. An amount of money is considered to be a natural number. Only the deposit operation is going to be shown. 6 / 34
- The Z Speciﬁcation Types and State Space [ACCNUM] A type for account numbers BALANCE == N A type for balances AMOUNT == N A type for amounts of money Bank sa : ACCNUM → BALANCE 7 / 34
- The Z Speciﬁcation The Deposit Operation DepositOk ∆Bank n? : ACCNUM a? : AMOUNT n? ∈ dom sa a? > 0 sa = sa ⊕ {n? → sa n? + a?} DepositE 1 == [ΞBank; n? : ACCNUM | n? ∈ dom sa] / DepositE 2 == [ΞBank; a? : Z | a? ≤ 0] Deposit == DepositOk ∨ DepositE 1 ∨ DepositE 2 loadspec bank.tex selop Deposit 8 / 34
- The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree. 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Reﬁne abstract test cases. reﬁne abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
- Domain Partition Given a Z operation, take its input space and get a partition of it. input testing input testing input testing space tactic space tactic space tactic P Q P P1 Q Q1 atomic predicates Each member of a partition is given by a Z predicate (i.e., ﬁrst-order logic and Zermelo-Fraenkel set theory). Partitioning can be represented as a tree. Test cases are derived only from the “leaves” Leaves are called test conditions. 10 / 34
- Testing Tactics The TTF performs domain partition by applying testing tactics. Disjunctive Normal Form (DNF) Write the operation predicate in DNF and take the precondition of each disjunct. Standard Partition (SP) for R ⊕ G R = {}, G = {} R = {}, G = {} R = {}, G = {} R = {}, G = {}, dom R = dom G R = {}, G = {}, dom G ⊂ dom R R = {}, G = {}, (dom R ∩ dom G ) = {} R = {}, G = {}, dom R ⊂ dom G R = {}, G = {}, (dom R ∩ dom G ) = {}, ¬ (dom G ⊆ dom R), ¬ (dom R ⊆ dom G ) 11 / 34
- Applying DNF and SP to Deposit First DNF DNF == [Deposit VIS | n? ∈ dom sa ∧ a? > 0] Deposit1 DNF == [Deposit VIS | n? ∈ dom sa] Deposit2 / DNF == [Deposit VIS | a? ≤ 0] Deposit3 DNF Second SP, but only to Deposit1 SP Deposit1 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit2 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit3 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit4 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {} ∧ dom sa = dom{n? → (sa n? + a?)}] SP DNF Deposit5 == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {} ∧ dom{n? → (sa n? + a?)} ⊂ dom sa] (three more test conditions not shown here) genalltt addtactic Deposit_DNF_1 SP oplus sa oplus {(n? , sa~n? + a?)} 12 / 34
- The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Reﬁne abstract test cases. reﬁne abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
- Pruning the Testing Tree Pruning involves eliminating unsatisﬁable test conditions. Before pruning After pruning Deposit VIS Deposit VIS Deposit DNF 1 Deposit DNF 1 Deposit SP 1 Deposit SP 4 Deposit SP 2 Deposit SP 5 Deposit SP 3 Deposit DNF 2 Deposit SP 4 Deposit DNF 3 Deposit SP 5 Deposit SP 6 prunett Deposit SP 7 Deposit SP 8 Deposit DNF 2 Deposit DNF 3 14 / 34
- Pruning in Detail The problem – I Many test conditions are unsatisﬁable. Most of them are mathematical contradictions. Then, no test cases can be derived from them. Therefore, unsatisﬁable test conditions should be pruned from testing trees. In general, this problem is undecidable. The problem – II Z includes many mathematical theories (sets, relations, partial functions, sequences, etc.), making the problem harder. 15 / 34
- Detecting Contradictions Detecting logical contradictions Since every test condition is a conjunction of atomic predicates, then the only possible logical contradiction is of the form: ··· ∧ p ∧ ··· ∧ ¬ p ∧ ... This is the easy part. Detecting mathematical contradictions Most of the contradictions are of a mathematical nature. Some depend on the particular semantics of the Z mathematical theories. 16 / 34
- A Library of Mathematical Contradictions The core of our method is a library of elimination theorems. Each elimination theorem should represent a contradiction. ETheorem ETheoremName [parameters] atomicPred ............... atomicPred parameters is roughly any Z declaration. atomicPred is any Z atomic predicate and some keywords. Subtyping and equivalence rules. A User extensible (LTEX). No deduction, only pattern-matching. 17 / 34
- Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
- Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
- Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
- The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Reﬁne abstract test cases. reﬁne abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
- Generating Abstract Test Cases Find constants satisfying test conditions. TC Deposit1 == SP [Deposit4 | n? = accnum2 ∧ sa = {(accnum2 , 1)} ∧ a? = 1] TC Deposit2 == SP [Deposit5 | n? = accnum2 ∧ sa = {(accnum2 , 1), (accnum1 , 1)} ∧ a? = 1] TC DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0] Deposit3 == [Deposit2 2 TC DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0] Deposit4 == [Deposit3 2 genalltca 20 / 34
- Test Case Generation in Detail Currently, Fastest implements a rough satisﬁability algorithm: 1 Finite sets are bound to variables and types in a test template. ACCNUM, n? → {accnum0 , accnum1 , accnum2 } AMOUNT , BALANCE , a? → {0, 1, 2} ACCNUM → BALANCE , sa → {∅, {accnum2 → 1}, {accnum2 → 1, accnum1 → 1}, ...} 2 The predicate is evaluated on some elements of the Cartesian product of those ﬁnite sets. ZLive (from CZT) is used as evaluator. Some simple heuristics are applied. The mathematical structure of the predicate is not considered. 21 / 34
- Is it good enough? Fastest has been able to ﬁnd test cases for an average of 80 % of the satisﬁable test conditions from eleven case studies. For the remaining 20 %, Fastest provides a command to help it to ﬁnd a test case. setfinitemodel Deposit_SP_4 -fm "sa=={{accnum_2 mapsto 1}}" Nevertheless, Fastest’s algorithm needs to be improved or replaced by more sophisticated ones. 22 / 34
- Diﬃculties with Test Case Generation in the TTF The ideas of test templates and of automatic partition of Z schema have been proposed several years ago. Their actual application to systems of industrial size was problematic due to the lack of eﬃcient and adequate constraint solvers and theorem provers. Given the progresses recently achieved for these kinds of tools, these techniques are worth revisiting . . . Ana Cavalcanti and Marie-Claude Gaudel (2011): Testing for reﬁnement in Circus. Acta Informatica, 48(2). 23 / 34
- SMT Solvers for the TTF Satisﬁability Modulo Theory (SMT) solvers SMT solvers are tools that solve the problem of satisﬁability for a range of mathematical and logical theories. No SMT solver supports the Zermelo-Fraenkel set theory. Must deﬁne a shallow embedding. Many questions: Is the language of SMT solvers expressive enough? Is there just one shallow embedding? Will the chosen embedding solve all the satisﬁable test speciﬁcations appearing in the TTF and real Z speciﬁcations? Which SMT solver and which embedding will be the best in satisfying more test speciﬁcations in less time? Will it be better than Fastest in ﬁnding test cases? Or should the SMT solver complement Fastest in this task? 24 / 34
- First Results Two shallow embeddings We have deﬁned a shallow embedding of the Zermelo-Franekel set theory for two mainstream SMT solvers: Yices and CVC3. Initial empirical assessment We manually translated 69 satisﬁable test templates, for which Fastest cannot ﬁnd a test case, into these shallow embeddings and ran the SMT solvers over them. 25 / 34
- Empirical assessment Main embeddings Variant embeddings Case study Yices CVC3 Yices CVC3 Sat Unk Sat Unk Sat Unk Sat Unk Savings accounts (3) 8 8 8 8 Savings accounts (1) 2 2 2 2 Launching vehicle 8 8 8 8 Plavis 29 29 29 29 SWPDC 13 13 13 13 Scheduler 4 4 4 4 Security class 4 4 4 4 Pool of sensors 1 1 1 1 Totals 9 60 17 52 9 60 17 52 Time spent in processing all the 69 test templates: Yices: 3 s CVC3: 420 s Fastest: 390 s (but no test case is discovered) 26 / 34
- The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Reﬁne abstract test cases. reﬁne abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
- Reﬁning Abstract Test Cases The implementation (C programming language) Assume sa is implemented as follows: struct sanode {int an; float bal; struct sanode *n;} *sa; where sa is a global variable. Besides, Deposit is implemented by the following function: int deposit(int n, float a) Reﬁnement rules We need to link each speciﬁcation variable to their implementation counterparts. This process is called abstract test case reﬁnement. Write reﬁnement rules and then run the refine command. 28 / 34
- Reﬁning Abstract Test Cases (II) A reﬁnement rule for Deposit @RRULE deposit @PREAMBLE #include <bank.h> @LAWS l1:n? ==> n l2:a? ==> a l3:sa ==> sa AS LIST[SLL,n] WITH[sa.@dom ==> sa.an, sa.@ran ==> sa.bal] @UUT deposit(n, a) loadrefrule deposit.txt refine Deposit to test /bank/src implemented in C with deposit 29 / 34
- Concrete Test Cases TC Reﬁnement of Deposit2 . #include <bank.h> int main() { int n = 345; n? = accnum2 float a = 1; a? = 1 sa = {(accnum2, 1), (accnum1, 1)} struct sanode sanode0 = {876,1,NULL}; struct sanode sanode1 = {345,1,NULL}; sa = &sanode0; sanode0.n = &sanode1; deposit(n,a); return 1; } 30 / 34
- The Remaining Steps Before executing a concrete test case it is necessary to insert code to get the program output. Abstraction rules, similar to reﬁnement rules. Once a test case has been executed its output must be abstracted. Same abstraction rules, diﬀerent command. The ﬁnal veriﬁcation is performed with another command. (Optional) Translating abstract test cases into natural language involves writing translation rules. Quite diﬀerent from reﬁnement rules. Work in progress Fastest is still a prototype: many of these features are roughly implemented and there can be bugs. 31 / 34
- Our Experience with Fastest We have applied Fastest in the following projects (only up to abstract test case generation): On-board ﬂight control software of a satellite launcher vehicle Along with Instituto de Aeron´utica e Espa¸o (IAE), Brazil. a c 139 lines of Z. One operation, 36 abstract test cases. Satellite communication protocols Two protocols, along with Instituto Nacional de Pesquisas Espaciais (INPE), Brazil. Two speciﬁcations comprising a total of 1,800 lines of Z. 26 operations, 195 abstract test cases. 32 / 34
- Our Experience with Fastest (II) The ECSS-E-70-41A aerospace standard Along with INVAP, Argentina. A single speciﬁcation of 2,011 lines of Z. Abstract test case translation to English. 256 test conditions. Many challenges to solve. Tokeneer First speciﬁcation not written by us, 2,057 lines of Z. 139 schemas selected as operations. More than 25,000 test conditions, only DNF was applied. We had to modify the DNF algorithm due to “term explosion”. 33 / 34
- Current and Future Work Continue with the SMT solvers. Z3. More satisﬁable test speciﬁcations. A decision procedure for the Z notation? Test case reﬁnement: greatly improve the implementation. Z speciﬁcations as program annotations. Implement the remaining steps of the TTF. Validate with more case studies. 34 / 34

Advertisement