Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
×

Graph Methods for Generating Test Cases with Universal and Existential Constraints (Talk @ ICTSS 2015)

451 views

Published on

We introduce a generalization of the t -way test case generation problem, where parameter t is replaced by a set Φ of Boolean conditions on attribute values. We then present two reductions of this problem to graphs; first, to graph colouring, where we link the the minimal number of tests to the chromatic number of some graph; second, to hypergraph vertex covering. This latter formalization allows us to handle problems with constraints of two kinds: those that must be true for every generated test case, and those that must be true for at least one test case. Experimental results show that the proposed solution produces test suites of slightly smaller sizes than a range of existing tools, while being more general: to the best of our knowledge, our work is the first to allow existential constraints over test cases.

Published in: Technology
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Be the first to comment

• Be the first to like this

Graph Methods for Generating Test Cases with Universal and Existential Constraints (Talk @ ICTSS 2015)

1. 1. Graph Methods for Generating Test Cases with Universal and Existential Constraints Sylvain Hallé, Edmond La Chance and Sébastien Gaboury Université du Québec à Chicoutimi, Canada http://www.liﬂab.ca November 23rd, 2015
2. 2. Example: How to test the system in such a way that we cover all possible interactions between two inputs? can each be in position 1, 2 or 3, ,
3. 3. Naïve method: enumerate all combinations of values All combinations of values for any two inputs are present How many test cases are there? 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3
4. 4. In fact, this table veriﬁes all combinations of 3 inputs exactly once... but combinations of 2 appear more than once Can we observe them all by doing fewer tests? 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3
5. 5. Here's how: Each combination of values for 2 given parameters appears the same number of times (here, once) We call this an orthogonal array (OA) 3 2 1 2 1 3 1 3 2 1 2 3 1 2 3 1 2 3 1 1 1 2 2 2 3 3 3 How many test cases this time?
6. 6. Pairwise testing Testing strategy that identiﬁes interactions between two input parameters, and tests all combinations of values for the selected pairs Intuition: some problems occur only through an interaction between many (e.g. 2) parameters By varying each parameter individually, it is unlikely to generate the pair causing the error We can use orthogonal arrays to get the test cases to try
7. 7. Exemple of an interaction: The error occurs only when altitude_adj == 0 AND volume < 2.2 if (altitude_adj == 0) { // do something if (volume < 2.2) { faulty code! BOOM! } else { good code, no problem } } else { // do something else }
8. 8. 10 eﬀects, each with 2 possible values (on/oﬀ) All combinations: 210 = 1024 tests How many tests to cover all interactions of 3 eﬀects?
9. 9. There are = 120 triplets of eﬀects For each triplet, we have to test 23 = 8 combinations of values. We need a maximum of 960 tests. But each test exercises three triplets; hence we need no more than 320 test to cover everything... 0 1 1 0 1 0 1 0 1 0 ...and in fact, each test covers much more than three triplets 10 3( ) Quel est le nombre minimum de tests?
10. 10. Actually, only 13 tests are necessary! Is it an orthogonal array? 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 0 0 0 1 1 0 1 1 0 1 0 1 0 0 1 0 0 0 1 1 1 0 0 0 0 1 1 0 0 1 0 0 1 0 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 1 1 0 1
11. 11. Covering array An array CA(N, k, v, t) such that: The previous array is CA(13, 10, 2, 3) The array contains N lines We consider k parameters Each parameter can have v values Each combination of values for t parameters occurs at least once in the array t-way test case generation
12. 12. Back to our example: But... can each be in position 1, 2 or 3, , must always be greater than and can never be at position 1 at the same time
13. 13. Back to our example: But... can each be in position 1, 2 or 3, , must always be greater than and can never be at position 1 at the same time Universal constraints
14. 14. Universal constraint Constraint on the possible values for a test case Every test case must follow the constraint to be considered valid We wish to ﬁnd the smallest covering array CA(N, k, v, t), for a given set of constraints May lead to a smaller or a larger covering array, depending on the constraints Example: mutually exclusive parameters (subscript/superscript)
15. 15. Back to our example: Also... The test suite should include a case where all inputs are diﬀerent At least one test should try and at the same position can each be in position 1, 2 or 3, ,
16. 16. Back to our example: Also... The test suite should include a case where all inputs are diﬀerent At least one test should try and at the same position can each be in position 1, 2 or 3, , Existential constraints
17. 17. Existential constraint Constraint on the possible values for a test case At least one test case must follow the constraint for the test suite to be considered valid We wish to ﬁnd the smallest covering array CA(N, k, v, t), for a given set of constraints May lead to a smaller or a larger covering array, depending on the constraints Same as before
18. 18. Why use existential constraints? Three scenarios (at least): See the paperb Extending an existing test suite (add one condition for each existing test) Relaxed t-way (Mone combination suﬀicesM) MC/DC testing
19. 19. Let D1, D2, ..., Dk be the domains for k parameters named p1, p2, ..., pk Let Φ = {φ1, φ2, ..., φm} be a set of m arbitrary Boolean expressions, whose ground terms are of the form pi = d, or pi = pj, for d ∈ Di Let Σ ⊆ D1 × D2 × ... × Dk be a set of value assignments for each parameter We call Σ a Φ-way covering if, for every φ ∈ Φ, there exists an assignment σ ∈ Σ such that σ |= φ Φ is the set of existential constraints Σ is the set of test cases
20. 20. Back to our example: Also... The test suite should include a case where all inputs are diﬀerent At least one test should try and at the same position can each be in position 1, 2 or 3, ,
21. 21. Back to our example: Also... The test suite should include a case where all inputs are diﬀerent At least one test should try and at the same position D1 = D2 = D3 = {1,2,3}
22. 22. Back to our example: Also... The test suite should include a case where all inputs are diﬀerent p2 = p3 D1 = D2 = D3 = {1,2,3}
23. 23. Back to our example: Also... p1 ≠ p2 ∧ p2 ≠ p3 ∧ p1 ≠ p3 p2 = p3 D1 = D2 = D3 = {1,2,3}
24. 24. Any t-way problem can be converted into a Φ-way problem... Let I = {S ⊆ [1,k] : |S| = t} Let DS = Di Deﬁne Φ as: ∩ S ∈ I {d1,...,dn} ∈ DS ∏ i ∈ S ∩ pi = di ∧i ∈ S
25. 25. How do we ﬁnd a Φ-way covering? Two solutions based on reductions to graph problems Graph colouring Hypergraph vertex covering
26. 26. Reduction to graph colouring Create the graph G = <V,E> such that: Let κ : V → [1,n] be a colouring of G with n colours. Deﬁne σκ,i as any variable assignment satisfying: Then Σ = σκ,i is a Φ-way covering (theorem) V = Φ E = { (φ,φ') : φ ∧ φ' is unsatisﬁable } φ ∧φ ∈ κ-1 (i) ∩ i ∈ [1,n]
27. 27. Informally... Create one vertex per constraint Link pairs of vertices that cannot be true at the same time Find a colouring for this graph Create a test case for each colour (take the conjunction of all conditions for nodes ot this colour)
28. 28. Example with a = 0 ∧ b = 0, a = 0 ∧ b = 1, a = 1 ∧ b = 0, a = 1 ∧ b = 1, a = 0 ∧ c = 0, a = 0 ∧ c = 1, a = 1 ∧ c = 0, a = 1 ∧ c = 1, b = 0 ∧ c = 0, b = 0 ∧ c = 1, b = 1 ∧ c = 0, b = 1 ∧ c = 1 } Φ = { a=0 ∧ b=0 a=1 ∧ b=0 a=1 ∧ b=1 a=1 ∧ c=0 a=1 ∧ c=1 a=0 ∧ b=1 b=1 ∧ c=0 b=1 ∧ c=1 b=0 ∧ c=0 b=0 ∧ c=1 a=0 ∧ c=0 a=0 ∧ c=1 Pairwise testing for a, b, c ∈ {0,1} a=1,b=0,c=1 a=1,b=0,c=0 a=1,b=1,c=0 a=1,b=1,c=1 a=0,b=1,c=1 Test cases
29. 29. Some remarks: The conjunction of all conditions of the same colour must be satisﬁable. Counter-example: Classical t-way problems (and many others) always satisfy this constraint The minimum number of tests is the chromatic number of G ⇒ "Guaranteed" lower bound a=0 ∨ b=0 a=1 b=1 1. 2. 3.
30. 30. Reduction to hypergraph vertex covering Create the hypergraph G = <V,E> such that: Let Σ ⊆ V be a vertex covering of G. Then Σ is a Φ-way covering (theorem) V = D1 × D2 × ... × Dk E = {S ⊆ V : there exists φ ∈ Φ such that s ∈ S iﬀ s |= φ} Works all the time! Can even deal with universal constraints (remove vertices that violate any of them)
31. 31. Example with a = 0, b = 0, a ≠ 0 ∨ b ≠ 0 }Φ = { a=0,b=0 Test cases a=0, b=0 a=0, b=1 1 a=1, b=0 2 3 a=1, b=1 3 3 a=1,b=1
32. 32. a : 0, 1, 2 b : 0, 1 c : 2, 3, 4, 5 Once a != b Once a < b Always !(a > b) || c == 2 Both techniques implemented in a test case generation tool Only known tool to handle existential AND universal constraints Uses a simple extension of the input format used by PICT Domains Existential constraints Universal constraints
33. 33. t=2 AllPairs QICT t >2 E A Vertex cover TCases Jenny Forbidden tuples Colouring
34. 34. 5 10 15 20 25 30 35 40 45 50 55 2 3 4 5 6 7 8 9 Testsize n Test size with t:2, D:3 QICT Colouring AllPairs TCases Jenny hypergraph For pairwise test case generation...
35. 35. With t = 3... 20 30 40 50 60 70 80 90 100 110 120 3 4 5 6 7 8 Testsize n Test size with t:3, D:3 Colouring TCases Jenny Hitting Fewer tools remain...
36. 36. With universal constraints... With existential constraints... Still fewer tools... Hypergraph 60 60 118 TCases 158 203 254 n 5 6 7 Hypergraph 65 66 110 n 5 6 7 Last one standing!
37. 37. Existential constraints in test case generation arise in a variety of situations, but are not addressed by existing tools/research Finding an optimal test suite can be reduced to ﬁnding the optimal solution of two well-known graph problems Empirically, existing heuristics for these problems perform at least as well as a variety of existing tools https://bitbucket.org/ sylvainhalle/gcases Take-home points...