Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Michaël Pilaeten
Reducing your testing efforts…
magically!
Magic?
3© 2013 CTG, Inc.
4© 2013 CTG, Inc.
Huh?
simplification
age
civil
status
residence
How many test cases do you need?
10© 2013 CTG, Inc.
Again….
age
civil
status
work
residence
savings
wage
ecological
children properties charity
All combinations
How many test cases do you need?
No negative tests
One test per combination
17© 2013 CTG, Inc.
5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10
= 36.288.000 test cases
Writing a test case: 4 minutes
Executing...
18© 2013 CTG, Inc.
One more thing….
Alternatives?
21© 2013 CTG, Inc.
Automation?
CHERRY
PICKING
26© 2013 CTG, Inc.
Bug research
Single mode Double mode
28© 2013 CTG, Inc.
The solution …
29© 2013 CTG, Inc.
Pairwise Testing
Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arr...
Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arr...
Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arr...
Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arr...
Non-Binary orthogonal arrays…
Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibili...
Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibili...
Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibili...
Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibili...
Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibili...
Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibili...
Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibili...
Catalogs of Non-Binary orthogonal arrays:
http://neilsloane.com/oadir/index.html
http://www.york.ac.uk/depts/maths/tables/...
James Bach
@jamesmarcusbach
www.satisfice.org
46© 2013 CTG, Inc.
47© 2013 CTG, Inc.
5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10
= 36.288.000 test cases
Writing a test case: 4 minutes
Executing...
48© 2013 CTG, Inc.
99,8% is not 100%
Different types of tests
Defect taxonomy
49© 2013 CTG, Inc.
REDUCE
FOCUS
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
How to reduce your test cases... magically!
Upcoming SlideShare
Loading in …5
×

How to reduce your test cases... magically!

664 views

Published on

Доклад Michael Pilaeten на конференции SQA Days-20. 24-26 ноября 2016. Минск
www.sqadays.com

Published in: Education
  • Be the first to comment

How to reduce your test cases... magically!

  1. 1. Michaël Pilaeten Reducing your testing efforts… magically!
  2. 2. Magic?
  3. 3. 3© 2013 CTG, Inc.
  4. 4. 4© 2013 CTG, Inc. Huh?
  5. 5. simplification
  6. 6. age civil status residence
  7. 7. How many test cases do you need?
  8. 8. 10© 2013 CTG, Inc. Again….
  9. 9. age civil status work residence savings wage ecological children properties charity
  10. 10. All combinations How many test cases do you need? No negative tests One test per combination
  11. 11. 17© 2013 CTG, Inc. 5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10 = 36.288.000 test cases Writing a test case: 4 minutes Executing a test case: 1 minute 36.288.000 test cases x 5 minutes = 181.440.000 minutes of testing = 378.000 man days of testing = 2016 man years of testing
  12. 12. 18© 2013 CTG, Inc. One more thing….
  13. 13. Alternatives?
  14. 14. 21© 2013 CTG, Inc.
  15. 15. Automation?
  16. 16. CHERRY PICKING
  17. 17. 26© 2013 CTG, Inc. Bug research
  18. 18. Single mode Double mode
  19. 19. 28© 2013 CTG, Inc. The solution …
  20. 20. 29© 2013 CTG, Inc. Pairwise Testing
  21. 21. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 Step 1: # TC = (# conditions + 1) Step 2: TC 1 all true Step 3: Left diagonal true Step 4: Remaining untrue 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 Only yes/no options
  22. 22. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 A) Each single condition value is tested B) Each couple of condition values is tested single mode defects double mode defects Only yes/no options
  23. 23. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 100% certainty of finding all combinatorial defects 25 or 2 x 2 x 2 x 2 x 2 = 32 -> 32 Test Cases Only yes/no options
  24. 24. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 100% certainty of finding all combinatorial defects 99,8% certainty of finding all combinatorial defects Only yes/no options 32 Test Cases 6 Test Cases
  25. 25. Non-Binary orthogonal arrays…
  26. 26. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities
  27. 27. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101
  28. 28. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82
  29. 29. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71
  30. 30. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71 62
  31. 31. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71 62 52
  32. 32. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71 62 52 32
  33. 33. Catalogs of Non-Binary orthogonal arrays: http://neilsloane.com/oadir/index.html http://www.york.ac.uk/depts/maths/tables/orthogonal.htm The Taguchi Orthogonal Array Selector: http://www.freequality.org/documents/tools/Tagarray_files/tamatrix.htm
  34. 34. James Bach @jamesmarcusbach www.satisfice.org
  35. 35. 46© 2013 CTG, Inc.
  36. 36. 47© 2013 CTG, Inc. 5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10 = 36.288.000 test cases Writing a test case: 4 minutes Executing a test case: 1 minute 36.288.000 test cases x 5 minutes = 181.440.000 minutes of testing = 2016 man years of testing Allpairs algorithm: = 97 test cases 97 test cases x 5 minutes = 485 minutes of testing = 1 man day of testing 1 man day versus 2016 man years ?
  37. 37. 48© 2013 CTG, Inc. 99,8% is not 100% Different types of tests Defect taxonomy
  38. 38. 49© 2013 CTG, Inc. REDUCE FOCUS

×