Доклад Michael Pilaeten на конференции SQA Days-20. 24-26 ноября 2016. Минск www.sqadays.com

1. Michaël Pilaeten
Reducing your testing efforts…
magically!

2. Magic?

3. 3© 2013 CTG, Inc.

4. 4© 2013 CTG, Inc.
Huh?

5. simplification

6. age
civil
status
residence

8. How many test cases do you need?

10. 10© 2013 CTG, Inc.
Again….

11. age
civil
status
work
residence
savings
wage
ecological
children properties charity

15. All combinations
How many test cases do you need?
No negative tests
One test per combination

17. 17© 2013 CTG, Inc.
5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10
= 36.288.000 test cases
Writing a test case: 4 minutes
Executing a test case: 1 minute
36.288.000 test cases x 5 minutes
= 181.440.000 minutes of testing
= 378.000 man days of testing
= 2016 man years of testing

18. 18© 2013 CTG, Inc.
One more thing….

20. Alternatives?

21. 21© 2013 CTG, Inc.

22. Automation?

23. CHERRY
PICKING

26. 26© 2013 CTG, Inc.
Bug research

27. Single mode Double mode

28. 28© 2013 CTG, Inc.
The solution …

29. 29© 2013 CTG, Inc.
Pairwise Testing

31. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arrays are easy !
1
1
1
1
1
1
1
1
1
1
Step 1: # TC = (# conditions + 1)
Step 2: TC 1 all true
Step 3: Left diagonal true
Step 4: Remaining untrue
0
0
0
0
0 0 0 0
0 0 0
0 0 0
0 0 0
000
Only yes/no options

32. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arrays are easy !
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0 0 0 0
0 0 0
0 0 0
0 0 0
000
A) Each single condition value is tested
B) Each couple of condition values is tested
single mode
defects
double mode
defects
Only yes/no options

33. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arrays are easy !
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0 0 0 0
0 0 0
0 0 0
0 0 0
000
100% certainty of finding all combinatorial defects
25 or 2 x 2 x 2 x 2 x 2 = 32 -> 32 Test Cases
Only yes/no options

34. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Binary orthogonal arrays are easy !
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0 0 0 0
0 0 0
0 0 0
0 0 0
000
100% certainty of finding all combinatorial defects
99,8% certainty of finding all combinatorial defects
Only yes/no options
32 Test Cases
6 Test Cases

35. Non-Binary orthogonal arrays…

36. Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibilities
Savings:
8 possibilities
Residence:
3 possibilities
Ecological:
10 possibilities
Wage:
7 possibilities
Charity:
8 possibilities

37. Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibilities
Savings:
8 possibilities
Residence:
3 possibilities
Ecological:
10 possibilities
Wage:
7 possibilities
Charity:
8 possibilities
We need this* orthogonal array:
* or larger
L 101

38. Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibilities
Savings:
8 possibilities
Residence:
3 possibilities
Ecological:
10 possibilities
Wage:
7 possibilities
Charity:
8 possibilities
We need this* orthogonal array:
* or larger
L 101 82

39. Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibilities
Savings:
8 possibilities
Residence:
3 possibilities
Ecological:
10 possibilities
Wage:
7 possibilities
Charity:
8 possibilities
We need this* orthogonal array:
* or larger
L 101 82 71

40. Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibilities
Savings:
8 possibilities
Residence:
3 possibilities
Ecological:
10 possibilities
Wage:
7 possibilities
Charity:
8 possibilities
We need this* orthogonal array:
* or larger
L 101 82 71 62

41. Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibilities
Savings:
8 possibilities
Residence:
3 possibilities
Ecological:
10 possibilities
Wage:
7 possibilities
Charity:
8 possibilities
We need this* orthogonal array:
* or larger
L 101 82 71 62 52

42. Age:
3 possibilities
Civil status:
5 possibilities
Children:
6 possibilities
Properties:
6 possibilities
Work:
5 possibilities
Savings:
8 possibilities
Residence:
3 possibilities
Ecological:
10 possibilities
Wage:
7 possibilities
Charity:
8 possibilities
We need this* orthogonal array:
* or larger
L 101 82 71 62 52 32

43. Catalogs of Non-Binary orthogonal arrays:
http://neilsloane.com/oadir/index.html
http://www.york.ac.uk/depts/maths/tables/orthogonal.htm
The Taguchi Orthogonal Array Selector:
http://www.freequality.org/documents/tools/Tagarray_files/tamatrix.htm

45. James Bach
@jamesmarcusbach
www.satisfice.org

46. 46© 2013 CTG, Inc.

47. 47© 2013 CTG, Inc.
5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10
= 36.288.000 test cases
Writing a test case: 4 minutes
Executing a test case: 1 minute
36.288.000 test cases x 5 minutes
= 181.440.000 minutes of testing
= 2016 man years of testing
Allpairs algorithm:
= 97 test cases
97 test cases x 5 minutes
= 485 minutes of testing
= 1 man day of testing
1 man day versus 2016 man years ?

48. 48© 2013 CTG, Inc.
99,8% is not 100%
Different types of tests
Defect taxonomy

49. 49© 2013 CTG, Inc.
REDUCE
FOCUS