How to use pairwise testing on your projects. The main idea. Mechanism of calculation. Secrets and tools. PICT. Testing Techniques in Software Testing Orthogonal arrays ACTs

1. E Pluribus Unum
or
How to use Pairwise Testing
VLADIMIR ARUTIN

2. What is the purpose of test design techniques ?
Boundary Value Analysis
Equivalence Class Partitioning
Decision Table testing
Cause-Effect Graphing Technique
Error Guessing
Pairwise testing
State transition testing
Use case testing
User story testing
Domain analysis
Syntax testing
Combining technique
Exhaustive Testing

3. THE MAIN PURPOSE IS
TO ACHIEVE AS MUCH COVERAGE
AS POSSIBLE
USING AS LESS TESTS
AS POSSIBLE

4. Divide
Et
Impera

5. Example#1 Online Shop

6. Combinatorial explosion

8. You can split all checkboxes
into groups

9. What if there is a dependency
between сheckboxes?

10. LOAN TYPE COMMISSION
CASH LOAN 500
CREDIT CARD 1000
4 tests
100% coverage
10 minutes
Example#2 Bank Loan

11. But Life Isn’t Easy…
LOAN TYPE Limit ROI COMMISSION M Class Outstanding
CASH LOAN 10000 10 500 6 A 1
CREDIT CARD 20000 30 1000 9 B 2
50000 50 12 C 3
100000 18 D 4
120000 24 …
150000 57
190000 58
200000 59
60

12. How many tests do we need to achieve
minimum coverage?

13. OUR CUSTOMER FOUND A BUG
If Class is D and Period is 24 months
Bank approves 200 000 loan amount
BUT should REFUSE!
AND WE HAVE 40% OF SUCH CLIENTS

14. LOAN TYPE Limit ROI COMMISSION M Class Outstanding
CASH LOAN 10000 10 500 6 A 1
CREDIT CARD 20000 30 1000 9 B 2
50000 50 12 C 3
100000 18 D 4
120000 24 …
150000 57
190000 58
200000 59
60
How many tests will be failed?
5760 tests

15. How many test inputs are needed to
trigger defects in production?
84%
11%
4%
1%
1 input or 2 inputs (pairwise)
3 inputs
4 inputs
5 or 6 inputs

16. Let’s optimize our data
• Combine them into Equivalence Classes
• Use Boundary Values
LOAN TYPE Limit ROI COMMISSION Y Class Outstanding
CASH LOAN 10000 10 500 <1 A 1
CREDIT CARD 20000 30 1000 1 B 2
190000 50 >1 C 59
200000 D 60

17. Combine Pairs
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000
1 20000
1 190000
1 200000
2 10000
2 20000
2 190000
2 200000
59 10000
59 20000
59 190000
59 200000
60 10000
60 20000
60 190000
60 200000

18. Combine Pairs
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000 A
1 20000 B
1 190000 C
1 200000 D
2 10000 B
2 20000 A
2 190000 D
2 200000 C
59 10000 D
59 20000 C
59 190000 A
59 200000 B
60 10000 C
60 20000 D
60 190000 B
60 200000 A

19. Combine Pairs
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000 A <1
1 20000 B 1
1 190000 C >1
1 200000 D <1
2 10000 B 1
2 20000 A >1
2 190000 D 1
2 200000 C <1
59 10000 D >1
59 20000 C <1
59 190000 A 1
59 200000 B >1
60 10000 C 1
60 20000 D >1
60 190000 B <1
60 200000 A 1

20. Combine Pairs
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000 A <1 30
1 20000 B 1 50
1 190000 C >1 50
1 200000 D <1 10
2 10000 B 1 10
2 20000 A >1 50
2 190000 D 1 50
2 200000 C <1 10
59 10000 D >1 50
59 20000 C <1 30
59 190000 A 1 10
59 200000 B >1
60 10000 C 1 10
60 20000 D >1 10
60 190000 B <1 30
60 200000 A 1 50

21. Combine Pairs
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000 A <1 30 CASH LOAN
1 20000 B 1 50 CREDIT CARD
1 190000 C >1 50
1 200000 D <1 10
2 10000 B 1 10 CASH LOAN
2 20000 A >1 50 CREDIT CARD
2 190000 D 1 50 CREDIT CARD
2 200000 C <1 10 CASH LOAN
59 10000 D >1 50 CREDIT CARD
59 20000 C <1 30
59 190000 A 1 10 CASH LOAN
59 200000 B >1 CREDIT CARD
60 10000 C 1 10 CREDIT CARD
60 20000 D >1 10 CASH LOAN
60 190000 B <1 30 CREDIT CARD
60 200000 A 1 50 CASH LOAN

22. Combine Pairs
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000 A <1 30 CASH LOAN 500
1 20000 B 1 50 CREDIT CARD 1000
1 190000 C >1 50 500
1 200000 D <1 10
2 10000 B 1 10 CASH LOAN 1000
2 20000 A >1 50 CREDIT CARD 500
2 190000 D 1 50 CREDIT CARD 500
2 200000 C <1 10 CASH LOAN 1000
59 10000 D >1 50 CREDIT CARD 1000
59 20000 C <1 30 1000
59 190000 A 1 10 CASH LOAN 1000
59 200000 B >1 CREDIT CARD 500
60 10000 C 1 10 CREDIT CARD 1000
60 20000 D >1 10 CASH LOAN
60 190000 B <1 30 CREDIT CARD 500
60 200000 A 1 50 CASH LOAN 1000

23. 106 pairs, 16 tests, about 30 minutes
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000 A <1 30 CASH LOAN 500
1 20000 B 1 50 CREDIT CARD 1000
1 190000 C >1 50 500
1 200000 D <1 10
2 10000 B 1 10 CASH LOAN 1000
2 20000 A >1 50 CREDIT CARD 500
2 190000 D 1 50 CREDIT CARD 500
2 200000 C <1 10 CASH LOAN 1000
59 10000 D >1 50 CREDIT CARD 1000
59 20000 C <1 30 1000
59 190000 A 1 10 CASH LOAN 1000
59 200000 B >1 CREDIT CARD 500
60 10000 C 1 10 CREDIT CARD 1000
60 20000 D >1 10 CASH LOAN
60 190000 B <1 30 CREDIT CARD 500
60 200000 A 1 50 CASH LOAN 1000

24. Let’s add some conditions
• IF ROI < 50% THEN LOAN CAN’T BE 200000
• IF CLASS < B THEN PERIOD <1 YEAR
• IF CASH LOAN THEN COMMISSION=1000
OUTSTANDING LIMIT CLASS PERIOD ROI LOAN TYPE COMMISSION
1 10000 A <1 30 CASH LOAN 500
1 20000 B 1 50 CREDIT CARD 1000
1 190000 C >1 50 500
1 200000 D <1 10
2 10000 B 1 10 CASH LOAN 1000
2 20000 A >1 50 CREDIT CARD 500
2 190000 D 1 50 CREDIT CARD 500
2 200000 C <1 10 CASH LOAN 1000
59 10000 D >1 50 CREDIT CARD 1000
59 20000 C <1 30 1000
59 190000 A 1 10 CASH LOAN 1000
59 200000 B >1 CREDIT CARD 500
60 10000 C 1 10 CREDIT CARD 1000
60 20000 D >1 10 CASH LOAN
60 190000 B <1 30 CREDIT CARD 500
60 200000 A 1 50 CASH LOAN 1000
No, God,
please….

25. USE AVAILABLE TOOLS

26. Pairwise Independent Combinatorial Testing
(PICT)
NOW MY LIFE WILL
NEVER BE THE
SAME…

27. REQUIREMENTS
COMPANY: Microsoft, Apple, Samsung, BlackBerry, Xiaomi, Sony
SCREEN: 4.5', 5', 5.2', 5.5', 6', 6.9‘
RESOLUTION: HD, FullHD, 4K
RAM: 2Gb, 3Gb, 4Gb, 6Gb, 8Gb
PROCESSOR MAKE: QS650, MSM8230, QS820, A10, A11
OS: Android, BlackBerry OS, iOS, Windows Phone
NET STANDART: CDMA, GSM, LTE, 3G
STORAGE: 16Gb, 32Gb, 64Gb, 128Gb, 512Gb
Example#3 Mobile Game

28. REQUIREMENTS
COMPANY: Microsoft, Apple, Samsung, BlackBerry, Xiaomi, Sony
SCREEN: 4.5', 5', 5.2', 5.5', 6', 6.9‘
RESOLUTION: HD, FullHD, 4K
RAM: 2Gb, 3Gb, 4Gb, 6Gb, 8Gb
PROCESSOR MAKE: QS650, MSM8230, QS820, A10, A11
OS: Android, BlackBerry OS, iOS, Windows Phone
NET STANDART: CDMA, GSM, LTE, 3G
STORAGE: 16Gb, 32Gb, 64Gb, 128Gb, 512Gb
Example#3 Mobile Game
(6)
(6)
(3)
(5)
(5)
(4)
(4)
(5)
MINIMUM TEST AMOUNT=6 x 6 x 3 x 5 x 5 x 4 x 4 x 5= 216000

29. CONDITIONS AND DEPENDENCIES
Can be used: =, <>, >, >=, <, <=, and LIKE
Operator IN The IF, THEN, and ELSE parts of a
predicate may contain multiple terms joined
by logical operators: NOT, AND, and OR.
IF [RAM] in {"2Gb", "3Gb", "4Gb"} OR ([COMPANY] =
"Xiaomi" AND [STORAGE] < "64Gb") THEN [PROCESSOR
MAKE] = "QS650“;
IF [Company] ="Apple" THEN [PROCESSOR
MAKE] IN {"A10", "A11"};
IF[Company] ="Samsung" THEN [SCREEN]= "5'";

31. Conclusion
• The number of bugs detected by pairwise testing is bigger than
found by checking all values for each parameter.
• Pairwise testing reduces the number of test cases needed
for system checking.
• It accelerates the testing process itself.
• It takes 50 percent less efforts.