A basic problem in software testing often is choosing a subset from the near infinite number of possible test cases. Consider the challenges of testing multiple browsers, multiple mobile devices, mobile applications, or use case paths. Testers must select test cases to design, create, and then execute to obtain sufficient coverage—all while managing the time it takes to test relative to risks. Even though test resources are limited, you still want to select the best possible set of tests. Peter Kruse shares his experiences designing test cases with TESTONA, the most popular tool for systematic test design of classification tree-based tests. Peter shows how to integrate expected test outcomes and how to obtain executable test scripts directly from the test specification or user stories. If you are looking to jumpstart your systematic test design and want to avoid unnecessary tests and overhead, this session is for you!

1. W14
Session
10/26/2016 3:00:00 PM
Combinatorial Black-Box Testing with
Classification Trees
Presented by:
Peter Kruse
Berner & Mattner Systemtechnik GmbH
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-­‐268-­‐8770 ·∙ 904-­‐278-­‐0524 - info@techwell.com - http://www.starcanada.techwell.com/

2. Peter Kruse
Berner & Mattner Systemtechnik GmbH
Software engineer and test consultant Peter Kruse works in the domain of
testing, including evolutionary testing and the classification tree method. He is an
experienced software developer and tester in the German automotive industry.
Peter's project experience includes hardware-in-the-loop testing, model driven-
development, ISO 26262, and evolutionary structural and functional testing.
Apart from combinatorial testing, Peter's focuses on improving the costs and
benefits in any test process and related test activity, striving to balance emerging
trends and ideas with approved methods.

3. COMBINATORIAL BLACK-BOX TESTING WITH
CLASSIFICATION TREES
PETER M. KRUSE

4. Test Process and Test Activities
Testmanagement, Testdokumentation
Test
Planning
Test
Design
Test
Evaluation
Test
Execution
Test
Implementation
Test Management / Test Documentation
e.g. Module Test, Integration Test, System Test
Specification Tested (Sub-)
System
Motivation: How to obtain and structure test goals and efforts –
in a reliable, standard-conform, reproducible way?

5. Test Process and Test Activities
Testmanagement, Testdokumentation
Test
Planning
Test
Design
Test
Evaluation
Test
Execution
Test
Implementation
Test Management / Test Documentation
Specification Tested (Sub-)
System
e.g. Module Test, Integration Test, System Test
Solution: Put emphasis here.
Motivation: How to obtain and structure test goals and efforts –
in a reliable, standard-conform, reproducible way?

6. Classification Tree Method
• Test Design Technique
• Systematic Process
• Graphical Representation of Test Problem
• Independent of Test Domain, Test Level and Test Object
• Input: (Functional) Specification of System under Test
• Output: Set of Tests to be Implemented
• Tool Support: TESTONA
• Recommended for Certified Tester
• Part of ISO 29119-4

7. Classification Tree Method Example
• Test object: Database Management System
Select a test object: decompose

8. Classification Tree Method Example
Input domain
Determine input data space
Select a test object: decompose

9. Classification Tree Method Example
Privileges Operation Access Method
Input domain
Identify relevant aspects (e.g.
from specification)
Determine input data space
Select a test object: decompose

10. Input domain
Classification Tree Method Example
Privileges Operation Access Method
Classify the input data
space into classes

11. Classification Tree Method Example
1
2
3
Input domain
Privileges Operation Access Method
Combine classes
into test cases
Classify the input data
space into classes

12. Classification Tree Method – Steps
This is how TESTONA structures test goals:
reliable, standard-conform, and reproducible.
Combine classes into
test cases
Identify relevant
aspects (e.g. from
specification)
Determine input data
space
Select a test object:
decompose
Classify the input data
space into classes
Privileges
Operation
Access Method

13. Test Generation with TESTONA
• Equivalence Class Testing
• Boundary Value Analysis
• Combinatorial Testing
• Constraints support (dependency rules)
• Minimal Combination (test each input value at least once)
• All-Pairs test (Pairwise)
• Mixed-Strength Generation
• Full Combination (Cartesian product of all input values)
• Prioritized and Statistical Testing
This is how TESTONA obtains test efforts:
reliable, standard-conform, and reproducible.
Combine classes
into test cases
Automate test
generation
TESTONA

14. TESTONA
• Graphical editor for test design (e.g. classification tree method, boundary value
analysis, equivalence classes, etc.)
• Automatic verification of test cases and dependency rules
• Automatic test case generation
• Tag approach for annotation of any
information and meta-data
• Generic and customized
tool coupling
Used by

15. EXPECTED TEST OUTCOMES

16. Test Oracle
William E. Howden (1978):
non-trivial challenge of
deciding whether a test case
has passed or failed
Non-automated Oracles
• Trivial case: Manual Assignment
Implicit Oracles
• Indirect evaluation (e.g. no exception)
Derived Oracles
• A posteriori (e.g. Regression Test)
Specified Oracles
• Based on (formal) Specification
Categorization by Barr, Harman, McMinn,
Shahbaz, and Yoo (2015)

17. Non-automated Oracle Example
• Manual assignment for each test case

18. Implicit Oracle Example
• Monitor system
No exceptions at
runtime,
CPU load, …

19. Derived Oracle Example
• Run once, record results, inspect, assign to tests

20. Specified Oracle Example
(New AND Success ↔ more) AND (Delete AND Success ↔ less)
(empty AND NOT New) OR (full AND New) ↔ Fail

21. TEST IMPLEMENTATION
(TEST SCRIPT GENERATION)

22. Code Generation: Motivation
Classification
Tree Test Cases
Specification
(Functional Requirements,
Agile Board, etc.)

23. Code Generation: Motivation
Classification
Tree Test Cases
TESTONA
generic
code
export
• Template-based export of test cases to JUnit, XML, text, ...
Specification
(Functional Requirements,
Agile Board, etc.)

24. Technical Details / Demonstration: JUnit
setDB("%CLASS%");
int oldCount = getCount();
setUser("%CLASS%");
perform("%CLASS%");
assertTrue(getCount() %CLASS% oldCount);
assertEquals(isSuccess(), %CLASS%);
Include these per mark in test case
@Test
public void %TEST% throws Exception {
}
Include this per test case

25. Technical Details / Demonstration: JUnit
setDB("%CLASS%");
int oldCount = getCount();
setUser("%CLASS%");
perform("%CLASS%");
assertTrue(getCount() %CLASS% oldCount);
assertEquals(isSuccess(), %CLASS%);
Include these per mark in test case
@Test
public void %TEST% throws Exception {
}
import org.junit.Assert.*;
public class Example {
}
Include this per test case
Include this once at start

26. Technical Details / Demonstration: JUnit

27. Technical Details / Demonstration: XML
<database>%CLASS%</database>
<user>%CLASS%</user>
<action>%CLASS%</action>
<expect val="count">%CLASS%</expect>
<expect val="ok">%CLASS%</expect>
Include these per mark in test case
<testcase>
</testcase>
<testspec>
</testspec>
Include this per test case
Include this once at start

28. Technical Details / Demonstration: XML

29. CONCLUSION

30. •Expected result prediction
•Test Script Generation
Conclusion
•Classification Tree Method to
structure test goals
•TESTONA to obtain test efforts
Peter M. Kruse @pmk99
peter.kruse@berner-mattner.com
www.testona.net