We all want "better" test suites. But what makes for a good test suite? Certainly, test suites ought to aim for good coverage, at least at the statement coverage level. To be useful, test suites should run quickly enough to provide timely feedback. This talk will investigate a number of other dimensions on which to evaluate test suites. The talk claims that better test suites are more maintainable, more usable (for instance, because they run faster, or use fewer resources), and have fewer unjustified failures. In this talk, I'll present and synthesize facts about 10 open-source test suites (from 8,000 to 246,000 lines of code) and evaluate how they are doing.

1. Beyond Coverage:
What Lurks in Test Suites?
Patrick Lam, @uWaterlooSE
(and Felix Fang)
University of Waterloo

2. Test Suites: Myths vs Realities.

3. Subjects: Open-Source Test Suites

4. Basic Test Suite Properties
Benchmark sizes:
30 kLOC (google-visualization) to
495 kLOC (weka)
% of system represented by tests:
5.3% (weka) to 50.4% (joda-time)

5. Static Test Suite Properties

6. Test suite versus benchmark size
m = 0.3002
m = 0.03514

7. # test cases versus # test methods

8. apache-commons-collection tests
Consider map.TestFlat3Map:
contains 14 test methods
yet, 156 test cases
superclass tests: 42 tests
+ 4 Apache Commons Collections “bulk tests”

9. Run-time Test Suite Properties

10. Test suites run quickly
joda-time 4.9s
jdom 5.0s
google-vis 5.1s
jgrapht 16.9s
weka 28.9s
apache-cc 34.0s
poi 36.5s
jmeter 53.0s
jfreechart 241.0s

11. Failing tests
76/384
0
n/a 0
1
0
3/1109
0
0
0

12. Continuous Integration: Daily Builds

13. Continuous Integration: Daily Tests
(via SonarQube,
Travis CI, Surefire)

14. Myth #1:
Coverage is a key property
of test suites.

15. Coverage is central in textbooks
Ammann and Offutt, Introduction to Software Testing

16. Coverage metrics from EclEmma

17. Coverage metrics

18. Reality #1
Coverage sometimes important,
but tools only give limited data.

19. Guideline #1
Consider metrics beyond
reported coverage results:
- weka uses peer review for QA
- not measured by tools:
input space coverage

20. Myth #2
Tests are simple.
- test complexity
- test dependencies

21. Static Code Complexity

22. Test methods with at least 5 asserts
e.g. from Joda-Time:
public void testEquality() {
assertSame(getInstance(TOKYO), getInstance(TOKYO));
assertSame(getInstance(LONDON), getInstance(LONDON));
assertSame(getInstance(PARIS), getInstance(PARIS));
assertSame(getInstanceUTC(), getInstanceUTC());
assertSame(getInstance(), getInstance(LONDON));
}

23. % Test methods with ≥ 5 asserts

24. Test Methods with Branches
if (isAllowNullKey() == false) {
try {
assertEquals(null, o.nextKey(null));
} catch (NullPointerException ex) {}
} else {
assertEquals(null, o.nextKey(null));
}
// from apache-cc

25. Test Methods with Loops
counter = 0;
while (this.complexPerm.hasNext()) {
this.complexPerm.getNext();
counter++;
}
assertEquals(maxPermNum, counter);
// from jgrapht

26. % Test Methods with Control-Flow

27. Tests Which Use the Filesystem

28. Filesystem Usage Details
new File(tempDir, "tzdata");
verifies vs canonical forms
of serialized collections on disk

29. More Filesystem Usage Details
resources, serialization
creates charts, tests their existence
some comparisons vs test data

30. Tests Which Use the Network
*

31. Network Usage Details
connects to
http://sc.openoffice.org
tests HTTP mirror server
at localhost

32. flip side: Mocks and Stubs
True mocks only in Google Visualization.

33. flip side: Mocks and Stubs
True mocks only in Google Visualization.
Found stubs/fakes in
4 other suites.

34. Reality #2
Test cases are mostly simple.
few asserts, little branching
some filesystem/net usage

35. Consequence #2
Many tests don’t need
high expertise to write,
but some do!

36. Myth #3
Test cases are written by hand.

37. Types of reuse (standard Java)
1. test class setUp()/tearDown()
2. inheritance: e.g. in apache-cc,
TestFastHashMap extends AbstractTestMap
3. composition: e.g. in jfreechart,
helper class RendererChangeDetector

38. JUnit setup/tearDown usage

39. Inheritance is heavily used
(> 50% test classes inherit functionality)

40. Test Classes with Custom Superclasses

41. Helper Classes Example
from poi:
/** Test utility class to get Records
* out of HSSF objects. */
public final class RecordInspector {
public static Record[] getRecords(...) {}
}

42. Helper Class Count
weka 1
google-vis 3
jdom 6
joda-time 7
jfreechart 7
jmeter 12
jgrapht 15
apache-cc 22
hsqldb 31
poi 54

43. Test Clone Example
public void testNominalFiltering() {
m_Filter = getFilter(Attribute.NOMINAL);
Instances r = useFilter();
for (int i = 0; i < r.numAttributes(); i++)
assertTrue(r.attribute(i).type() != Attribute.NOMINAL);}
public void testStringFiltering() {
m_Filter = getFilter(Attribute.STRING);
Instances r = useFilter();
for (int i = 0; i < r.numAttributes(); i++)
assertTrue(r.attribute(i).type() != Attribute.STRING);}

44. Assertion Fingerprints
detect clones
by identifying
similar tests

45. Incidence of cloning

46. How to Refactor?
● setUp/tearDown/subclassing
● JUnit 4:
Parametrized Unit Tests
● Test Theories

47. apache-cc: Bulk tests
public BulkTest bulkTestKeySet() {
return new TestSet(makeFullMap().keySet());
}
● runs all tests in the TestSet class
with the object returned from makeFullMap().
keySet()

48. jdom: Generated Test Case Stubs
class ClassGenerator makes e.g.:
class TestDocument {
void test_TCC__List();
void test_TCM__int_hashCode();
}
Developer still needs to populate tests.

49. Automated Testing Technology
In our test suites,
the principal automation technology
was cut-and-paste.

50. Reality #3
Automated test generation
is uncommon in our test suites.

51. Guideline
Maximize reuse:
setUp/tearDown,
inheritance,
parametrized tests,
whatever works for you!

52. Suggestion
Use automated test generation tools!
Some examples:
● Korat (structurally complex tests)
● Randoop (random testing)
● CERT Basic Fuzzing Framework
http://mit.bme.hu/~micskeiz/pages/code_based_test_generation.html

53. Summary
Myths:
1. Coverage is a key property
of test suites. ≈
2. Tests are simple. ✓
3. Tests are written by hand. ✓

55. Data
https://docs.google.
com/spreadsheets/d/1xAsdk35tJAOM4WGbGloliS4ovDJ8_
MDn6_Gzk0DXEZQ