Does investing in high quality developer test code really matter? And does it help a team to become more productive?

1. First Steps in Testing Analytics:
Does Test Code Quality Matter?
Andy Zaidman
Delft University of Technology
Nederlandse Testdag
17 november 2014, Amersfoort

2. Joint work with Moritz Beller & Georgios Gousios
(TU Delft)
Dimitrios Athanasiou, Ariadi
Nugroho & Joost Visser
(Software Improvement Group)

3. Who am I?
Software engineering
teacher by day
Testing
evangelist
Software engineering
researcher by night

4. What would you like to know?
What analytics?
• ???

6. WHAT EVER HAP P ENS…

7. @azaidman

8. TODAY
2 test analytics stories

9. STORY 1
Test Code Quality
Matters

10. Andy teaching
First year programming course @ TU Delft
 In week 3 everyone is unit testing
“I don’t have time for testing”
“I don’t like testing, my code is good
anyway”
“How much should I test?”
“How good should my test code be?”

11. @Test
public void testConstructor() {
assertTrue(true)
}
Solution 1

13. Solution 2
@Test
public void testConstructor() {
House h = new House(“Street”, “9”);
assertEquals(“Street 9”, h.toString());
}

14. Solution 3
@Test
public void testConstructor() {
House h = new House(“Street”, “9”);
assertEquals(“Street”, h.getStreet());
assertEquals(9, h.getNumber());
}

15. Solution 4
@Test
public void testConstructorStreet() {
House h = new House(“Street”, “9”);
assertEquals(“Street”, h.getStreet());
}
@Test
public void testConstructorNumber() {
House h = new House(“Street”, “9”);
assertEquals(9, h.getNumber());
}

16. My first reaction?

17. SHOOT
FIRST

18. BRINGS US
NOWHERE

19. Analytics?

20. Testing Analytics!

25. Thou shalt not
have less than
70% coverage

27. What makes a
good
unit test?
What makes a
good
integration test?

28. What makes a
good
developer test?

29. What makes a
good
developer test?
Complete
Effective
Maintainable

30. Complete
• How much of the code is covered by the tests?
• How many of the decision points in the code are
tested?
Code coverage
#assertions/cyclomatic complexity

31. Effective
• How able is the test code to detect defects?
• How able is the test code to locate the cause?
Assertion density #assertions / LOCtest
Directness unit versus integration testing

32. Maintainable
• How maintainable is the system’s test code?
Software Improvement Group quality model

33. Test Code Quality Model*
Complete
+ Effective
+ Maintainable
* Calibrated using 86 systems

34. SO
WHAT?

36. Better
tests
Easier
development
?

37. Observation 1*
High quality tests do not lead to quicker/easier
bug fixing
Why?
The low-hanging fruit is already solved if you
have good tests… the difficult bugs remain
* While studying 75 years of combined development of 18 software systems

38. High quality tests do lead to higher productivity
• More issues fixed per month per developer
• More issues fixed per month per KLOC
Why?
Implementing new features does seem to
benefit from having a safety net for existing
functionality.
Observation 2

39. High quality tests do make your team
more productive!

40. Action(able)

41. #assertions
McCabe cyclomatic complexity
≥ 1
The single most
important thing for test
code quality

42. …

43. STORY 2
Testing takes 50% of
your time

44. Says this guy
• Study from 1975
• Questionable research
approach

45. Differences?
• Waterfall
• Developer testing
• Programming language
• Tools (e.g. IDE)

46. New research

48. + +

49. Recording
1. Time spent on developing production code
2. Time spent on developing test code
3. How frequently are Junit tests executed?
4. How frequently do these test runs fail?

50. 124 developers participating

52. You might win

53. http://testroots.org/testroots_watchdog.html

54. …

55. My ultimate dream?
Continuous
background
testing

57. Take home
• Test code quality matters for the productivity
of your team
• #assertions/cyclomatic complexity ≥ 1
• Help us out with our research!
a.e.zaidman@tudelft.nl
http://azaidman.wordpress.com
httpp://www.st.ewi.tudelft.nl/~zaidman
@azaidman