This document summarizes findings from software analytics research on developer testing practices. It finds that developers overestimate the amount of time spent on testing, with most spending 25-75% of their time on test code compared to an estimated 50%. Tests are rarely executed in IDEs, with only 20% failing, compared to 60% failing in IDEs. Most projects have test code but over half of developers did not interact with tests. Testing is crucial for continuous integration, with 98% of projects failing builds when tests fail. The research helps developers understand their own behaviors and identifies challenges for improving tools and education.

1. Fact or Fiction?
What Software Analytics
Can Do For Us
Andy Zaidman
Delft University of Technology
SPLASH-I 2016, Amsterdam
azaidman

2. I like to tell stories…
Andy

3. EVERYBODY
LIES

4. EVERYBODY
LIES

5. EVERYBODY
Andy
Zaidman

6. ANDY
LIES

7. EVERYBODY
Andy
Zaidman
Moritz
Beller
Georgios
Gousios
Annibale
Panichella
Niels
Spruit

8. Let’s consider software testing

9. ANDY’S
LIE
Grant proposal
“It is well-known
that testing
consumes 50% of a
developer’s time”

11. “it depends”
“50% of your time”

12. What is the (likely) origin
of the 50%?

13. The Mythical Man-Month?
Brooks “observed”
that testing and
debugging took 50%
of the time
How?
“Calendar research”
When
± 40 years ago…

14. ANDY’S
LIE FACTS
Grant proposal
“It is well-known that
testing consumes
50% of a developer’s
time [1]”

17. Chinese whispers / telephone game

18. “Truth begins in lies”
-House

19. Lies begin in truth
“Truth begins in lies”
-House

20. Folklore/evidence
test approximately 50%
Lack of knowledge
How much time is actually
spent on testing?

22. Research Tool Software Analytics
“Big Brother” in your IDE
 measures testing activities
• 416 Softw. Engineers, 68 countries
• 5 month observation period
• Java / Eclipse
• 13.7 man years of development
observed
Immediate feedback
for developers

23. WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on
engineering test code versus production code?
50% 50%test code production code
30% 70%test code production code

24. WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on
engineering test code versus production code?
48% - 52%test code production code

25. After measuring ≥ 3 months
25% - 75%test code
engineering
Most participants
overestimate their
test engineering
activities
production code
engineering

27. Not all Java projects do unit testing
Likely, an underestimation…

28. No test engineering
For projects with test code:
53% of developer did not execute, read, or
modify a single test within five months.

29. Do sets of unit test executions take…
< 5 seconds ≥ 5 seconds

30. Swift tests
Execution of set of unit tests takes 0.54
seconds (median).

31. Ad hoc test selection
Developers frequently select a specific set of
tests to run in the IDE. In most cases,
developers execute one single test.

32. Test execution in the IDE is rare
In 85% of development sessions no tests were
run…

33. Tests fail
65% of test executions in the IDE fail.

34. Test Driven Development
TDD is not widely practiced. Programmers
who claim to do it, neither follow it strictly nor
for all their modifications.

35. • Be critical: are long-standing
beliefs/insights/claims
• Still valid?
• Valid in your context?
• Triangulate
• Developer opinions deviate greatly from
what we objectively measured
• Mixed-method research
• Longitudinal studies

36. • Estimating is difficult
• Estimating something that you don’t
like is more difficult
• Beliefs are to be questioned all the time

38. Differential diagnosis
- Tests are written, but not run
- Tests are written, but run on Command Line
- Tests are written, but run in Continuous Integration

39. Differential diagnosis
- Tests are written, but not run
- (Tests are written, but run on Command Line)
- Tests are written, but run in Continuous Integration

40. • 1,359 Java and Ruby projects
• 2,640,825 builds

41. Testing in Continuous Integration
In our population of 1,359 Java and Ruby projects

42. Builds with at least one failing test
From 1,108 projects which executed tests

43. Testing #1 reason for build to break
From 1,108 projects which executed tests
Test failure
Other failure

44. Tests are decisive for overall build
From 1,108 projects which executed tests
98.3% projects
Test fails  Build breaks

45. Integrating in multiple environments
From 1,108 projects which executed tests
60%projects
Different build result for at least one build in
multi-environment setup

46. Differential diagnosis
- Tests are written, but not run
- (Tests are written, but run on Command Line)
- Tests are written, but run in Continuous Integration
- 60% failing tests in the IDE
- <20% failing tests in CI
- Command Line?

47. ANDY’S LIES
1. 50% of time goes to
testing
2. This talk was on
software analytics

48. Empirical
Software
Engineering
Software
Analytics

49. Software analytics
=
bait / intrinsic motivation

52. From an experienced developer…
“Estimated time working on tests: 20%,
actual time: 6%. Cool statistics! ”
Testing analytics helps developers to get a
better understanding about their own
development behavior!

53. Software analytics
Empowers software engineers, software
teams, managers
Make better tools

54. Debugging

55. WatchDog 2.0
We are looking into how developers debug
Product improvement: In contact with Eclipse + IntelliJ
New features coming to Eclipse based on our
analytics
Educational improvement: We need to teach our
students to debug more efficiently (with the IDE
debugger)

56. What to expect?

57. Software Analytics Challenges
for the SPLASH community
• What works in languages? What doesn’t?
• How should we design the next version of
languages?
• How should we design the next generation of
tools?
• How should we improve education?

58. Software
engineering
folklore
Perception w.r.t. dev. testing
Measured developer testing
48% - 52%
25% - 75%
Software analytics
help engineers to
better understand
their own behavior
azaidman
WatchDogTeam
Challenges
for SPLASH

59. Pointers to literature
1. Moritz Beller, Georgios Gousios, Andy Zaidman: How (Much) Do
Developers Test? ICSE 2015: 559-562
2. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy
Zaidman: When, how, and why developers (do not) test in their IDEs.
ESEC/SIGSOFT FSE 2015: 179-190
3. Moritz Beller, Georgios Gousios, Andy Zaidman: Oops, my tests broke
the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints
4: e1984
4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 1975
5. Laurent Bossavit: The Leprechauns of Software Engineering, 2012-
2014, https://leanpub.com/leprechauns