- The document summarizes a presentation given by Andy Zaidman at the International Conference on Automation of Software Test (AST 2023) in Melbourne, Australia on May 16th, 2023. - It discusses findings from studies on how developers engineer test cases and their testing behaviors in IDEs, including strategies like being guided by documentation or code. - It also presents recommendations to improve developer testing through better tool support, clear adequacy criteria in education, and a focus on improving the user and developer experience of testing tools and processes.

1. Andy Zaidman
Delft University of Technology
The Netherlands
International Conference on
Automation of Software Test
(AST 2023)
May 16th, 2023
Melbourne, Australia

2. A story on testing… and a bit of music

3. Human-in-the-Loop
important in testing
⎼Sigrid Eldh, during AST 2023 on Monday May 15th

4. Automatic for the people?
Automatic by the people?
Automatic by the machine?
Automatic for the machine?

5. Automatic for the people
Automatic by the people?
Automatic by the machine?
Automatic for the machine?

7. 8. Ignoreland

8. What do we know about
developers’ testing activities in
the IDE?

10. 2443 SW Engineers – 118 countries
Java/C# – Eclipse, IntelliJ, Android Studio, Visual Studio
Minimal 5 month observation period (max. 2.5 years)
161 person years of development work

11. Not all Java projects do unit testing (in the IDE)
3 508
1 498
(43%)
Write/look at test code
or execute tests

12. Estimated when installing tool
Time spent on test code engineering
test code
engineering
production code
engineering
47% - 53%

13. Estimated when installing tool After measuring min. 5 months
Time spent on test code engineering
25% - 75%
test code
engineering
production code
engineering
47% - 53% test code
engineering
production code
engineering

14. Overestimating the testing effort stems from:
(1) the tedious nature of testing
(2) developers disliking it

15. 10. Man on the Moon
Lyrics: Now Andy did you hear about this one?
…
If you believed they put a man on the moon
…

18. High Coverage
&
Enormous Potential

19. Difficult to Read Test Cases
&
Are They Asserting Correctness?

20. Tools like TestDescriber

21. Tools like TestDescriber
Key issue:
Understanding
the scenario
under test

22. Tools like TestDescriber
Key issue:
Understanding
the scenario
under test

23. Which test makes
more sense? Which
scenario is easier to
grasp?

26. Do test need to be
UNDERSTANDABLE?

27. A good test can catch a bug
and returns feedback that can
help you identify the issue.

28. A good test can catch a bug
and returns feedback that can
help you identify the issue.
A good developer test forms
executable documentation that
tells you how to use the methods.

29. A good test can catch a bug
and returns feedback that can
help you identify the issue.
A good developer test forms
executable documentation that
tells you how to use the methods.
Understandability of a test
(scenario) is important

30. What is the purpose of a generated test?
Throw away; find faults in current version of the software

31. What is the purpose of a generated test?
Throw away; find faults in current version of the software
Inspiration to write a manually written test case

32. What is the purpose of a generated test?
Throw away; find faults in current version of the software
Inspiration to write a manually written test case
To become part of a maintained test suite

33. What is the purpose of a generated test?
Throw away; find faults in current version of the software
Inspiration to write a manually written test case
To become part of a maintained test suite
A more specific generated test, e.g., a crash replicating test

34. What is the purpose of a generated test?
Throw away; find faults in current version of the software
Inspiration to write a manually written test case
To become part of a maintained test suite
A more specific generated test, e.g., a crash replicating test

35. 4. Sweetness follows

36. Some solution spaces…
• Test amplification
• Test generation with information carving
• Documenting generated tests
• Better support for manual writing of test cases
• …

37. Test amplification

38. Test amplification
• Starts from existing test cases
Applies systematic “mutations” to test code to see whether more
coverage is obtained and/or different scenarios are tested

39. Test amplification
• Starts from existing test cases
Applies systematic “mutations” to test code to see whether more
coverage is obtained and/or different scenarios are tested
+ Easier to understand test
scenarios compared to
freshly generated tests
- Difficult to cover entire
search space, depends on
existing tests
- Inspection cost

40. Test Impact Graph

42. Test generation with information
carving

43. Carving to explore the search space

44. Carving to explore the search space
vs.
Carving to improve the
understandability of tests

45. MicroTestCarver approach

46. EvoSuite
MicroTestCarver

47. Documenting generated tests

50. Better support for manual writing
of test cases

52. 4. Everybody Hurts

55. When are developers
discouraged to test?

56. When do developers
aspire to test or become
better testers?

57. Emerging theory

59. 1. We observe 13 developers thinking-aloud while testing methods in
open-source software
2. We challenge and augment our findings by surveying 72 software
developers

60. Main test case engineering strategies of
software engineers
1. intensively guided by documentation,
2. intensively guided by source code,
3. or ad-hoc.

61. Recommendations
1. Tool support
• Creation of test skeletons
• Quick code coverage indications
• Copy/paste support for test code
2. Developers
• Have a clear adequacy criterion to avoid uncertainty
3. Education
• Teach how to use code coverage tools to steer testing, not just as a
metric

62. Recommendations
1. Tool support
• Creation of test skeletons
• Quick code coverage indications
• Copy/paste support for test code
2. Developers
• Have a clear adequacy criterion to avoid uncertainty
3. Education
• Teach how to use code coverage tools to steer testing, not just as a
metric
Come see our work on SW testing education
at SEENG in the next session!

63. 12. Follow the river

64. Where does it leave us?

65. We are going to have to make our hands dirty…

66. Whether we like it or not…

67. Test cost Inspection cost

68. Test cost Inspection cost
“You can’t test software with your hands in your pants”

69. We cannot
neglect testing

70. Great tools

71. We should be obsessive in improving the
user experience

72. We should be obsessive in improving the
user experience UX

73. We should be obsessive in improving the
developer experience DX

74. We should be obsessive in improving the
tester experience TX

75. Automatic for the people

76. Automatic for the people
by the machine

77. Automatic for the people
by the machine and
the people

78. Automatic for the people
by the machine and
the people

79. All great
work by…

and
many
more
Team

80. Thank you!
International Conference on
Automation of Software Test
(AST 2023)
May 16th, 2023
Melbourne, Australia
azaidman

81. Some pointers
• Maurício Aniche, Christoph Treude, Andy Zaidman. How Developers Engineer Test Cases: An
Observational Study; IEEE Trans. on Software Engineering, 2022.
• Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, Andy
Zaidman. Developer Testing in The IDE: Patterns, Beliefs, And Behavior; IEEE Trans. on Software
Engineering, 2019.
• Sebastiano Panichella, Annibale Panichella, Moritz Beller, Andy Zaidman, Harald Gall. The Impact
of Test Case Summaries on Bug Fixing Performance: An Empirical Investigation; ICSE 2016.
• Carolin Brandt, Andy Zaidman. Developer-Centric Test Amplification: The Interplay Between
Automatic Generation and Human Exploration; Empirical Software Engineering, 2022.
• Mark Swillus, Andy Zaidman. Sentiment Overflow in the Testing Stack: Analysing Software Testing
Posts on Stack Overflow, Arxiv 2022.