14. Contest Methodology
Search budgets
1 min. 3 min.
Classes under test
70 classes
Repetitions
10 repetitions
Execution environment Statistical analysis
Friedman’s test
Post-hoc Conover
18. Lessons Learnt
• Results are consistent with other independent evaluations
• Shows that the bugs discovered in the infrastructure could be fixed
• The two-steps selection procedure for the classes under test is useful
• Discover configuration issues early
• Still largely a manual process
• Docker simplifies the evaluation procedure
19. What’s Next?
• Contest Infrastructure
• https://github.com/JUnitContest/junitcontest
• Improve usability
• Facilitate setup of an evaluation
• Facilitate evaluation in other contexts
• Update the user documentation
• Storage and versioning of the results (and participating tools?)
• For the next edition
• More tools
• More CUTs