Vincent Blondeau conducted an experiment comparing static and dynamic test selection approaches after code changes. The static approach uses call graph analysis to select tests related to changed code, while the dynamic approach executes tests to map them to covered code. The experiment found that the dynamic approach was more precise and had higher recall, identifying more relevant tests, though it required test execution time. Considering code commits, which group related changes, tended to decrease results for both approaches. The ratios of selected tests to total tests were low, indicating the approaches could still help reduce test suite execution time. Future work includes better understanding how developers use tests and providing a test selection tool.

1. 25 – August – 16
Test Selection
with Moose In Industry:
Impact of Granularity
Vincent Blondeau

2. Vincent Blondeau | 25 - August - 16 | 2
Context
▶ Industrial PhD in a major international IT company
– +7 000 employees
– 17 countries
– Problems from the field

3. Vincent Blondeau | 25 - August - 16 | 3
Test Case Selection After a Change
?
Test All
Takes a long time
(3 hours)
Test Nothing
Not a solution

4. Vincent Blondeau | 25 - August - 16 | 4
Test Case Selection After a Change
▶ Question
– Does test selection modify developers’ habits and
enhance software quality?
▶ Goal
– Save time and improve quality
– Select tests to relaunch after a change in the source
code
– Any kind of test: End to end, performance…
▶ Comparison of approaches on real changes
– Call graph analysis by static approach
– Comparison with dynamic approach

5. Vincent Blondeau | 25 - August - 16 | 5
Two approaches
▶ Static Approach
– Create a model of the system with Moose
– Navigate the call graph from a changed source code
to find the tests
Properties
– Allows to find multiple tracks to the changed source
– No test execution

6. Vincent Blondeau | 25 - August - 16 | 6
Two approaches
▶ Dynamic Approach
– Execute the tests
• Map the tests to the covered code
– Relaunch the tests related to changed source code
Properties
– Dependent to the test data
– The tests have to be executed

7. Vincent Blondeau | 25 - August - 16 | 7
Issue Classification
▶ Problems in test selection approaches arise when
there is a break in the dependency graph representing
the system.
– Third-party breaks
– Multi-program breaks
– Dynamic breaks
– Polymorphism breaks

8. Vincent Blondeau | 25 - August - 16 | 8
Experiment
▶ Hypothesis: Dynamic approach is the oracle
– With some flaws:
• Does not work on failing or in error tests
• Requires time to be performed
▶ Approach
– Compare influence of real source code changes
– Simulate code change on several existing projects

9. Vincent Blondeau | 25 - August - 16 | 9
Experiment
▶ Consider real commits by mining repositories
– Weight each covered method by the number of
commits
– Group covered methods in commits
• Considered real method commit groups

10. Vincent Blondeau | 25 - August - 16 | 10
Projects: Metrics
Metric P1 P2 P3
KLOC Core 447 716 302
# Green Tests 5 323 168 3 035
# Total Methods 9 808 56 661 45 671
# Methods Covered
4720
(48%)
3 261
(6%)
8 143
(18%)
#Commits 2 217 467 2 115
Avg Methods/Commit 24 129 37
Avg Files/Commit 7 18 17

11. Vincent Blondeau | 25 - August - 16 | 11
Metrics
▶ Number of selected tests
– Ratio of the total test suite to relaunch
▶ Precision
– How many selected tests are relevant?
▶ Recall
– How many relevant tests are selected?

12. Vincent Blondeau | 25 - August - 16 | 12
#Selected Tests Precision Recall
1 Meth. Weig. 1 Meth. Weig. 1 Meth. Weig.
P1 3% 3%
43%
42% 91%
92%
P2 0.8%
1% 61% 64%
59% 62%
P3 2% 2%
34% 41%
39%33%
Weighting of methods with the
number of commits

13. Vincent Blondeau | 25 - August - 16 | 13
#Selected Tests Precision Recall
1 Meth. Commit 1 Meth. Commit 1 Meth. Commit
P1
3%
4%
43%
55% 91%
81%
P2
0.8%
3% 61% 64%
45% 45%
P3
2%
6% 49%
41%
56%
34%
Methods grouped in commits

14. Vincent Blondeau | 25 - August - 16 | 14
Conclusion
▶ Considering commits instead of individual methods
tends to worsen the results
▶ Impact on projects is different
▶ Low ratio of selected tests, so still acceptable
Future steps
▶ Better understand how tests are used by developers
▶ Provide a tool for developers to select tests

15. Worldline is a registered trademark of Atos Worldline SAS. June 2013
© 2013 Atos. Confidential information owned by Atos Worldline, to be
used by the recipient only. This document, or any part of it, may not
be reproduced, copied, circulated and/or distributed nor quoted
without prior written approval from Atos Worldline.
25-August-16
Thanks!