Talk from IWST at ESUG19, Cologne, Germany

1. Exposing Test
Analyses with DrTests
Dayne Guerra, Julien Deplanque, Stephane Ducasse
guerra.dayne@gmail.com , {first}.{last}@inria.fr

2. Roadmap
• Variety of test analyses available
• What architecture to handle these analyses?
• DrTests
• Evaluating DrTests on various analyses
• Conclusion
2

3. Roadmap
• Variety of test analyses available
• What architecture to handle these analyses?
• DrTests
• Evaluating DrTests on various analyses
• Conclusion
3

4. Runner Coverage Profile
Parametrisable
test
Mutation
testing
Run
examples in
comment
Rotten
Tests
Finder
…
Input Test cases
Outpu
t
Results of
tests run
Variety of analyses on tests
4

5. Runner Coverage Profile
Parametrisable
test
Mutation
testing
Run
examples in
comment
Rotten
Tests
Finder
…
Input Test cases Test cases …
Outpu
t
Results of
tests run
Percentage
of methods
covered
…
Variety of analyses on tests
5

6. Runner Coverage Profile
Parametrisable
test
Mutation
testing
Run
examples in
comment
Rotten
Tests
Finder
…
Input Test cases Test cases
Test
cases
…
Outpu
t
Results of
tests run
Percentage
of methods
covered
Time
taken for
each test
run
…
Variety of analyses on tests
6

7. Runner Coverage Profile
Parametrisable
test
Mutation
testing
Run
examples in
comment
Rotten
Tests
Finder
…
Input Test cases Test cases
Test
cases
Test cases
+
parameters
Outpu
t
Results of
tests run
Percentage
of methods
covered
Time
taken for
each test
run
Results of
tests run
depending
on
parameter
Variety of analyses on tests
7

8. Runner Coverage Profile
Parametrisable
test
Mutation
testing
Run
examples in
comment
Rotten
Tests
Finder
…
Input Test cases Test cases
Test
cases
Test cases
+
parameters
Test
cases
+
mutation
s
Outpu
t
Results of
tests run
Percentage
of methods
covered
Time
taken for
each test
run
Results of
tests run
depending
on
parameter
Mutants
to be
killed
Variety of analyses on tests
8

9. Runner Coverage Profile
Parametrisable
test
Mutation
testing
Run
examples in
comment
Rotten
Tests
Finder
…
Input Test cases Test cases
Test
cases
Test cases
+
parameters
Test
cases
+
mutation
s
Comments
Outpu
t
Results of
tests run
Percentage
of methods
covered
Time
taken for
each test
run
Results of
tests run
depending
on
parameter
Mutants
to be
killed
Comments
containing
failing
examples
Variety of analyses on tests
9

10. Runner Coverage Profile
Parametrisable
test
Mutation
testing
Run
examples in
comment
Rotten
Tests
Finder
…
Input Test cases Test cases
Test
cases
Test cases
+
parameters
Test
cases
+
mutation
s
Comments Test cases …
Outpu
t
Results of
tests run
Percentage
of methods
covered
Time
taken for
each test
run
Results of
tests run
depending
on
parameter
Mutants
to be
killed
Comments
containing
failing
examples
Rotten
tests
…
Variety of analyses on tests
10

11. Roadmap
• Variety of test analyses available
• What architecture to handle these analyses?
• DrTests
• Evaluating DrTests on various analyses
• Conclusion
11

12. How to design an
infrastructure dealing
with all these analyses?
Different
Inputs
Different
Outputs
12
Test Runner
Test Coverage
Test profiling
…

13. Roadmap
• Variety of test analyses available
• What architecture to handle these analyses?
• DrTests
• Evaluating DrTests on various analyses
• Conclusion
13

14. Test Runner
Test Coverage
Test profiling
Configuration Plugin Result
14
An architecture to handle
test analyses variety:
DrTests

15. Dr Tests Model
Plugin
Configuration
Plugin
Result
Plugin
Plugin run,
provides updates
through announcements
Created from UI
or by scripting
Can be queried from
UI or by scripting
15

16. Dr Tests UI
Packages
under analysis
Plugin selected
Logging label
16

17. Dr Tests UI
Plugin input
17

18. Dealing with different inputs
Packages with
comments Tests
Test package
&
Package under
analysis
18

19. Dr Tests UI
Start plugin
19

20. Dr Tests UI
Logging label
Results
tree
20

21. Different results
Comments Percentage &
Uncovered methods
Rotten Tests
21

22. Roadmap
• Variety of test analyses available
• What architecture to handle these analyses?
• DrTests
• Evaluating DrTests on various analyses
• Conclusion
22

23. We implemented 3 plugins
with various input
and output

24. Dr Tests model validation
Test Runner Rotten Test Test Coverage
INPUT
Tests Cases
(packages and classes)
Tests Cases
(packages and
classes)
Tests and Package
under analysis
ACTION Run the test cases
Find Rotten Green
Tests
Run tests and verify
that methods inside
the package were
executed
RESULT
Test result sorting by
groups
A list of rotten tests
Percentage of
covering and list of
uncovered methods
24

25. Test Runner Rotten Test Test Coverage
INPUT
Tests Cases
(packages and classes)
Tests Cases
(packages and
classes)
Tests and Package
under analysis
ACTION Run the test cases
Find Rotten Green
Tests
Run tests and verify
that methods inside
the package were
executed
RESULT
Test result sorting by
groups
A list of rotten tests
Percentage of
covering and list of
uncovered methods
Dr Tests model validation
25
These analyses are implemented
inside DrTests architecture and it
works!

26. Roadmap
• Variety of test analyses available
• What architecture to handle these analyses?
• DrTests
• Evaluating DrTests on various analyses
• Conclusion
26

27. Conclusion
‣ Extensible via plugins
‣ Provides good model to configure, run and gather
results from plugins
‣ Be integrated in Pharo 8.0
‣Future work:
‣ Extend the validation by implementing more
plugins
@dayneStorm_
GDayne
juliendelplanque/DrTests
27

28. TestRunner UI
Packages
containing tests
TestCases
Results
summary
Failed tests
Errors
Run tests Profile test
execution
Analyse code
coverage
Re-run failures
or errors only Export results
28

29. TestRunner UI: coverage and profile
Extra windows generated
for results
29

30. Dr Tests
Packages
under analysis
Plugin input Plugin selected
Results
tree Kind of
visualization
Start pluginLogging label
Plugin-defined
action(s)30

31. Dr Tests UI
Kind of
visualization
Skip if no time
31

32. Dr Tests UISkip if no time
32

33. Dr Tests UI
Plugin-defined
action(s)
Skip if no time
33