We run a k€ 21,752 EU project to push the test automation research front. This talk motivates why this is (tax) money well spent and presents some research highlights: 1) test result visualization, 2) mutation testing, and 3) AI-assisted bug assignment.

1. Research Institutes of Sweden
TEST AUTOMATION
RESEARCH… IS
THAT REALLY
NEEDED IN 2018?
@mrksbrg
mrksbrg.com
Markus Borg
Swedish Institute of Computer Science

2. Intrinsic human
fascination with
automation
Research helps to automate:
1) better
2) more
3) sustainably
(CC Flickr: dalbera)

3. Development engineer, ABB, Malmö, Sweden 2007-2010
▪ Editor and compiler development
▪ Safety-critical systems
PhD student, Lund University, Sweden 2010-2015
▪ Machine learning for software engineering
▪ Bug reports and traceability
Senior researcher, RISE AB, Lund, Sweden 2015-
▪ Software engineering for machine learning
▪ Software testing and V&V
3
Who is Markus Borg?

4. 4
▪ TESTOMAT project
▪ Test visualization
▪ Mutation testing
▪ AI-assisted bug report assignment
Agenda

5. TESTOMAT project
5

6. Problem Statement
▪ Contemporary dilemma. Modern software
teams must optimize for both:
▪ bug free software
▪ ease of change
▪ Ever-faster release cycles => more automation
6
Project Goal
▪ Help software teams to increase the
development speed without sacrificing quality
▪ Advance the state-of-the-art in test
automation

7. 7
Plan
Design
Develop
Maintain
Execute
Analyze
Take
action
Three years
34 partners
€ 21,752,000

8. TRL 0
TRL 1
TRL 2
TRL 3
TRL 7
TRL 8
TRL 9
WP4 Test
Priorities
WP6 Test Automation
Maturity
WP5 Automated Testing
for Quality Standards
WP3 Test
Effectiveness
From technology validated in research labs
To technology demonstrated in relevant
environment
TRL 4
TRL 5
TRL 6
8

9. Test visualization
9

10. 10

11. Large amounts of test results
Simple result matrices hide information
11
The backside…
Wikimedia Commons, Brukar:Bep
Creative Commons CC-BY 2.5

12. 12
Solution Approach
Visual analytics
(Illinois Applied Research Institute)

13. ▪ Why game engine?
▪ Interaction out-of-the-box
▪ Why Unity?
▪ Fairly simple
▪ Scales well
▪ Very popular
▪ Unity???
▪ Cross-platform game engine and IDE
▪ Drag-and-drop 2D and 3D scenes
▪ Scripting in C#
13

14. 14

16. 16
Plan
Design
Develop
Maintain
Execute
Analyze
Take
action

17. Mutation testing
17

18. Do you
trust your
test cases?
18
Ali Parsai
(Staff Sgt. Ryan Callaghan, US Air Force)

19. High Quality Test
Suite
Fewer Bugs
How do you know that your test suite is good?

20. Test Suite
Software
0

21. Fault Injection
Mutant
1

22. SurvivedKilled
2

23. 3
Mutation Operator

24. a-ba+b
a<<ba>>b
a!=ba==b
a<=ba<b
!aa
B::bA::b
a(b,c)a(b)
a|ba&b
Very many test executions to kill mutants!

25. 25
Solution Approach
Toward mutation testing in the cloud
Sten Vercammen
▪ Goal
▪ Make mutation testing fast enough to fit during nightly build
▪ Approach
▪ Distribute the work
▪ Investigate bottlenecks and recommend optimizations

26. 26
Lessons learned from proof-of-concept tool
▪ Most steps are independent
▪ Speed-ups of 12x-13x with 16
workers
▪ Good chance to finish during
nightly builds

27. 27
Plan
Design
Develop
Maintain
Execute
Analyze
Take
action

28. AI-assisted bug
assignment
28

29. 29
Dr. Leif Jonsson

30. 30

31. Bug
tracker

32. 32
Model of bug report assignment
Bug
tossing!

33. ▪ Goal
▪ Useful tool deployable with minimum configuration effort
▪ Approach
▪ Bug reports = textual data + basic metadata
▪ Train machine learning classifiers on historical bug reports
▪ Combine them using state-of-the-art ensemble learning
33
Solution Approach
AI-assisted bug report assignment

34. 34
Experimental setup
Company A
Company B
Machine
learning
4 x
Pre-processing &
feature selection
50,000+ bug reports

35. ▪ In line with human activity
– But instantaneous!
35
Results

36. ▪ Productification of solution in internal tool
▪ Simplified solution without ensemble
▪ Deployed in bug tracker for large project
▪ Presents instantaneous recommendation of responsible team
▪ Accuracy 8% lower than manual work
36
Prototype deployed at Ericsson

37. 37
Bug tracker
Machine
Learning

38. 38
Bug tracker
Machine
Learning

39. 39
Plan
Design
Develop
Maintain
Execute
Analyze
Take
action

40. Research on
test automation
=
Good use of
tax money
40

41. Intrinsic human
fascination with
automation
Research helps to automate:
1) better
2) more
3) sustainably
(CC Flickr: dalbera)

42. Research Institutes of Sweden
Swedish Institute of Computer Science
markus.borg@ri.se
mrksbrg.com
@mrksbrg
© Musée des arts et métiers-CNAM
Research =>
1) better
2) more
3) sustainable
test automation