Overview of research highlights from my time as a PhD student with the Software Engineering Research Group at Lund University.

1. FROM BUGS TO
DECISION SUPPORT
- Selected Research Highlights
Markus Borg
SSE Meeting Nov 10, 2015

2. Per Runeson

3. PHD STUDENT POSITION
”Aligning RE and V&V”
Bjarnason, Runeson, Borg, et al. Challenges and Practices in Aligning Requirements with Verification
and Validation: A Case Study of Six Companies, Empirical Software Engineering, 19(6), 2014.
EMSE
14

4. ”LARGE-SCALE REQUIREMENTS TO
TEST LINKING”
• Traceability recovery
• Establish trace links in system after-the-fact
• ”if two artifacts share much text, they are more
likely to be associated by a link”
- Giuliano Antoniol (2002)

5. TRACEABILITY RECOVERY TOOLS

6. LUND CONTRIBUTIONS
• Johan Natt och Dag’s PhD Thesis
• ”Textual approach” to find similar requirements
• Tool: ReqSimile
• A systematic review of traceability recovery:
• Overview of techniques, evaluations, and results
• https://sites.google.com/site/tracerepo/

7. Issue
reports
Represent
Calculate angles in vector
space
TEXTUAL SIMILARITY FROM THE 60S

8. PREVIOUS FOCUS ON REQTS.
Borg, Runeson, and Ardö. Recovering from a Decade: A Systematic Mapping of Information
Retrieval Approaches to Software Traceability, Empirical Software Engineering, 19(6), 2014.
EMSE
14

9. Licentiate Thesis 2012

11. ISSUE REPORTS AS HUBS
Issue tracker

12. ISSUE MANAGEMENT MODEL
Change control board
Developers
Resolution
New issue

13. Q1. Has this issue been
reported before?
Change control board
Developers
Resolution
New issue

14. Q1. Has this issue been
reported before?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
New issue

15. Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
New issue

16. Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
New issue

17. Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue?
Q5. If we make a corrective code
change, what is the impact?
Change control board
Developers
Resolution
New issue

18. Q1. HAS THIS ISSUE BEEN
REPORTED BEFORE?

19. ISSUE DUPLICATE DETECTION
Merge

20. FINDING TEXTUAL DUPLICATES
• Per’s second most cited paper (ICSE’07 – 278 cit.)
• Applied ReqSimile approach to issue reports
• Evaluated at Sony Ericsson Mobile Communications
?
1.
2.
3.
4.
5.

21. CONTINUED PER’S WORK
• Now standard feature of issue trackers
• Bugzilla, HP Quality Center, JIRA
• Replication
• Apache Lucene search engine library
• Issue reports from Android
• Fundamental for textual analysis in later thesis work
• Good at finding similar issue report
• Highly scalable solution – fast!
Borg, Runeson, Johansson, and Mäntylä. A Replicated Study on Duplicate Detection: Using
Apache Lucene to Search among Android Defects, In Proc. of the 8th Int’l Symp. On Empirical
Software Engineering and Measurement (ESEM), 2014.
ESEM
14

22. NETWORK ANALYSIS OF ISSUE
REPORTS
Borg, Pfahl, and Runeson. Analyzing Networks of Issue Reports, In Proc. of the 17th
European Conf. on Software Maintenance and Reengineering (CSMR), 2013.
CSMR
13

23. NETWORKS ARE POWERFUL!
Fundamental for
artifact ranking in
later thesis work

24. Q2. WHO SHOULD
INVESTIGATE THE ISSUE?

25. Q1. Has this issue been
reported before?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
New issue

26. AUTOMATED ISSUE
ASSIGNMENT
• Supervised machine learning
• Train on historical bugs
Issue tracker

27. FEATURE SELECTION
• How to represent an issue report?

28. COLLABORATION WITH ERICSSON
• Ensemble learners for team
assignment
• Features
• 100 dominant terms
• Development site
• Submitter type
• System version
• Priority
Leif Jonsson

29. EXPERIMENTAL SETUP
Ericsson
Company A
4 x
Jonsson, Borg, Broman et al. Automated Bug Assignment: Ensemble-based Machine
Learning in Large-scale Industrial Contexts, Empirical Software Engineering, 2015.
EMSE
15
> 50,000

30. RESULTS
• Prediction accuracy in line with humans
• But instantaneous!
• At least 2.000 bug reports in the training set

31. A WORD OF WARNING…
• Some systems need fresh training data

32. Q3. WILL THIS ISSUE RESULT
IN A CODE CHANGE?

33. Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
New issue

34. MSC THESIS AT SONY
• Limited resources to deal with issues
• 1,000s of bugs reported in projects
• Resources to resolve roughly 10%
• Will the bug cause a code change?

35. HIGHEST PREDICTIVE POWER?
• Machine learning to find
patterns
• Empirical validation of
previous suspicions
1. mastership
2. fix for
3. ratl mastership
4. externalsupplier
5. ratl keysite
6. project
7. proj id
8. attachment share saved
9. found during
10. business priority
11. found in product
12. found by
13. abc rank
14. detection
15. impact
16. occurrence
17. priority
18. is platform
19. qa state
Standard
priority far
down the list…• Prediction accuracy 75%
Gulin and Olofsson. Development of a Decision Support
System for Defect Reports, MSc Thesis, Lund
University, 2014.
MSc.
Thesis
14

36. Q4.
How long time will it take to
correct this bug?

37. Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
New issue

38. UNSUPERVISED MACHINE LEARNING
Automatic
clustering of
issue reports

39. REPLICATION OF RAJA (2013)
• ”Textual clusters of issue reports
have significantly different
resolution times”
• Conceptual replication
• Fully automatic clustering
• Issue reports from large projects
• Confirmed statistical differences

40. OPERATIONALIZATION
1. Put new issue in the right cluster
2. Resolution time based on cluster avg.
Assar, Borg, and Pfahl. Using Text Clustering to Predict Defect Resolution Time: A Conceptual
Replication and an Evaluation of Prediction Accuracy, Empirical Software Engineering, 2015.
EMSE
15

41. Q5.
If we make a corrective change,
what is the impact?

42. Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue?
Q5. If we make a corrective code
change, what is the impact?
Change control board
Developers
Resolution
New issue

43. De la Vara, Borg, Wnuk, and Moonen. Safety Evidence Change Impact Analysis in Practice,
In revision, Transactions on Software Engineering, 2015.
TSE
(In rev.)

44. RECOMMENDATION SYSTEM
• “a software application that aims to support users
in their decision-making while interacting with
large information spaces”
Decision-support system

45. TWO MAIN APPROACHES

46. RECOMMENDATIONS BASED ON
HISTORICAL IMPACT (NON-CODE)
Reqs.
Tests
Borg, Gotel, and Wnuk. Enabling Traceability Reuse for Impact Analyses: A Feasibility Study in
a Safety Context, In Proc. of the Int’l WS on Traceability in Emerging Forms of Software
Engineering (TEFSE), 2012.
EMSE
15

47. 1. Mine issue tracker
2. Create network of
previous impact
3. Index text with

48. 4. Calculate centrality measures

49. IDENTIFY POTENTIAL IMPACT
Find similar issues
using Apache Lucene
Design Doc. X.Y
Req. X.Y
Test case UTC56
Req. Z.Y
Design Doc. X.Y
Follow links to create
candidate impact set

50. RANK THE POTENTIAL IMPACT
Use centrality measures
to rank candidate impact
1. Requirement X.Y
2. Design Document X.Y
3. Test case UTC56
4. Design Document X.Y
5. Requirement Z.Y

51. Borg and Runeson. Changes, Evolution and Bugs – Recommendation Systems for Issue
Management, In Recommendation Systems in Software Engineering, Robillard, Maalej,
Walker, and Zimmermann (Ed.), 2013.
RSSE
book
ImpRec
https://github.com/mrksbrg/ImpRec
DELIVER THE RECOMMENDATIONS

52. TWO UNIT CASE STUDY
• Deployed ImpRec in two teams
• Malmö and Bangalore
• Increases awareness and findability
• Helps project newcomers
Borg, Wnuk, Regnell, and Runeson. Supporting Change Impact Analysis Using a
Recommendation System: An Industrial Case Study in a Safety-Critical Context, In
submission, 2015.
In
subm.

53. PARAMETER TUNING
• Finding a feasible parameter
setting is difficult
• Presented framework to do it in R
• Factorial designs
• Response surface methodology
Borg. TuneR: A Framework for Tuning Software Engineering Tools with Hands-On
Instructions in R, In revision, Journal of Software: Evolution and Process, 2015.
JSEP
(in Rev)

54. SUMMARY

55. Bug tracker
Machine
Learning
“Humans obscured by bug overload, but
machine learning benefits from plentiful
training data. Practitioners confirm value of
developed tools.”
Tiny Transactions on Computer Science
(@TinyToCS), Volume 3, 2015

56. Bug tracker
Machine
Learning
Thank you!
markus.borg@sics.se
mrksbrg.com
@mrksbrg