Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
FROM BUGS TO
DECISION SUPPORT
- Selected Research Highlights
Markus Borg
SSE Meeting Nov 10, 2015
Per Runeson
PHD STUDENT POSITION
”Aligning RE and V&V”
Bjarnason, Runeson, Borg, et al. Challenges and Practices in Aligning Requireme...
”LARGE-SCALE REQUIREMENTS TO
TEST LINKING”
• Traceability recovery
• Establish trace links in system after-the-fact
• ”if ...
TRACEABILITY RECOVERY TOOLS
LUND CONTRIBUTIONS
• Johan Natt och Dag’s PhD Thesis
• ”Textual approach” to find similar requirements
• Tool: ReqSimile
•...
Issue
reports
Represent
Calculate angles in vector
space
TEXTUAL SIMILARITY FROM THE 60S
PREVIOUS FOCUS ON REQTS.
Borg, Runeson, and Ardö. Recovering from a Decade: A Systematic Mapping of Information
Retrieval ...
Licentiate Thesis 2012
ISSUE REPORTS AS HUBS
Issue tracker
ISSUE MANAGEMENT MODEL
Change control board
Developers
Resolution
New issue
Q1. Has this issue been
reported before?
Change control board
Developers
Resolution
New issue
Q1. Has this issue been
reported before?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
...
Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue...
Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in...
Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in...
Q1. HAS THIS ISSUE BEEN
REPORTED BEFORE?
ISSUE DUPLICATE DETECTION
Merge
FINDING TEXTUAL DUPLICATES
• Per’s second most cited paper (ICSE’07 – 278 cit.)
• Applied ReqSimile approach to issue repo...
CONTINUED PER’S WORK
• Now standard feature of issue trackers
• Bugzilla, HP Quality Center, JIRA
• Replication
• Apache L...
NETWORK ANALYSIS OF ISSUE
REPORTS
Borg, Pfahl, and Runeson. Analyzing Networks of Issue Reports, In Proc. of the 17th
Euro...
NETWORKS ARE POWERFUL!
Fundamental for
artifact ranking in
later thesis work
Q2. WHO SHOULD
INVESTIGATE THE ISSUE?
Q1. Has this issue been
reported before?
Q2. Who should investigate
the issue?
Change control board
Developers
Resolution
...
AUTOMATED ISSUE
ASSIGNMENT
• Supervised machine learning
• Train on historical bugs
Issue tracker
FEATURE SELECTION
• How to represent an issue report?
COLLABORATION WITH ERICSSON
• Ensemble learners for team
assignment
• Features
• 100 dominant terms
• Development site
• S...
EXPERIMENTAL SETUP
Ericsson
Company A
4 x
Jonsson, Borg, Broman et al. Automated Bug Assignment: Ensemble-based Machine
Le...
RESULTS
• Prediction accuracy in line with humans
• But instantaneous!
• At least 2.000 bug reports in the training set
A WORD OF WARNING…
• Some systems need fresh training data
Q3. WILL THIS ISSUE RESULT
IN A CODE CHANGE?
Q1. Has this issue been
reported before?
Q3. Will this issue result in
a code change?
Q2. Who should investigate
the issue...
MSC THESIS AT SONY
• Limited resources to deal with issues
• 1,000s of bugs reported in projects
• Resources to resolve ro...
HIGHEST PREDICTIVE POWER?
• Machine learning to find
patterns
• Empirical validation of
previous suspicions
1. mastership
...
Q4.
How long time will it take to
correct this bug?
Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in...
UNSUPERVISED MACHINE LEARNING
Automatic
clustering of
issue reports
REPLICATION OF RAJA (2013)
• ”Textual clusters of issue reports
have significantly different
resolution times”
• Conceptua...
OPERATIONALIZATION
1. Put new issue in the right cluster
2. Resolution time based on cluster avg.
Assar, Borg, and Pfahl. ...
Q5.
If we make a corrective change,
what is the impact?
Q4. How long time will it take
to correct this bug?
Q1. Has this issue been
reported before?
Q3. Will this issue result in...
De la Vara, Borg, Wnuk, and Moonen. Safety Evidence Change Impact Analysis in Practice,
In revision, Transactions on Softw...
RECOMMENDATION SYSTEM
• “a software application that aims to support users
in their decision-making while interacting with...
TWO MAIN APPROACHES
RECOMMENDATIONS BASED ON
HISTORICAL IMPACT (NON-CODE)
Reqs.
Tests
Borg, Gotel, and Wnuk. Enabling Traceability Reuse for I...
1. Mine issue tracker
2. Create network of
previous impact
3. Index text with
4. Calculate centrality measures
IDENTIFY POTENTIAL IMPACT
Find similar issues
using Apache Lucene
Design Doc. X.Y
Req. X.Y
Test case UTC56
Req. Z.Y
Design...
RANK THE POTENTIAL IMPACT
Use centrality measures
to rank candidate impact
1. Requirement X.Y
2. Design Document X.Y
3. Te...
Borg and Runeson. Changes, Evolution and Bugs – Recommendation Systems for Issue
Management, In Recommendation Systems in ...
TWO UNIT CASE STUDY
• Deployed ImpRec in two teams
• Malmö and Bangalore
• Increases awareness and findability
• Helps pro...
PARAMETER TUNING
• Finding a feasible parameter
setting is difficult
• Presented framework to do it in R
• Factorial desig...
SUMMARY
Bug tracker
Machine
Learning
“Humans obscured by bug overload, but
machine learning benefits from plentiful
training data....
Bug tracker
Machine
Learning
Thank you!
markus.borg@sics.se
mrksbrg.com
@mrksbrg
From Bugs to Decision Support - Selected Research Highlights
Upcoming SlideShare
Loading in …5
×

From Bugs to Decision Support - Selected Research Highlights

488 views

Published on

Overview of research highlights from my time as a PhD student with the Software Engineering Research Group at Lund University.

Published in: Science
  • Be the first to comment

From Bugs to Decision Support - Selected Research Highlights

  1. 1. FROM BUGS TO DECISION SUPPORT - Selected Research Highlights Markus Borg SSE Meeting Nov 10, 2015
  2. 2. Per Runeson
  3. 3. PHD STUDENT POSITION ”Aligning RE and V&V” Bjarnason, Runeson, Borg, et al. Challenges and Practices in Aligning Requirements with Verification and Validation: A Case Study of Six Companies, Empirical Software Engineering, 19(6), 2014. EMSE 14
  4. 4. ”LARGE-SCALE REQUIREMENTS TO TEST LINKING” • Traceability recovery • Establish trace links in system after-the-fact • ”if two artifacts share much text, they are more likely to be associated by a link” - Giuliano Antoniol (2002)
  5. 5. TRACEABILITY RECOVERY TOOLS
  6. 6. LUND CONTRIBUTIONS • Johan Natt och Dag’s PhD Thesis • ”Textual approach” to find similar requirements • Tool: ReqSimile • A systematic review of traceability recovery: • Overview of techniques, evaluations, and results • https://sites.google.com/site/tracerepo/
  7. 7. Issue reports Represent Calculate angles in vector space TEXTUAL SIMILARITY FROM THE 60S
  8. 8. PREVIOUS FOCUS ON REQTS. Borg, Runeson, and Ardö. Recovering from a Decade: A Systematic Mapping of Information Retrieval Approaches to Software Traceability, Empirical Software Engineering, 19(6), 2014. EMSE 14
  9. 9. Licentiate Thesis 2012
  10. 10. ISSUE REPORTS AS HUBS Issue tracker
  11. 11. ISSUE MANAGEMENT MODEL Change control board Developers Resolution New issue
  12. 12. Q1. Has this issue been reported before? Change control board Developers Resolution New issue
  13. 13. Q1. Has this issue been reported before? Q2. Who should investigate the issue? Change control board Developers Resolution New issue
  14. 14. Q1. Has this issue been reported before? Q3. Will this issue result in a code change? Q2. Who should investigate the issue? Change control board Developers Resolution New issue
  15. 15. Q4. How long time will it take to correct this bug? Q1. Has this issue been reported before? Q3. Will this issue result in a code change? Q2. Who should investigate the issue? Change control board Developers Resolution New issue
  16. 16. Q4. How long time will it take to correct this bug? Q1. Has this issue been reported before? Q3. Will this issue result in a code change? Q2. Who should investigate the issue? Q5. If we make a corrective code change, what is the impact? Change control board Developers Resolution New issue
  17. 17. Q1. HAS THIS ISSUE BEEN REPORTED BEFORE?
  18. 18. ISSUE DUPLICATE DETECTION Merge
  19. 19. FINDING TEXTUAL DUPLICATES • Per’s second most cited paper (ICSE’07 – 278 cit.) • Applied ReqSimile approach to issue reports • Evaluated at Sony Ericsson Mobile Communications ? 1. 2. 3. 4. 5.
  20. 20. CONTINUED PER’S WORK • Now standard feature of issue trackers • Bugzilla, HP Quality Center, JIRA • Replication • Apache Lucene search engine library • Issue reports from Android • Fundamental for textual analysis in later thesis work • Good at finding similar issue report • Highly scalable solution – fast! Borg, Runeson, Johansson, and Mäntylä. A Replicated Study on Duplicate Detection: Using Apache Lucene to Search among Android Defects, In Proc. of the 8th Int’l Symp. On Empirical Software Engineering and Measurement (ESEM), 2014. ESEM 14
  21. 21. NETWORK ANALYSIS OF ISSUE REPORTS Borg, Pfahl, and Runeson. Analyzing Networks of Issue Reports, In Proc. of the 17th European Conf. on Software Maintenance and Reengineering (CSMR), 2013. CSMR 13
  22. 22. NETWORKS ARE POWERFUL! Fundamental for artifact ranking in later thesis work
  23. 23. Q2. WHO SHOULD INVESTIGATE THE ISSUE?
  24. 24. Q1. Has this issue been reported before? Q2. Who should investigate the issue? Change control board Developers Resolution New issue
  25. 25. AUTOMATED ISSUE ASSIGNMENT • Supervised machine learning • Train on historical bugs Issue tracker
  26. 26. FEATURE SELECTION • How to represent an issue report?
  27. 27. COLLABORATION WITH ERICSSON • Ensemble learners for team assignment • Features • 100 dominant terms • Development site • Submitter type • System version • Priority Leif Jonsson
  28. 28. EXPERIMENTAL SETUP Ericsson Company A 4 x Jonsson, Borg, Broman et al. Automated Bug Assignment: Ensemble-based Machine Learning in Large-scale Industrial Contexts, Empirical Software Engineering, 2015. EMSE 15 > 50,000
  29. 29. RESULTS • Prediction accuracy in line with humans • But instantaneous! • At least 2.000 bug reports in the training set
  30. 30. A WORD OF WARNING… • Some systems need fresh training data
  31. 31. Q3. WILL THIS ISSUE RESULT IN A CODE CHANGE?
  32. 32. Q1. Has this issue been reported before? Q3. Will this issue result in a code change? Q2. Who should investigate the issue? Change control board Developers Resolution New issue
  33. 33. MSC THESIS AT SONY • Limited resources to deal with issues • 1,000s of bugs reported in projects • Resources to resolve roughly 10% • Will the bug cause a code change?
  34. 34. HIGHEST PREDICTIVE POWER? • Machine learning to find patterns • Empirical validation of previous suspicions 1. mastership 2. fix for 3. ratl mastership 4. externalsupplier 5. ratl keysite 6. project 7. proj id 8. attachment share saved 9. found during 10. business priority 11. found in product 12. found by 13. abc rank 14. detection 15. impact 16. occurrence 17. priority 18. is platform 19. qa state Standard priority far down the list…• Prediction accuracy 75% Gulin and Olofsson. Development of a Decision Support System for Defect Reports, MSc Thesis, Lund University, 2014. MSc. Thesis 14
  35. 35. Q4. How long time will it take to correct this bug?
  36. 36. Q4. How long time will it take to correct this bug? Q1. Has this issue been reported before? Q3. Will this issue result in a code change? Q2. Who should investigate the issue? Change control board Developers Resolution New issue
  37. 37. UNSUPERVISED MACHINE LEARNING Automatic clustering of issue reports
  38. 38. REPLICATION OF RAJA (2013) • ”Textual clusters of issue reports have significantly different resolution times” • Conceptual replication • Fully automatic clustering • Issue reports from large projects • Confirmed statistical differences
  39. 39. OPERATIONALIZATION 1. Put new issue in the right cluster 2. Resolution time based on cluster avg. Assar, Borg, and Pfahl. Using Text Clustering to Predict Defect Resolution Time: A Conceptual Replication and an Evaluation of Prediction Accuracy, Empirical Software Engineering, 2015. EMSE 15
  40. 40. Q5. If we make a corrective change, what is the impact?
  41. 41. Q4. How long time will it take to correct this bug? Q1. Has this issue been reported before? Q3. Will this issue result in a code change? Q2. Who should investigate the issue? Q5. If we make a corrective code change, what is the impact? Change control board Developers Resolution New issue
  42. 42. De la Vara, Borg, Wnuk, and Moonen. Safety Evidence Change Impact Analysis in Practice, In revision, Transactions on Software Engineering, 2015. TSE (In rev.)
  43. 43. RECOMMENDATION SYSTEM • “a software application that aims to support users in their decision-making while interacting with large information spaces” Decision-support system
  44. 44. TWO MAIN APPROACHES
  45. 45. RECOMMENDATIONS BASED ON HISTORICAL IMPACT (NON-CODE) Reqs. Tests Borg, Gotel, and Wnuk. Enabling Traceability Reuse for Impact Analyses: A Feasibility Study in a Safety Context, In Proc. of the Int’l WS on Traceability in Emerging Forms of Software Engineering (TEFSE), 2012. EMSE 15
  46. 46. 1. Mine issue tracker 2. Create network of previous impact 3. Index text with
  47. 47. 4. Calculate centrality measures
  48. 48. IDENTIFY POTENTIAL IMPACT Find similar issues using Apache Lucene Design Doc. X.Y Req. X.Y Test case UTC56 Req. Z.Y Design Doc. X.Y Follow links to create candidate impact set
  49. 49. RANK THE POTENTIAL IMPACT Use centrality measures to rank candidate impact 1. Requirement X.Y 2. Design Document X.Y 3. Test case UTC56 4. Design Document X.Y 5. Requirement Z.Y
  50. 50. Borg and Runeson. Changes, Evolution and Bugs – Recommendation Systems for Issue Management, In Recommendation Systems in Software Engineering, Robillard, Maalej, Walker, and Zimmermann (Ed.), 2013. RSSE book ImpRec https://github.com/mrksbrg/ImpRec DELIVER THE RECOMMENDATIONS
  51. 51. TWO UNIT CASE STUDY • Deployed ImpRec in two teams • Malmö and Bangalore • Increases awareness and findability • Helps project newcomers Borg, Wnuk, Regnell, and Runeson. Supporting Change Impact Analysis Using a Recommendation System: An Industrial Case Study in a Safety-Critical Context, In submission, 2015. In subm.
  52. 52. PARAMETER TUNING • Finding a feasible parameter setting is difficult • Presented framework to do it in R • Factorial designs • Response surface methodology Borg. TuneR: A Framework for Tuning Software Engineering Tools with Hands-On Instructions in R, In revision, Journal of Software: Evolution and Process, 2015. JSEP (in Rev)
  53. 53. SUMMARY
  54. 54. Bug tracker Machine Learning “Humans obscured by bug overload, but machine learning benefits from plentiful training data. Practitioners confirm value of developed tools.” Tiny Transactions on Computer Science (@TinyToCS), Volume 3, 2015
  55. 55. Bug tracker Machine Learning Thank you! markus.borg@sics.se mrksbrg.com @mrksbrg

×