Lund University - cs.lth.se/markus_borg/
Recommendation Systems for Issue
Management
MARKUS BORG, SOFTWARE ENGINEERING RES...
Lund University - cs.lth.se/markus_borg/
• PhD student 2010-
• Software engineering
• Requirements-test alignment
• Member...
Lund University - cs.lth.se/markus_borg/
Agenda
• Background
– Bug report inflow in large projects
– Recommendation system...
Lund University - cs.lth.se/markus_borg/
There Will Be Bugs!
Lund University - cs.lth.se/markus_borg/
Issue Reports as Batons in the
Development “Relay Race”
Release
Change Board
Proj...
Lund University - cs.lth.se/markus_borg/
Challenges in the Bug Overflow
• 10 000s of bug reports in large projects
– Conti...
Lund University - cs.lth.se/markus_borg/
Recommendation System
• General definition
“a software application that aim to su...
Lund University - cs.lth.se/markus_borg/
Two main approaches to recommendations
Bug
example
Lund University - cs.lth.se/markus_borg/
Q1. Which issues do we already
know about?
Duplicate Detection
Lund University - cs.lth.se/markus_borg/
Issue Duplicate Detection
Lund University - cs.lth.se/markus_borg/
Network Analysis of Bug Reports
Lund University - cs.lth.se/markus_borg/
Subgraph Analysis in Bug Networks
Bug star
One central bug report
pointing at sev...
Lund University - cs.lth.se/markus_borg/
Textual Similarity Analysis
1. Study natural
language content
2. Calculate overla...
Lund University - cs.lth.se/markus_borg/
Calculating Textual Similarity
Issue
reports
Represent
Calculate angles in
vector...
Lund University - cs.lth.se/markus_borg/
Reported Results
• Results typically reported as:
– ”% found among top 10 recomme...
Lund University - cs.lth.se/markus_borg/
Q2. Who should investigate the
issue?
Team assignment
Lund University - cs.lth.se/markus_borg/
Bug Assignment
Bug
tossing!
Lund University - cs.lth.se/markus_borg/
Machine Learning to Find Patterns
• Bugs found in India late in testing phase are...
Lund University - cs.lth.se/markus_borg/
Automated Bug Assignment –
Supervised Machine Learning
Machine
learning
Train
Rec...
Lund University - cs.lth.se/markus_borg/
How to Represent a Bug Report?
• Function
• Severity
• System Version
• Submit Da...
Lund University - cs.lth.se/markus_borg/
Experimental Setup
Ericsson
Company A
Machine
learning
4 x
Pre-processing
& featu...
Lund University - cs.lth.se/markus_borg/
Results
• Prediction accuracy using state-of-the-art machine learning:
– Ericsson...
Lund University - cs.lth.se/markus_borg/
Q3. How long time will it take to
correct this bug?
Effort prediction
Lund University - cs.lth.se/markus_borg/
Bug Report Clustering
– Unsupervised Machine Learning
Automatically cluster
bug r...
Lund University - cs.lth.se/markus_borg/
Prediction of Bug Resolution Time
1. A new bug is submitted
2. Put it in the righ...
Lund University - cs.lth.se/markus_borg/
Results
• Clustered bug reports based on textual content
– Android
– Eclipse
– Co...
Lund University - cs.lth.se/markus_borg/
Q4. If we make a corrective
change, what is the impact?
Change impact analysis
Lund University - cs.lth.se/markus_borg/
Recommendations for Impact Analysis
Mandated by safety standards!
Reqs.
Tests
Lund University - cs.lth.se/markus_borg/
Networks of Impacted Artifacts
Data mining in archive of previous
impact analysis...
Lund University - cs.lth.se/markus_borg/
Identification of Key Artifacts
Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
1. Textual similarity analysis
2. Follow links
in netwo...
Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
3. Rank recommendations
– Network statistics
– Textual ...
Lund University - cs.lth.se/markus_borg/
Lund University - cs.lth.se/markus_borg/
Results
• Experiments in a laboratory setting
– 30% of previous impact among top ...
Lund University - cs.lth.se/markus_borg/
Summary
Lund University - cs.lth.se/markus_borg/
Summary
• Issue inflow in large projects can be daunting
• On the other hand, iss...
Upcoming SlideShare
Loading in …5
×

Recommendation Systems for Issue Management

621 views

Published on

Presentation of research on recommendation systems for bug management in large software engineering projects. Contains a background and some highlights from my own research. Target audience: industry practitioners

Published in: Science
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
621
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Knowledge workers spend too much time finding infoFindability definition: “the degree to which a system or environment supports navigation and retrieval”Connection to alignment
  • Issue reports as information carriers.
  • Vi vet att buggar hänger ihop. Det här är nätverk av buggar i Android.
  • Before changes to production codeIn the safety audits to strengthen the safety case
  • Before changes to production codeIn the safety audits to strengthen the safety case
  • Before changes to production codeIn the safety audits to strengthen the safety case
  • Before changes to production codeIn the safety audits to strengthen the safety case
  • Recommendation Systems for Issue Management

    1. 1. Lund University - cs.lth.se/markus_borg/ Recommendation Systems for Issue Management MARKUS BORG, SOFTWARE ENGINEERING RESEARCH GROUP
    2. 2. Lund University - cs.lth.se/markus_borg/ • PhD student 2010- • Software engineering • Requirements-test alignment • Member of SWELL • Supervisor: Per Runeson Markus Borg Per Runeson • MSc CS and Eng. 2007 • Developer, ABB (2007-2010) • Compiler and editor development • Safety-critical systems • Embedded development
    3. 3. Lund University - cs.lth.se/markus_borg/ Agenda • Background – Bug report inflow in large projects – Recommendation systems • Overview of some ongoing bug research – Duplicate detection – Team assignment – Effort prediction – Change impact analysis
    4. 4. Lund University - cs.lth.se/markus_borg/ There Will Be Bugs!
    5. 5. Lund University - cs.lth.se/markus_borg/ Issue Reports as Batons in the Development “Relay Race” Release Change Board Project manager Tester Issue handover Developer Developer Tester Developer Architect Analyst Designer
    6. 6. Lund University - cs.lth.se/markus_borg/ Challenges in the Bug Overflow • 10 000s of bug reports in large projects – Continous inflow of bug reports – Limited resources to deal with it • Q1. Which issues do we already know about? • Q2. Who should investigate the issue? • Q3. How long time will it take to correct this bug? • Q4. If we make a corrective change, what is the impact?
    7. 7. Lund University - cs.lth.se/markus_borg/ Recommendation System • General definition “a software application that aim to support users in their decision-making while interacting with large information spaces” – Amazon, Netflix, RateBeer, Filmtipset... • In software engineering ”a software application that provides information items estimated to be valuable for a software engineering task in a given context”
    8. 8. Lund University - cs.lth.se/markus_borg/ Two main approaches to recommendations Bug example
    9. 9. Lund University - cs.lth.se/markus_borg/ Q1. Which issues do we already know about? Duplicate Detection
    10. 10. Lund University - cs.lth.se/markus_borg/ Issue Duplicate Detection
    11. 11. Lund University - cs.lth.se/markus_borg/ Network Analysis of Bug Reports
    12. 12. Lund University - cs.lth.se/markus_borg/ Subgraph Analysis in Bug Networks Bug star One central bug report pointing at several others Dense ring Most bug reports are connected
    13. 13. Lund University - cs.lth.se/markus_borg/ Textual Similarity Analysis 1. Study natural language content 2. Calculate overlap 3. Recommend duplicates
    14. 14. Lund University - cs.lth.se/markus_borg/ Calculating Textual Similarity Issue reports Represent Calculate angles in vector space
    15. 15. Lund University - cs.lth.se/markus_borg/ Reported Results • Results typically reported as: – ”% found among top 10 recommendations” • Sony Ericsson 40% • Firefox 60%, 90% • Eclipse 40%, 70% • Open Office 60%, 65% • Mozilla Foundation 45%, 65% • Android 60% (Jens?) • Are these good results? Good question.
    16. 16. Lund University - cs.lth.se/markus_borg/ Q2. Who should investigate the issue? Team assignment
    17. 17. Lund University - cs.lth.se/markus_borg/ Bug Assignment Bug tossing!
    18. 18. Lund University - cs.lth.se/markus_borg/ Machine Learning to Find Patterns • Bugs found in India late in testing phase are severe? • Bugs reports containing term ”termination” are costly? • Bugs reported on fridays with attached stack traces tend to be deferred to later releases?
    19. 19. Lund University - cs.lth.se/markus_borg/ Automated Bug Assignment – Supervised Machine Learning Machine learning Train Recommend team
    20. 20. Lund University - cs.lth.se/markus_borg/ How to Represent a Bug Report? • Function • Severity • System Version • Submit Date • Submitter Location etc. • … And the text! Title and description.
    21. 21. Lund University - cs.lth.se/markus_borg/ Experimental Setup Ericsson Company A Machine learning 4 x Pre-processing & feature selection
    22. 22. Lund University - cs.lth.se/markus_borg/ Results • Prediction accuracy using state-of-the-art machine learning: – Ericsson 30-70% – Company A 30% • At Ericsson – In line with accuracy of human assignment – Deploying tool in selected projects • Textual features improve predictions, but training times increase a lot! – Working on finding a good balance
    23. 23. Lund University - cs.lth.se/markus_borg/ Q3. How long time will it take to correct this bug? Effort prediction
    24. 24. Lund University - cs.lth.se/markus_borg/ Bug Report Clustering – Unsupervised Machine Learning Automatically cluster bug reports based on natural language in report
    25. 25. Lund University - cs.lth.se/markus_borg/ Prediction of Bug Resolution Time 1. A new bug is submitted 2. Put it in the right cluster 3. Predict its resolution time
    26. 26. Lund University - cs.lth.se/markus_borg/ Results • Clustered bug reports based on textual content – Android – Eclipse – Company A • Average resolution times in bug clusters are different – Statistically significant • Is it a good way to predict resolution times? – No, at least not yet…
    27. 27. Lund University - cs.lth.se/markus_borg/ Q4. If we make a corrective change, what is the impact? Change impact analysis
    28. 28. Lund University - cs.lth.se/markus_borg/ Recommendations for Impact Analysis Mandated by safety standards! Reqs. Tests
    29. 29. Lund University - cs.lth.se/markus_borg/ Networks of Impacted Artifacts Data mining in archive of previous impact analysis reports
    30. 30. Lund University - cs.lth.se/markus_borg/ Identification of Key Artifacts
    31. 31. Lund University - cs.lth.se/markus_borg/ Recommend Possible Impact 1. Textual similarity analysis 2. Follow links in network
    32. 32. Lund University - cs.lth.se/markus_borg/ Recommend Possible Impact 3. Rank recommendations – Network statistics – Textual similarity
    33. 33. Lund University - cs.lth.se/markus_borg/
    34. 34. Lund University - cs.lth.se/markus_borg/ Results • Experiments in a laboratory setting – 30% of previous impact among top 5 recommendations – 40% among top 10 recommendations • Prototype evaluation in industrial setting ongoing...
    35. 35. Lund University - cs.lth.se/markus_borg/ Summary
    36. 36. Lund University - cs.lth.se/markus_borg/ Summary • Issue inflow in large projects can be daunting • On the other hand, issue reports contain important information! • With the right tools, the information overflow can be tamed to recommendations!

    ×