Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Lund University - cs.lth.se/markus_borg/
Recommendation Systems for Issue
Management
MARKUS BORG, SOFTWARE ENGINEERING RES...
Lund University - cs.lth.se/markus_borg/
• PhD student 2010-
• Software engineering
• Requirements-test alignment
• Member...
Lund University - cs.lth.se/markus_borg/
Agenda
• Background
– Bug report inflow in large projects
– Recommendation system...
Lund University - cs.lth.se/markus_borg/
There Will Be Bugs!
Lund University - cs.lth.se/markus_borg/
Issue Reports as Batons in the
Development “Relay Race”
Release
Change Board
Proj...
Lund University - cs.lth.se/markus_borg/
Challenges in the Bug Overflow
• 10 000s of bug reports in large projects
– Conti...
Lund University - cs.lth.se/markus_borg/
Recommendation System
• General definition
“a software application that aim to su...
Lund University - cs.lth.se/markus_borg/
Two main approaches to recommendations
Bug
example
Lund University - cs.lth.se/markus_borg/
Q1. Which issues do we already
know about?
Duplicate Detection
Lund University - cs.lth.se/markus_borg/
Issue Duplicate Detection
Lund University - cs.lth.se/markus_borg/
Network Analysis of Bug Reports
Lund University - cs.lth.se/markus_borg/
Subgraph Analysis in Bug Networks
Bug star
One central bug report
pointing at sev...
Lund University - cs.lth.se/markus_borg/
Textual Similarity Analysis
1. Study natural
language content
2. Calculate overla...
Lund University - cs.lth.se/markus_borg/
Calculating Textual Similarity
Issue
reports
Represent
Calculate angles in
vector...
Lund University - cs.lth.se/markus_borg/
Reported Results
• Results typically reported as:
– ”% found among top 10 recomme...
Lund University - cs.lth.se/markus_borg/
Q2. Who should investigate the
issue?
Team assignment
Lund University - cs.lth.se/markus_borg/
Bug Assignment
Bug
tossing!
Lund University - cs.lth.se/markus_borg/
Machine Learning to Find Patterns
• Bugs found in India late in testing phase are...
Lund University - cs.lth.se/markus_borg/
Automated Bug Assignment –
Supervised Machine Learning
Machine
learning
Train
Rec...
Lund University - cs.lth.se/markus_borg/
How to Represent a Bug Report?
• Function
• Severity
• System Version
• Submit Da...
Lund University - cs.lth.se/markus_borg/
Experimental Setup
Ericsson
Company A
Machine
learning
4 x
Pre-processing
& featu...
Lund University - cs.lth.se/markus_borg/
Results
• Prediction accuracy using state-of-the-art machine learning:
– Ericsson...
Lund University - cs.lth.se/markus_borg/
Q3. How long time will it take to
correct this bug?
Effort prediction
Lund University - cs.lth.se/markus_borg/
Bug Report Clustering
– Unsupervised Machine Learning
Automatically cluster
bug r...
Lund University - cs.lth.se/markus_borg/
Prediction of Bug Resolution Time
1. A new bug is submitted
2. Put it in the righ...
Lund University - cs.lth.se/markus_borg/
Results
• Clustered bug reports based on textual content
– Android
– Eclipse
– Co...
Lund University - cs.lth.se/markus_borg/
Q4. If we make a corrective
change, what is the impact?
Change impact analysis
Lund University - cs.lth.se/markus_borg/
Recommendations for Impact Analysis
Mandated by safety standards!
Reqs.
Tests
Lund University - cs.lth.se/markus_borg/
Networks of Impacted Artifacts
Data mining in archive of previous
impact analysis...
Lund University - cs.lth.se/markus_borg/
Identification of Key Artifacts
Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
1. Textual similarity analysis
2. Follow links
in netwo...
Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
3. Rank recommendations
– Network statistics
– Textual ...
Lund University - cs.lth.se/markus_borg/
Lund University - cs.lth.se/markus_borg/
Results
• Experiments in a laboratory setting
– 30% of previous impact among top ...
Lund University - cs.lth.se/markus_borg/
Summary
Lund University - cs.lth.se/markus_borg/
Summary
• Issue inflow in large projects can be daunting
• On the other hand, iss...
Upcoming SlideShare
Loading in …5
×

Recommendation Systems for Issue Management

788 views

Published on

Presentation of research on recommendation systems for bug management in large software engineering projects. Contains a background and some highlights from my own research. Target audience: industry practitioners

Published in: Science
  • Be the first to comment

  • Be the first to like this

Recommendation Systems for Issue Management

  1. 1. Lund University - cs.lth.se/markus_borg/ Recommendation Systems for Issue Management MARKUS BORG, SOFTWARE ENGINEERING RESEARCH GROUP
  2. 2. Lund University - cs.lth.se/markus_borg/ • PhD student 2010- • Software engineering • Requirements-test alignment • Member of SWELL • Supervisor: Per Runeson Markus Borg Per Runeson • MSc CS and Eng. 2007 • Developer, ABB (2007-2010) • Compiler and editor development • Safety-critical systems • Embedded development
  3. 3. Lund University - cs.lth.se/markus_borg/ Agenda • Background – Bug report inflow in large projects – Recommendation systems • Overview of some ongoing bug research – Duplicate detection – Team assignment – Effort prediction – Change impact analysis
  4. 4. Lund University - cs.lth.se/markus_borg/ There Will Be Bugs!
  5. 5. Lund University - cs.lth.se/markus_borg/ Issue Reports as Batons in the Development “Relay Race” Release Change Board Project manager Tester Issue handover Developer Developer Tester Developer Architect Analyst Designer
  6. 6. Lund University - cs.lth.se/markus_borg/ Challenges in the Bug Overflow • 10 000s of bug reports in large projects – Continous inflow of bug reports – Limited resources to deal with it • Q1. Which issues do we already know about? • Q2. Who should investigate the issue? • Q3. How long time will it take to correct this bug? • Q4. If we make a corrective change, what is the impact?
  7. 7. Lund University - cs.lth.se/markus_borg/ Recommendation System • General definition “a software application that aim to support users in their decision-making while interacting with large information spaces” – Amazon, Netflix, RateBeer, Filmtipset... • In software engineering ”a software application that provides information items estimated to be valuable for a software engineering task in a given context”
  8. 8. Lund University - cs.lth.se/markus_borg/ Two main approaches to recommendations Bug example
  9. 9. Lund University - cs.lth.se/markus_borg/ Q1. Which issues do we already know about? Duplicate Detection
  10. 10. Lund University - cs.lth.se/markus_borg/ Issue Duplicate Detection
  11. 11. Lund University - cs.lth.se/markus_borg/ Network Analysis of Bug Reports
  12. 12. Lund University - cs.lth.se/markus_borg/ Subgraph Analysis in Bug Networks Bug star One central bug report pointing at several others Dense ring Most bug reports are connected
  13. 13. Lund University - cs.lth.se/markus_borg/ Textual Similarity Analysis 1. Study natural language content 2. Calculate overlap 3. Recommend duplicates
  14. 14. Lund University - cs.lth.se/markus_borg/ Calculating Textual Similarity Issue reports Represent Calculate angles in vector space
  15. 15. Lund University - cs.lth.se/markus_borg/ Reported Results • Results typically reported as: – ”% found among top 10 recommendations” • Sony Ericsson 40% • Firefox 60%, 90% • Eclipse 40%, 70% • Open Office 60%, 65% • Mozilla Foundation 45%, 65% • Android 60% (Jens?) • Are these good results? Good question.
  16. 16. Lund University - cs.lth.se/markus_borg/ Q2. Who should investigate the issue? Team assignment
  17. 17. Lund University - cs.lth.se/markus_borg/ Bug Assignment Bug tossing!
  18. 18. Lund University - cs.lth.se/markus_borg/ Machine Learning to Find Patterns • Bugs found in India late in testing phase are severe? • Bugs reports containing term ”termination” are costly? • Bugs reported on fridays with attached stack traces tend to be deferred to later releases?
  19. 19. Lund University - cs.lth.se/markus_borg/ Automated Bug Assignment – Supervised Machine Learning Machine learning Train Recommend team
  20. 20. Lund University - cs.lth.se/markus_borg/ How to Represent a Bug Report? • Function • Severity • System Version • Submit Date • Submitter Location etc. • … And the text! Title and description.
  21. 21. Lund University - cs.lth.se/markus_borg/ Experimental Setup Ericsson Company A Machine learning 4 x Pre-processing & feature selection
  22. 22. Lund University - cs.lth.se/markus_borg/ Results • Prediction accuracy using state-of-the-art machine learning: – Ericsson 30-70% – Company A 30% • At Ericsson – In line with accuracy of human assignment – Deploying tool in selected projects • Textual features improve predictions, but training times increase a lot! – Working on finding a good balance
  23. 23. Lund University - cs.lth.se/markus_borg/ Q3. How long time will it take to correct this bug? Effort prediction
  24. 24. Lund University - cs.lth.se/markus_borg/ Bug Report Clustering – Unsupervised Machine Learning Automatically cluster bug reports based on natural language in report
  25. 25. Lund University - cs.lth.se/markus_borg/ Prediction of Bug Resolution Time 1. A new bug is submitted 2. Put it in the right cluster 3. Predict its resolution time
  26. 26. Lund University - cs.lth.se/markus_borg/ Results • Clustered bug reports based on textual content – Android – Eclipse – Company A • Average resolution times in bug clusters are different – Statistically significant • Is it a good way to predict resolution times? – No, at least not yet…
  27. 27. Lund University - cs.lth.se/markus_borg/ Q4. If we make a corrective change, what is the impact? Change impact analysis
  28. 28. Lund University - cs.lth.se/markus_borg/ Recommendations for Impact Analysis Mandated by safety standards! Reqs. Tests
  29. 29. Lund University - cs.lth.se/markus_borg/ Networks of Impacted Artifacts Data mining in archive of previous impact analysis reports
  30. 30. Lund University - cs.lth.se/markus_borg/ Identification of Key Artifacts
  31. 31. Lund University - cs.lth.se/markus_borg/ Recommend Possible Impact 1. Textual similarity analysis 2. Follow links in network
  32. 32. Lund University - cs.lth.se/markus_borg/ Recommend Possible Impact 3. Rank recommendations – Network statistics – Textual similarity
  33. 33. Lund University - cs.lth.se/markus_borg/
  34. 34. Lund University - cs.lth.se/markus_borg/ Results • Experiments in a laboratory setting – 30% of previous impact among top 5 recommendations – 40% among top 10 recommendations • Prototype evaluation in industrial setting ongoing...
  35. 35. Lund University - cs.lth.se/markus_borg/ Summary
  36. 36. Lund University - cs.lth.se/markus_borg/ Summary • Issue inflow in large projects can be daunting • On the other hand, issue reports contain important information! • With the right tools, the information overflow can be tamed to recommendations!

×