Characterizing and Predicting
          Which Bugs Get Fixed


               Philip Guo, Stanford University
            ...
A confession…
Opinions expressed on this slide are the personal opinions of the presenter, not of Microsoft. ;-)
Incomplete bugs




Duplicate bugs
Too many bugs
Not all bugs get fixed
       • Bug #1 occurs rarely and affects only a few users.
             – Changes required to fix ...
Research question

       How do factors related to people and
       bug report edits affect whether a bug
       is succ...
Bug handling is pervasive


   24,000+ MSFT employees
   reported, commented on, or handled
   at least one bug in Windows...
Methodology


                    Qualitative    Quantitative
                     survey        data analysis

       Rep...
Qualitative survey
       • In your experience, how do each of these
         factors affect the chances of whether a bug
...
SURVEY: How does the bug report's activity affect the
 chances of it being successfully fixed?
               Severity upg...
SURVEY: How do the people involved in addressing the
 bug affect the chances of it being successfully fixed?
   Opener/ass...
Other qualitative factors
           Textual quality of bug report:
           “Just to re-emphasize: The quality of the b...
Quantitative data analysis
       • Correlate likelihood of bug report getting fixed
             – Reputation • Interest ...
#1: Reputation

                          “A big influence [on bug fixes] is the
                          reputation of t...
Quantifying reputation




                  For each bug, calculate opener’s reputation by
                  aggregating ...
Reputation vs. bug fixes




© Microsoft Corporation
Reputation vs. bug fixes




© Microsoft Corporation
#2: Interest
                     Windows Vista       Windows 7




© Microsoft Corporation
#3: Reassignments
               Severity upgraded

           Severity downgraded

                Few editors (<10)

   ...
#3: Reassignments
                      Windows Vista    Windows 7




© Microsoft Corporation
The need for reassignments
                          “Bugs many times are exposed in the
                          UI [use...
#4: Geographical distance




© Microsoft Corporation
#5: Organizational distance




© Microsoft Corporation
Trust vs. distance

                          “Whenever I open bugs assigned to people
                          I know, t...
Predictive model
       • At bug opening time, predict probability of fix
       • Logistic regression model with 7 factor...
Recommendations
       • Train and incentivize employees to write
         higher-quality bug reports
       • Improve awa...
Summary
       • Characterization of which bug reports get fixed.
             – At least one reassignment increases but t...
Characterizing and predicting which bugs get fixed
Characterizing and predicting which bugs get fixed
Upcoming SlideShare
Loading in …5
×

Characterizing and predicting which bugs get fixed

1,300 views
1,218 views

Published on

Presented at ICSE 2010.

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,300
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Characterizing and predicting which bugs get fixed

  1. 1. Characterizing and Predicting Which Bugs Get Fixed Philip Guo, Stanford University Thomas Zimmermann, Microsoft Research Nachiappan Nagappan, Microsoft Research Brendan Murphy, Microsoft Research © Microsoft Corporation
  2. 2. A confession…
  3. 3. Opinions expressed on this slide are the personal opinions of the presenter, not of Microsoft. ;-)
  4. 4. Incomplete bugs Duplicate bugs
  5. 5. Too many bugs
  6. 6. Not all bugs get fixed • Bug #1 occurs rarely and affects only a few users. – Changes required to fix Bug #1 could be large and expensive. • Fixing Bug #2 could introduce new bugs – Code changes that fix bugs are up to twice as likely to introduce new bugs as other kinds of changes. • Users are relying on existing behavior – Fixing the bug could break their systems. © Microsoft Corporation
  7. 7. Research question How do factors related to people and bug report edits affect whether a bug is successfully resolved as FIXED? → Inefficiencies in bug triaging process → Inform design of tools and policies © Microsoft Corporation
  8. 8. Bug handling is pervasive 24,000+ MSFT employees reported, commented on, or handled at least one bug in Windows (vs. 2,000+ Windows developers) © Microsoft Corporation
  9. 9. Methodology Qualitative Quantitative survey data analysis Reputation • Interest • Reassignments • Distance • Additional qualitative factors from survey © Microsoft Corporation
  10. 10. Qualitative survey • In your experience, how do each of these factors affect the chances of whether a bug will get successfully resolved as FIXED? – 7-point Likert scale • Sent to 1,773 Microsoft employees – Employees who opened OR were assigned to OR resolved most Windows Vista bugs – 358 responded (20%) © Microsoft Corporation
  11. 11. SURVEY: How does the bug report's activity affect the chances of it being successfully fixed? Severity upgraded Severity downgraded Few editors (<10) Many editors (>=10) No re-assignments (0) Few re-assignments (1-5) Many re-assignments (>30) At least 1 re-open -3 -2 -1 0 1 2 3 Decreases chances of fix Increases chances of fix © Microsoft Corporation
  12. 12. SURVEY: How do the people involved in addressing the bug affect the chances of it being successfully fixed? Opener/assignee same person Same manager Same building Same country Different managers Different countries Opened by temp. employee Assigned to temp. employee -3 -2 -1 0 1 2 3 Decreases chances of fix Increases chances of fix © Microsoft Corporation
  13. 13. Other qualitative factors Textual quality of bug report: “Just to re-emphasize: The quality of the bug description is very important. Not necessarily filling in the dozens of fields in the bug database with all sorts of crap (build numbers, dates, classifications, etc) - but just the plain-text description of the problem, the implication and maybe even the potential solution.” Perceived customer/business impact: “Customer impact can be a very big impact on a bug if evidence exists to show the cost of not fixing the bug for one or more customers.” Rank and seniority of bug opener: “A bug opened because something went wrong on a VPs [vice president’s] laptop has better chance [of being fixed] than a bug opened because the same thing happened to an intern.” Interpersonal skills of bug opener: “One other ‘soft’ factor is the speaking skill persuasiveness of the developer (or other representative) when arguing for the bug.” © Microsoft Corporation
  14. 14. Quantitative data analysis • Correlate likelihood of bug report getting fixed – Reputation • Interest • Reassignments • Distance • Windows Vista and Windows 7 bug reports – Bug lifetime: open, assign, edit, resolve, close • Microsoft employee data – Manager, Building, Country © Microsoft Corporation
  15. 15. #1: Reputation “A big influence [on bug fixes] is the reputation of the person opening the bug. If submitter has a history of submitting high quality bugs then new bugs from that person get better attention [...]” From anonymous survey respondent © Microsoft Corporation
  16. 16. Quantifying reputation For each bug, calculate opener’s reputation by aggregating over all bugs in the past. Hooimeijer and Weimer: Modeling bug report quality. ASE 2007. © Microsoft Corporation
  17. 17. Reputation vs. bug fixes © Microsoft Corporation
  18. 18. Reputation vs. bug fixes © Microsoft Corporation
  19. 19. #2: Interest Windows Vista Windows 7 © Microsoft Corporation
  20. 20. #3: Reassignments Severity upgraded Severity downgraded Few editors (<10) Many editors (>=10) No re-assignments (0) Few re-assignments (1-5) Many re-assignments (>30) At least 1 re-open -3 -2 -1 0 1 2 3 © Microsoft Corporation
  21. 21. #3: Reassignments Windows Vista Windows 7 © Microsoft Corporation
  22. 22. The need for reassignments “Bugs many times are exposed in the UI [user interface], but are not caused by the team writing the UI code. These bugs can pass down several layers of components before landing on a lower level component owner.” From anonymous survey respondent © Microsoft Corporation
  23. 23. #4: Geographical distance © Microsoft Corporation
  24. 24. #5: Organizational distance © Microsoft Corporation
  25. 25. Trust vs. distance “Whenever I open bugs assigned to people I know, they are investigated thoroughly as there is a trust in the report I write. Often when reporting a bug within areas where I don't know the owners, there is inherent distrust in the bug report.” From anonymous survey respondent © Microsoft Corporation
  26. 26. Predictive model • At bug opening time, predict probability of fix • Logistic regression model with 7 factors – Bug source – Reputation of bug opener – Reputation of first assignee – Initial severity level – Opened by temporary employee? – Opener & initial assignee had same manager? – Opener & initial assignee in same building? • Train on Windows Vista, predict Windows 7 fixes – 68% precision, 64% recall (0.50 cut-off) © Microsoft Corporation
  27. 27. Recommendations • Train and incentivize employees to write higher-quality bug reports • Improve awareness of developers’ expertise to minimize reassignments • Improve communication and trust amongst people in different teams and locations • Encourage more objectivity in prioritizing and handling bugs © Microsoft Corporation
  28. 28. Summary • Characterization of which bug reports get fixed. – At least one reassignment increases but too many reassignments decrease the likelihood – The higher reputation a bug opener has, the more likely his/her bugs are to get fixed. – Bugs handled by multiple teams and across multiple locations are less likely to get fixed. • Survey results provide insights into social factors that affect the bug triaging process. – Seniority, reputation, personal relations, and trust. © Microsoft Corporation

×