1. Characterizing and Predicting
Which Bugs Get Fixed
Philip Guo, Stanford University
Thomas Zimmermann, Microsoft Research
Nachiappan Nagappan, Microsoft Research
Brendan Murphy, Microsoft Research
© Microsoft Corporation

2. A confession…

3. Opinions expressed on this slide are the personal opinions of the presenter, not of Microsoft. ;-)

6. Incomplete bugs
Duplicate bugs

7. Too many bugs

8. Not all bugs get fixed
• Bug #1 occurs rarely and affects only a few users.
– Changes required to fix Bug #1 could be large and
expensive.
• Fixing Bug #2 could introduce new bugs
– Code changes that fix bugs are up to twice as likely to
introduce new bugs as other kinds of changes.
• Users are relying on existing behavior
– Fixing the bug could break their systems.
© Microsoft Corporation

9. Research question
How do factors related to people and
bug report edits affect whether a bug
is successfully resolved as FIXED?
→ Inefficiencies in bug triaging process
→ Inform design of tools and policies
© Microsoft Corporation

10. Bug handling is pervasive
24,000+ MSFT employees
reported, commented on, or handled
at least one bug in Windows
(vs. 2,000+ Windows developers)
© Microsoft Corporation

11. Methodology
Qualitative Quantitative
survey data analysis
Reputation • Interest • Reassignments • Distance •
Additional qualitative factors from survey
© Microsoft Corporation

12. Qualitative survey
• In your experience, how do each of these
factors affect the chances of whether a bug
will get successfully resolved as FIXED?
– 7-point Likert scale
• Sent to 1,773 Microsoft employees
– Employees who opened OR were assigned to OR
resolved most Windows Vista bugs
– 358 responded (20%)
© Microsoft Corporation

13. SURVEY: How does the bug report's activity affect the
chances of it being successfully fixed?
Severity upgraded
Severity downgraded
Few editors (<10)
Many editors (>=10)
No re-assignments (0)
Few re-assignments (1-5)
Many re-assignments (>30)
At least 1 re-open
-3 -2 -1 0 1 2 3
Decreases chances of fix Increases chances of fix
© Microsoft Corporation

14. SURVEY: How do the people involved in addressing the
bug affect the chances of it being successfully fixed?
Opener/assignee same person
Same manager
Same building
Same country
Different managers
Different countries
Opened by temp. employee
Assigned to temp. employee
-3 -2 -1 0 1 2 3
Decreases chances of fix Increases chances of fix
© Microsoft Corporation

15. Other qualitative factors
Textual quality of bug report:
“Just to re-emphasize: The quality of the bug description is very important. Not
necessarily filling in the dozens of fields in the bug database with all sorts of crap
(build numbers, dates, classifications, etc) - but just the plain-text description of
the problem, the implication and maybe even the potential solution.”
Perceived customer/business impact:
“Customer impact can be a very big impact on a bug if evidence exists to
show the cost of not fixing the bug for one or more customers.”
Rank and seniority of bug opener:
“A bug opened because something went wrong on a VPs [vice president’s]
laptop has better chance [of being fixed] than a bug opened because the
same thing happened to an intern.”
Interpersonal skills of bug opener:
“One other ‘soft’ factor is the speaking skill persuasiveness of the developer
(or other representative) when arguing for the bug.”
© Microsoft Corporation

16. Quantitative data analysis
• Correlate likelihood of bug report getting fixed
– Reputation • Interest • Reassignments • Distance
• Windows Vista and Windows 7 bug reports
– Bug lifetime: open, assign, edit, resolve, close
• Microsoft employee data
– Manager, Building, Country
© Microsoft Corporation

17. #1: Reputation
“A big influence [on bug fixes] is the
reputation of the person opening the bug.
If submitter has a history of submitting
high quality bugs then new bugs from that
person get better attention [...]”
From anonymous survey respondent
© Microsoft Corporation

18. Quantifying reputation
For each bug, calculate opener’s reputation by
aggregating over all bugs in the past.
Hooimeijer and Weimer: Modeling bug report quality. ASE 2007.
© Microsoft Corporation

19. Reputation vs. bug fixes
© Microsoft Corporation

20. Reputation vs. bug fixes
© Microsoft Corporation

21. #2: Interest
Windows Vista Windows 7
© Microsoft Corporation

22. #3: Reassignments
Severity upgraded
Severity downgraded
Few editors (<10)
Many editors (>=10)
No re-assignments (0)
Few re-assignments (1-5)
Many re-assignments (>30)
At least 1 re-open
-3 -2 -1 0 1 2 3
© Microsoft Corporation

23. #3: Reassignments
Windows Vista Windows 7
© Microsoft Corporation

24. The need for reassignments
“Bugs many times are exposed in the
UI [user interface], but are not caused
by the team writing the UI code.
These bugs can pass down several
layers of components before landing
on a lower level component owner.”
From anonymous survey respondent
© Microsoft Corporation

25. #4: Geographical distance
© Microsoft Corporation

26. #5: Organizational distance
© Microsoft Corporation

27. Trust vs. distance
“Whenever I open bugs assigned to people
I know, they are investigated thoroughly as
there is a trust in the report I write.
Often when reporting a bug within areas
where I don't know the owners, there is
inherent distrust in the bug report.”
From anonymous survey respondent
© Microsoft Corporation

28. Predictive model
• At bug opening time, predict probability of fix
• Logistic regression model with 7 factors
– Bug source
– Reputation of bug opener
– Reputation of first assignee
– Initial severity level
– Opened by temporary employee?
– Opener & initial assignee had same manager?
– Opener & initial assignee in same building?
• Train on Windows Vista, predict Windows 7 fixes
– 68% precision, 64% recall (0.50 cut-off)
© Microsoft Corporation

29. Recommendations
• Train and incentivize employees to write
higher-quality bug reports
• Improve awareness of developers’ expertise
to minimize reassignments
• Improve communication and trust amongst
people in different teams and locations
• Encourage more objectivity in prioritizing and
handling bugs
© Microsoft Corporation

30. Summary
• Characterization of which bug reports get fixed.
– At least one reassignment increases but too many
reassignments decrease the likelihood
– The higher reputation a bug opener has, the more
likely his/her bugs are to get fixed.
– Bugs handled by multiple teams and across
multiple locations are less likely to get fixed.
• Survey results provide insights into social factors
that affect the bug triaging process.
– Seniority, reputation, personal relations, and trust.
© Microsoft Corporation