Not my bug! Reasons for software bug report reassignments


Published on

Presented at CSCW 2011. Joint work with Philip J Guo, Nachiappan Nagappan, and Brendan Murphy.

Published in: Technology

Not my bug! Reasons for software bug report reassignments

  1. 1. Not my Bug! Reasons for Software Bug Report Reassignments. Philip J. Guo Stanford University Thomas Zimmermann Nachiappan Nagappan Brendan Murphy Microsoft Research© Microsoft Corporation
  2. 2. A confession…© Microsoft Corporation
  3. 3. Opinions expressed on this slide are the personal opinions of the presenter, not of Microsoft. ;-)© Microsoft Corporation
  4. 4. Backyard Safari Bug Vacuum Best Ever Bug JarPrice: $19.82 Price: $8.12 Big Bunch O Bugs Price: $8.25 Watch a Bug Price: $7.59Found viaCorporation © Microsoft
  5. 5. © Microsoft Corporation
  6. 6. © Microsoft Corporation
  7. 7. Steps toreproduceBug type How found?Assignee Severity© Microsoft Corporation
  8. 8. BUG TRACKING 101© Microsoft Corporation
  9. 9. Your software fails…© Microsoft Corporation
  10. 10. Slide provided by Sascha© Microsoft Corporation Just
  11. 11. © Microsoft Corporation Picture on the left provided by Rahul Premraj
  12. 12. Bugs with reassignments are more likely to be fixed! Windows Vista© Microsoft Corporation
  13. 13. BUG REPORT REASSIGNMENTS© Microsoft Corporation
  14. 14. Methodology Qualitative • “In your experience, what are reasons why a bug would be reassigned multiple times” survey • 358 out of 1,773 responded. Card sort. Quantitative • All bug reports for Windows Vista • Logistic regression model for bugs with analysis “excessive” reassignments Manual • Random sample of bugs with “excessive” reassignments (more than 5) inspection • 50 bug reports Follow-up • Reassignment patterns, bug pong survey • 118 out of 397 responded© Microsoft Corporation
  15. 15. What are reasons for bug report reassignments?© Microsoft Corporation
  16. 16. #1: Finding the root cause Root cause in different component. “Bugs many times are exposed in the UI [user interface], but are not caused by the team writing the UI code. “ Not enough domain knowledge. “The filer either lacked the expertise, will, or time to investigate deep enough to understand the issue at hand.” Laziness. “People are willing to do just enough to convince themselves it isn’t their problem and then reassign to the person who they think is closer to the right owner.”© Microsoft Corporation
  17. 17. #2: Determining ownership Unclear ownership. “The bug falls into an area between two teams. Say, the USB team and the WPD (Windows Portable Devices) team. The bug gets kicked around many times while the teams decide who is actually at fault.” Found in a bug report: “Dunno who gets this one, but it’s not me. I don’t have anything to do with this Component, as far as I know.” “Playing bug pong between teams who don’t agree on ownership.”© Microsoft Corporation
  18. 18. #2: Determining ownership Bug pong / Hot potato • Majority of respondents in follow-up survey replied that bug pong is “uncommon”. • However, in some situations bug pong occurs frequently: – for components shared by multiple teams, high in the system stack, or with unclear ownership; – near milestones; – for bugs with incomplete steps to reproduce.© Microsoft Corporation
  19. 19. #3: Poor bug report quality “In my experience, the most important factor in multiple reassignments is unclear bug reports. If the person assigned to the bug doesn’t understand the issue, they will either assign it back to the person who opened it, or (rarely, but it happens) assign it to the wrong person based on misunderstood information, and then it will become even worse.”© Microsoft Corporation
  20. 20. #4: Determining proper fix “There can be multiple possible fixes for a given issue which can straddle teams, so the bug can bounce back and forth until the bug fix strategy is solidified.”© Microsoft Corporation
  21. 21. #5: Workload balancing “Once the bug has found the right team, the biggest factor in reassigning is often load balancing issues across team members to drive down totals. As bug counts become more important, we’ll move issues around frequently.”© Microsoft Corporation
  22. 22. Descriptive statistical analysis • All pre- and post-release bug reports for Windows Vista until July 2009. • Logistic regression model for bug reports with excessive numbers of reassignments – We defined “excessive” as greater than 5 (90% of bugs had 5 or fewer reassignments) • Confirmed qualitative findings – Details see paper© Microsoft Corporation
  23. 23. Quantifying reassignment cycles First assignee Last assignee Cycle at the beginning First assignee Last assignee Cycle at the end© Microsoft Corporation
  24. 24. Quantifying reassignment cycles Bug reports with reassignment cycles in the beginning are more likely to be fixed. Cycle size Beginning End 2 1.11x 0.96x 3 1.10x 0.96x 4 1.12x 0.93x 5 1.04x 0.89x 6 1.07x 0.97x 7 1.03x 0.88x x is the base probability of any Windows Vista bug being successfully fixed© Microsoft Corporation
  25. 25. Cycles at the beginning • Search for correct owner • Solicit additional information “The initial bug report is incomplete or inaccurate and Alice sends back to the tester (Bob) for more information, better repro steps, etc. This is a common cycle. Once the bug is improved, it has a high likelihood of being fixed.”© Microsoft Corporation
  26. 26. Cycles at the end • Discussion on whether bug should be fixed “This example feels more like a triage cycle where Alice is the PM [program manager] (or opener) and Bob is the war team/triage team, etc. The war team is sending the bug back to PM/opener for justification why the bug should be fixed (and not punted). The fact that this conversation is happening at all means the bug is at risk and likely to be punted.” • Unclear ownership “When ABA is at the end, I think the bug is likely going back and forth between two developers, who either do not agree, or do not want the responsibility of fixing the bug.”© Microsoft Corporation
  27. 27. LESSONS LEARNED© Microsoft Corporation
  28. 28. Reassignments are part of bug fixing • Bug fixing is a highly collaborative process that involves many people. • Not all reassignments are bad. – Contrary to common belief in software engineering – Follow-up survey: developers considered only 17.6% of reassignments to be detrimental (median 10%)© Microsoft Corporation
  29. 29. Finding root causes and owners • Most reassignments are related to finding root causes and people. – Bug triagers acting as information hubs can help to reduce unneeded reassignments. • Better tools for finding code ownership and expertise are needed.© Microsoft Corporation
  30. 30. Fluid bug report assignment • Give developers a proactive role. – For example, show developers a list of bug reports that are related to them and let them pick. • Assign bug reports to multiple developers rather than just individuals. • Assign bug reports to arbitrary artifacts – Current bug tracking systems allow only assignment to people and components.© Microsoft Corporation
  31. 31. Awareness and coordination • Increase the awareness of the changes happening around bug (re)assignments. • Visualizations of reassignment pattern. • Recommender systems based on past assignment history.© Microsoft Corporation
  32. 32. Conclusions • Reassignments are beneficial to find the best person to fix a bug. • Excessive reassignments are harmful. • Primary reasons for reassignments: – finding the root cause, determining ownership, poor bug report quality, hard to determine proper fix, and workload balancing. • Role of reassignments changes over time© Microsoft Corporation