Quality of Bug Reports
  in Open Source


  Nicolas Bettenburg           Adrian Schröter               Rahul Premraj
   Sa...
Developers
Developers
  175222




I wand to create a new plugin in Eclipse using CDT. Shall it
possible. I had made a R&D in eclipse...
Developers
  175222




I20030205

Run the following example. Double click on a tree item and
I wand to create a new plugi...
Researchers
Researchers
What makes a
good bug report?
Ask Google
Ask Google
Ask developers

Experts
Survey




    Five minutes
Participants




809 contacts   156 responses
Information
Question 1
Which of the following items have you previously used when fixing bugs?
(select as many items as you...
Results
Helped most




                        Used when fixing a bug
Results                                  steps to reproduce
Helped most




                                              ...
Problems
Question 3
Which of the following problems have you encountered when fixing bugs?
(select as many items as you wis...
Results
Most delay




                 Encountered when fixing a bug
Results                         incomplete information
Most delay




                                                    ...
bug bugzilla case checkbox
          behavior biggest
based

          comments common component crucial delay
clearly
det...
bug bugzilla case checkbox
          behavior biggest
based

          comments common component crucial delay
clearly
det...
Good vs. poor reports

   Create a new plugin using CDT.




   I wand to create a new plugin in Eclipse using CDT. Shall ...
Good vs. poor reports

   Create a new plugin using CDT.




   I wand to create a new plugin in Eclipse using CDT. Shall ...
Good vs. poor reports
+++ APACHE: 229 +++ ECLIPSE: 397 +++ MOZILLA: 560 +++
            Create a new plugin using CDT.



...
How can we
measure quality?
Measure quality
Report
Measure quality
                         Quality Score
Report        CUEzilla
Measure quality
                                                          Quality Score
Report                    CUEzilla...
Leave-one out validation

Training – build a model




                           Testing
Results
                 Predicted by CUEzilla

             0       0     1     0       0
             0       2     0   ...
Results
                        Predicted by CUEzilla

                    0       0     1     0       0
                 ...
Results
                        Predicted by CUEzilla

                    0       0     1     0       0
                 ...
Recommendations

     Add a stack trace
     to improve your
    bug report by 20%.
Recommendations

     Add a stack trace
     to improve your
    bug report by 20%.
Recommendations
                         Why should
                          I care?
     Add a stack trace
     to impro...
Creating incentive
Creating incentive


    Bug reports with
    stack traces get
      fixed sooner.
Mining facts
     Keywords



    Readability



    Stack Trace



    Repro Steps



   Code Examples



    Screenshots...
Mining facts
     Keywords



    Readability



    Stack Trace
                                  Time to fix
    Repro St...
Mining facts
     Keywords



    Readability



    Stack Trace

                     Correlate    Time to fix
    Repro S...
Did you know?
Bug reports containing stack traces
get fixed sooner.
(APACHE/ECLIPSE/MOZILLA)
Bug reports that are easier to...
Future work
Reporter survey
Question 1
What information have you previously provided in bug reports?
(select as many items as you wish...
Comments


    Developers		 48
    Reporters	 	 127
    Comments	 	 175
Better tool support
Better tool support
                 Add a stack trace
                 to improve your
                bug report by 20%....
bug cases cdt code create cuezilla defined delay
behaviour
developers                            duplicates eclipse
      ...
Upcoming SlideShare
Loading in …5
×

Quality of Bug Reports in Open Source

4,105 views

Published on

Presented at Dagstuhl Seminar 07491, "Mining Programs and Processes", 2007.

Published in: Technology
  • Thanks so much for sharing your thoughts on the quality of bug reports! You might also enjoy my slides on the best do's and don'ts of bug reporting: http://www.slideshare.net/usersnap/t3con15-the-best-dos-and-donts-of-a-bug-tracking-workflow
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Of course, when you have a stack trace you should add it to the bug report. But that's not the essence of the presentation. The talk shows that bug reports submitted to bug tracking systems are far from being perfect (no surprise) and what information developers actually want (Slide 15). At the end, we present a prototype tool that interactively recommends what you should add to your bug.

    In a more recent study we compared the information that developers want to the information that users actually submit (huge mismatch). However, users knew what information developers need (they just don't send it), which might indicate that they need better tool support to submit relevant information. More about this: http://thomas-zimmermann.com/better-bug-tracking/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Is the essence of this presentation, that I should add a Stacktrace when i submit a bug report?
    :-((

    - What about automatic bug reports?
    - What about HOW to communicate bug reports?
    Webservices, Mail, Central logging,...
    - What about WHAT to communicate?
    Which details I should submit?
    Screenshot? Stacktrace, Machine Infos, Environment, single dll file versions...
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Quality of Bug Reports in Open Source

  1. 1. Quality of Bug Reports in Open Source Nicolas Bettenburg Adrian Schröter Rahul Premraj Saarland University Saarland University Saarland University Sascha Just Cathrin Weiss Tom Zimmermann Saarland University University of Zürich University of Calgary
  2. 2. Developers
  3. 3. Developers 175222 I wand to create a new plugin in Eclipse using CDT. Shall it possible. I had made a R&D in eclipse documentation. I had get an idea about create a plugin using Java. But i wand to create a new plugin ( user defined plugin ) using CDT. After that I wand to impliment it in my programe. If it possible?. Any one can help me please...
  4. 4. Developers 175222 I20030205 Run the following example. Double click on a tree item and I wand to create a new plugin in Eclipse using CDT. Shall it notice that it does not expand. possible. I had made a R&D in eclipse documentation. I had get an idea about create a plugin using Java. But i wand to Comment out the Selection listener and now double click create a new plugin ( user defined plugin ) using CDT. After on any tree item and notice that it expands. that I wand to impliment it in my programe. If it possible?. Any one can help me please... public static void main(String[] args) { Display display = new Display(); ...
  5. 5. Researchers
  6. 6. Researchers
  7. 7. What makes a good bug report?
  8. 8. Ask Google
  9. 9. Ask Google
  10. 10. Ask developers Experts
  11. 11. Survey Five minutes
  12. 12. Participants 809 contacts 156 responses
  13. 13. Information Question 1 Which of the following items have you previously used when fixing bugs? (select as many items as you wish) Question 2 Which three items helped you the most? (select at most three items)
  14. 14. Results Helped most Used when fixing a bug
  15. 15. Results steps to reproduce Helped most stack traces screenshots observed behavior test cases/examples expected behavior version hardware severity Used when fixing a bug
  16. 16. Problems Question 3 Which of the following problems have you encountered when fixing bugs? (select as many items as you wish) Question 4 Which three problems caused you most delay in fixing bugs? (select at most three items)
  17. 17. Results Most delay Encountered when fixing a bug
  18. 18. Results incomplete information Most delay erroneous steps to reproduce wrong observed behaviour wrong expected behaviour wrong product information bad grammar errors in test cases duplicates wrong system information viruses/spam Encountered when fixing a bug
  19. 19. bug bugzilla case checkbox behavior biggest based comments common component crucial delay clearly determine developer error etc expected filed fixing information issue missing helpful include incomplete mozilla number people problem product project question reports reproduce reproduction scope sometimes status steps system test testcase understand used user usually version wrong
  20. 20. bug bugzilla case checkbox behavior biggest based comments common component crucial delay clearly determine developer error etc expected filed fixing information issue missing helpful include incomplete mozilla number people problem product project question reports reproduce reproduction scope sometimes status steps system test testcase understand used user usually version wrong
  21. 21. Good vs. poor reports Create a new plugin using CDT. I wand to create a new plugin in Eclipse using CDT. Shall it possible. I had made a R&D in eclipse documentation. I had get an idea about create a plugin using Java. But i wand to create a new plugin ( user defined plugin ) using CDT. After that I wand to impliment it in my programe. If it possible?. Any one can help me please...
  22. 22. Good vs. poor reports Create a new plugin using CDT. I wand to create a new plugin in Eclipse using CDT. Shall it possible. I had made a R&D in eclipse documentation. I had get an idea about create a plugin using Java. But i wand to create a new plugin ( user defined plugin ) using CDT. After that I wand to impliment it in my programe. If it possible?. Any one can help me please... Poor Average Good
  23. 23. Good vs. poor reports +++ APACHE: 229 +++ ECLIPSE: 397 +++ MOZILLA: 560 +++ Create a new plugin using CDT. I wand to create a new plugin in Eclipse using CDT. Shall it possible. I had made a R&D in eclipse documentation. I had get an idea about create a plugin using Java. But i wand to create a new plugin ( user defined plugin ) using CDT. After that I wand to impliment it in my programe. If it possible?. Any one can help me please... Poor Average Good
  24. 24. How can we measure quality?
  25. 25. Measure quality Report
  26. 26. Measure quality Quality Score Report CUEzilla
  27. 27. Measure quality Quality Score Report CUEzilla Set of Features Keywords Stack Trace Code Examples Readability Repro Steps Screenshots
  28. 28. Leave-one out validation Training – build a model Testing
  29. 29. Results Predicted by CUEzilla 0 0 1 0 0 0 2 0 0 0 Rated by developers 4 11 29 17 4 0 1 6 12 5 0 1 2 4 1
  30. 30. Results Predicted by CUEzilla 0 0 1 0 0 0 2 0 0 0 Rated by developers 4 11 29 17 4 0 1 6 12 5 0 1 2 4 1 44% agreement
  31. 31. Results Predicted by CUEzilla 0 0 1 0 0 0 2 0 0 0 Rated by developers 4 11 29 17 4 0 1 6 12 5 0 1 2 4 1 44% agreement 87% close by one
  32. 32. Recommendations Add a stack trace to improve your bug report by 20%.
  33. 33. Recommendations Add a stack trace to improve your bug report by 20%.
  34. 34. Recommendations Why should I care? Add a stack trace to improve your bug report by 20%.
  35. 35. Creating incentive
  36. 36. Creating incentive Bug reports with stack traces get fixed sooner.
  37. 37. Mining facts Keywords Readability Stack Trace Repro Steps Code Examples Screenshots Set of Features
  38. 38. Mining facts Keywords Readability Stack Trace Time to fix Repro Steps Code Examples Screenshots Set of Features Fixed or not
  39. 39. Mining facts Keywords Readability Stack Trace Correlate Time to fix Repro Steps Code Examples Screenshots Set of Features Fixed or not
  40. 40. Did you know? Bug reports containing stack traces get fixed sooner. (APACHE/ECLIPSE/MOZILLA) Bug reports that are easier to read have lower lifetimes. (APACHE/ECLIPSE/MOZILLA) Including code samples in your bug report increases the chances of it getting fixed. (MOZILLA)
  41. 41. Future work
  42. 42. Reporter survey Question 1 What information have you previously provided in bug reports? (select as many items as you wish) Question 2 What of the above information was most difficult to provide? (select at most three items) Question 3 In your opinion, what information is most relevant to developers? (select at most three items) 310 responses
  43. 43. Comments Developers 48 Reporters 127 Comments 175
  44. 44. Better tool support
  45. 45. Better tool support Add a stack trace to improve your bug report by 20%. Bug reports with stack traces get fixed sooner.
  46. 46. bug cases cdt code create cuezilla defined delay behaviour developers duplicates eclipse documentation fixing encountered examples helped impliment expected information items java mozilla observed please plugin poor possible problems programe quality report results question researchers saarland stack select test screenshots steps system shall thanks trace university user viruses wand wrong

×