Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Studying the Fix-Time for                Bugs in Large Open Source                          Projects                      ...
If life was like that                                      we don’t need                                  software predict...
Reality                      Many simple feature                       requests or defect                      reports do ...
4Wednesday, 21 September, 11
Feature request /       Defect report filed                              4Wednesday, 21 September, 11
Feature request /       Defect report filed                              Triage                                       4Wedn...
Implementation                                       plan / cause of defect       Feature request /                       ...
Implementation                                       plan / cause of defect       Feature request /                       ...
Implementation                                       plan / cause of defect       Feature request /                       ...
Implementation                                       plan / cause of defect       Feature request /                       ...
Implementation                                       plan / cause of defect       Feature request /                       ...
Implementation                                                plan / cause of defect       Feature request /              ...
Implementation                                                plan / cause of defect       Feature request /              ...
Implementation                                                plan / cause of defect       Feature request /              ...
Location                 properties                    (7)                              5Wednesday, 21 September, 11
Product /           Version /          Component                  Location                 properties                    (...
Product /             Number           Version /          of completed          Component               WI*               ...
Product /             Number           Version /          of completed          Component               WI*               ...
Location                 properties                    (7)                 Reporter                 properties            ...
Location                 properties     Industry /                              local / public                    (7)     ...
Location                 properties     Industry /                              local / public                    (7)     ...
Location                 properties     Industry /                              local / public                    (7)     ...
Location                 properties     Industry /                              local / public                    (7)     ...
Location                 properties                    (7)                 Reporter                 properties            ...
Location                 properties                    (7)                 Reporter                 properties   Severity ...
Location                 properties                    (7)                 Reporter                 properties   Severity ...
Location                 properties                    (7)                 Reporter                 properties   Severity ...
Location                 properties                    (7)                 Reporter                 properties   Severity ...
Location                 properties                    (7)                 Reporter                 properties       Sever...
Location                 properties                    (7)                 Reporter                              Predictor...
When will it be                                            fixed?                  Location                 properties     ...
When will it be                                            fixed?                  Location                                ...
When will it be                                            fixed?                  Location                                ...
When will it be                                            fixed?                  Location                                ...
Which                              one should we fix                                  this iteration?                  Loca...
Which                              one should we fix                                  this iteration?                  Loca...
Which                              one should we fix                                  this iteration?                  Loca...
Which                              one should we fix                                  this iteration?                  Loca...
Case study                    Number of                               <3             <1     <3            Project   work  ...
Random Forest                       We use Random Forest because:                         • Decision tree based models are...
Random                              Data                     J48                                                  M5P     ...
Goals of our case study                   • G1: What is the accuracy of fix-time                          prediction model?...
G1: Accuracy of the                                    model                      We build 10 random forests for each     ...
Accuracy of the model:               Overall misclassification                              13Wednesday, 21 September, 11
Accuracy of the model:               Overall misclassification    60               51.9                                    ...
Accuracy of the model:               Overall misclassification    60                                                       ...
Accuracy of the model:                      G1: We can correctly classify ∼65% of the                      time the fix-tim...
G2: Model sensitivity -                Importance of each property                      We use technique called permutatio...
Location                    Mozilla              Eclipse                                   Project fix-time                ...
Location                              Reporter                    Mozilla              Eclipse          Mozilla           ...
Location                               Reporter                    Mozilla               Eclipse           Mozilla        ...
Location                               Reporter                    Mozilla               Eclipse           Mozilla        ...
Location                              Reporter                    Mozilla              Eclipse          Mozilla           ...
(18%)   (7%)   (0%)   (7                  Why is time the the Resolution Type of Bu                        Table 5: Statis...
)               (7%)              (9%)   (59%) (100%)               Why is timeProjects mostype of Bugs for the Mozilla an...
G3: How applicable are                  models in practice?                      If a prediction model is stable, it shoul...
Implementation                                          plan / cause of defect            Feature request /               ...
Feature request /            Defect report filed                                 Triage                                    ...
Number of CCed?            Feature request /            Defect report filed                                 Triage         ...
Number of CCed?                                                        Serverity change!            Feature request /     ...
Assigned to                                           Number of CCed?             someone else                            ...
Accuracy of the predictor models                 using only available properties                                          ...
Stability of the mode               • Training size stability: As more data is added                      to the training ...
Apache - Training size                          0.70                          0.65                          0.60          ...
Apache - Time stability               Precisions                            0.42                            0.40          ...
Apache - Time stability                                   G3: Fix-time prediction model may                            0.4...
24Wednesday, 21 September, 11
24Wednesday, 21 September, 11
24Wednesday, 21 September, 11
24Wednesday, 21 September, 11
Upcoming SlideShare
Loading in …5
×

Promise 2011: "Studying the Fix-Time for Bugs in Large Open Source Projects"

2,828 views

Published on

Promise 2011:
"Studying the Fix-Time for Bugs in Large Open Source Projects"
Lionel Marks, Ahmed E. Hassan and Ying Zou.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Promise 2011: "Studying the Fix-Time for Bugs in Large Open Source Projects"

  1. 1. Studying the Fix-Time for Bugs in Large Open Source Projects Lionel Marks, Ying Zou, Ahmed E. Hassan, Thanh Nguyen Queen’s University, Kingston, Ontario, Canada 1Wednesday, 21 September, 11
  2. 2. If life was like that we don’t need software prediction. 2Wednesday, 21 September, 11
  3. 3. Reality Many simple feature requests or defect reports do NOT get fixed for years. 3Wednesday, 21 September, 11
  4. 4. 4Wednesday, 21 September, 11
  5. 5. Feature request / Defect report filed 4Wednesday, 21 September, 11
  6. 6. Feature request / Defect report filed Triage 4Wednesday, 21 September, 11
  7. 7. Implementation plan / cause of defect Feature request / determined Defect report filed Triage 4Wednesday, 21 September, 11
  8. 8. Implementation plan / cause of defect Feature request / determined Defect report filed Triage Implement 4Wednesday, 21 September, 11
  9. 9. Implementation plan / cause of defect Feature request / determined Defect report filed Triage Verify Implement 4Wednesday, 21 September, 11
  10. 10. Implementation plan / cause of defect Feature request / Close determined Defect report filed Triage Verify Implement 4Wednesday, 21 September, 11
  11. 11. Implementation plan / cause of defect Feature request / Close determined Defect report filed Triage Verify Implement Work item fix-time 4Wednesday, 21 September, 11
  12. 12. Implementation plan / cause of defect Feature request / Close determined Defect report filed Triage Verify Implement Work item fix-time When will it be fixed? 4Wednesday, 21 September, 11
  13. 13. Implementation plan / cause of defect Feature request / Close determined Defect report filed Triage Verify Implement Work item fix-time When will it be Which fixed? one should we fix this iteration? 4Wednesday, 21 September, 11
  14. 14. Implementation plan / cause of defect Feature request / Close determined Defect report filed Triage Verify Implement Work item fix-time When will it be Which fixed? one should we fix this iteration? Can we predict the work item fix-time? 4Wednesday, 21 September, 11
  15. 15. Location properties (7) 5Wednesday, 21 September, 11
  16. 16. Product / Version / Component Location properties (7) 5Wednesday, 21 September, 11
  17. 17. Product / Number Version / of completed Component WI* Location properties (7) 5Wednesday, 21 September, 11
  18. 18. Product / Number Version / of completed Component WI* Average fix time* Location properties (7) 5Wednesday, 21 September, 11
  19. 19. Location properties (7) Reporter properties (4) 5Wednesday, 21 September, 11
  20. 20. Location properties Industry / local / public (7) Reporter properties (4) 5Wednesday, 21 September, 11
  21. 21. Location properties Industry / local / public (7) Popularity* Reporter properties (4) 5Wednesday, 21 September, 11
  22. 22. Location properties Industry / local / public (7) Popularity* Reporter properties Number (4) of past requests* 5Wednesday, 21 September, 11
  23. 23. Location properties Industry / local / public (7) Popularity* Reporter properties Number (4) of past requests* Average fix time* 5Wednesday, 21 September, 11
  24. 24. Location properties (7) Reporter properties (4) Work item properties (12) 5Wednesday, 21 September, 11
  25. 25. Location properties (7) Reporter properties Severity / Priority (4) Work item properties (12) 5Wednesday, 21 September, 11
  26. 26. Location properties (7) Reporter properties Severity / Priority (4) Number of interested parties* Work item properties (12) 5Wednesday, 21 September, 11
  27. 27. Location properties (7) Reporter properties Severity / Priority (4) Number of interested parties* Work item properties Morning / Day / night (12) 5Wednesday, 21 September, 11
  28. 28. Location properties (7) Reporter properties Severity / Priority (4) Number of interested parties* Work item properties Morning / Day / night (12) Description length* 5Wednesday, 21 September, 11
  29. 29. Location properties (7) Reporter properties Severity / Priority (4) Number of interested parties* Work item properties Morning / Day / night (12) Code Description attachment length* 5Wednesday, 21 September, 11
  30. 30. Location properties (7) Reporter Predictor properties (4) Work item properties (12) 5Wednesday, 21 September, 11
  31. 31. When will it be fixed? Location properties (7) Reporter Predictor properties (4) Work item properties (12) 6Wednesday, 21 September, 11
  32. 32. When will it be fixed? Location Short properties (7) Reporter Predictor properties (4) Work item properties (12) 6Wednesday, 21 September, 11
  33. 33. When will it be fixed? Location Short properties (7) Reporter Predictor Normal properties (4) Work item properties (12) 6Wednesday, 21 September, 11
  34. 34. When will it be fixed? Location Short properties (7) Reporter Predictor Normal properties (4) Work item Long properties (12) 6Wednesday, 21 September, 11
  35. 35. Which one should we fix this iteration? Location properties (7) Reporter Predictor properties (4) Work item properties (12) 7Wednesday, 21 September, 11
  36. 36. Which one should we fix this iteration? Location Next minor properties revision (7) Reporter Predictor properties (4) Work item properties (12) 7Wednesday, 21 September, 11
  37. 37. Which one should we fix this iteration? Location Next minor properties revision (7) Reporter Next major Predictor properties revision (4) Work item properties (12) 7Wednesday, 21 September, 11
  38. 38. Which one should we fix this iteration? Location Next minor properties revision (7) Reporter Next major Predictor properties revision (4) Work item Next properties version (12) 7Wednesday, 21 September, 11
  39. 39. Case study Number of <3 <1 <3 Project work months year years items Mozilla 85,616 46% 27% 27% Eclipse 63,402 76% 18% 6% 8Wednesday, 21 September, 11
  40. 40. Random Forest We use Random Forest because: • Decision tree based models are explainable comparing to SVM or neural network. • Random Forest out-performs C4.5 because it is more resistive to data with highly correlated attributes. • It is easy to analyze the sensitivity of each property. 9Wednesday, 21 September, 11
  41. 41. Random Data J48 M5P Forest Linux* 11.09 (3.09) 18.42 (2.53) 16.22(2.19) Apache* 38.64 (1.35) 34.45 (1.68) 25.39 (1.64) Jazz 17.75 (2.76) 18.43 (2.74) 11.67 (2.27) *Akinori Ihara (Kinki University, Japan) and Yasutaka Kamei (Kyushu University, Japan) 10Wednesday, 21 September, 11
  42. 42. Goals of our case study • G1: What is the accuracy of fix-time prediction model? • G2: Which properties are the most important predictors of fix-time? • G3: How applicable are models in practice? 11Wednesday, 21 September, 11
  43. 43. G1: Accuracy of the model We build 10 random forests for each project: • Each random forest uses randomly 2/3 of the data for training. • We evaluate the prediction on the rest 1/3 of the data 12Wednesday, 21 September, 11
  44. 44. Accuracy of the model: Overall misclassification 13Wednesday, 21 September, 11
  45. 45. Accuracy of the model: Overall misclassification 60 51.9 50.8 45 39.6 36.2 30 15 0 Reporter Location Description All Mozilla 13Wednesday, 21 September, 11
  46. 46. Accuracy of the model: Overall misclassification 60 50 51.9 50.8 43 45 37.5 39.6 35.8 34.7 36.2 32.8 30 25 15 12.5 0 0 Reporter Location Description All Reporter Location Description All Mozilla Eclipse 13Wednesday, 21 September, 11
  47. 47. Accuracy of the model: G1: We can correctly classify ∼65% of the time the fix-time for work items in Eclipse and Overall misclassification Mozilla, twice as good as random. 60 50 51.9 50.8 43 45 37.5 39.6 35.8 34.7 36.2 32.8 30 25 15 12.5 0 0 Reporter Location Description All Reporter Location Description All Mozilla Eclipse 13Wednesday, 21 September, 11
  48. 48. G2: Model sensitivity - Importance of each property We use technique called permutation accuracy importance measure as follow: • For each property, we randomly alter values and rerun the classification. • We give an importance score (1 to 10) depend on the change in the classification result. • We sum its score across all ten forests. 14Wednesday, 21 September, 11
  49. 49. Location Mozilla Eclipse Project fix-time Product Product opened Component work items 15Wednesday, 21 September, 11
  50. 50. Location Reporter Mozilla Eclipse Mozilla Eclipse Project fix-time Fix-time Fix-time Product Product opened Requests Overall popularity Component work items 15Wednesday, 21 September, 11
  51. 51. Location Reporter Mozilla Eclipse Mozilla Eclipse Project fix-time Fix-time Fix-time Product Product opened Requests Overall popularity Component work items Description Mozilla Eclipse Year Year Has target milestone Severity 15Wednesday, 21 September, 11
  52. 52. Location Reporter Mozilla Eclipse Mozilla Eclipse Project fix-time Fix-time Fix-time Product Product opened Requests Overall popularity Component work items Description All Mozilla Eclipse Mozilla Eclipse Year Year Year Severity Has target milestone Severity Product Number of CCed 15Wednesday, 21 September, 11
  53. 53. Location Reporter Mozilla Eclipse Mozilla Eclipse Project fix-time Fix-time Fix-time Product Product opened Requests Overall popularity Component work items G2: The time of bug filing and its location are the most important Description in the Mozilla project. In the properties All Eclipse project, bug severity is the Mozilla Eclipse Mozilla Eclipse most important property. Year Year Year Severity Has target milestone Severity Product Number of CCed 15Wednesday, 21 September, 11
  54. 54. (18%) (7%) (0%) (7 Why is time the the Resolution Type of Bu Table 5: Statistics for most important factor in Mozilla? (a) Mozilla 16Wednesday, 21 September, 11
  55. 55. ) (7%) (9%) (59%) (100%) Why is timeProjects mostype of Bugs for the Mozilla and Eclipse the important factor in Mozilla? (b) Eclipse 17 Wednesday, 21 September, 11
  56. 56. G3: How applicable are models in practice? If a prediction model is stable, it should: • Use only available properties • Be stable 18Wednesday, 21 September, 11
  57. 57. Implementation plan / cause of defect Feature request / Close determined Defect report filed Triage Verify Implement Work item fix-timeWednesday, 21 September, 11
  58. 58. Feature request / Defect report filed Triage Work item fix-timeWednesday, 21 September, 11
  59. 59. Number of CCed? Feature request / Defect report filed Triage Work item fix-timeWednesday, 21 September, 11
  60. 60. Number of CCed? Serverity change! Feature request / Defect report filed Triage Work item fix-timeWednesday, 21 September, 11
  61. 61. Assigned to Number of CCed? someone else Serverity change! Feature request / Defect report filed Triage Work item fix-timeWednesday, 21 September, 11
  62. 62. Accuracy of the predictor models using only available properties Data size Misclassification rate Eclipse* 86490 0.51 Linux* 2024 0.55 Jazz 16672 0.57 Apache* 1466 0.37 *Akinori Ihara (Kinki University, Japan) and Yasutaka Kamei (Kyushu University, Japan) 20Wednesday, 21 September, 11
  63. 63. Stability of the mode • Training size stability: As more data is added to the training set, the accuracy should improve. • Time stability: The accuracy should be stable overtime. 21Wednesday, 21 September, 11
  64. 64. Apache - Training size 0.70 0.65 0.60 stability Precisions 0.55 0.50 0.45 1 2 3 4 5 6 7 8 9 x 10% of training data 22Wednesday, 21 September, 11
  65. 65. Apache - Time stability Precisions 0.42 0.40 0.38 0.36 0.34 1 2 3 4 5 6 7 8 x 10% of training data 23Wednesday, 21 September, 11
  66. 66. Apache - Time stability G3: Fix-time prediction model may 0.42 work on project such as Apache in 0.40 practice. Apache prediction model Precisions have data stability and time 0.38 stability. 0.36 0.34 1 2 3 4 5 6 7 8 x 10% of training data 23Wednesday, 21 September, 11
  67. 67. 24Wednesday, 21 September, 11
  68. 68. 24Wednesday, 21 September, 11
  69. 69. 24Wednesday, 21 September, 11
  70. 70. 24Wednesday, 21 September, 11

×