Automatic Identification of Bug-Introducing Changes

1,388
-1

Published on

Presented at ASE 2006.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,388
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
43
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Automatic Identification of Bug-Introducing Changes

  1. 1. Automatic Identification of Bug-Introducing Changes Sunghun Kim Kai Pan E. James Whitehead, Jr. University of California, Santa Cruz, USA Tom Zimmermann Saarland University, Saarbrücken, Germany
  2. 2. Motivation Bugs
  3. 3. Motivation Statistics Bugs
  4. 4. Motivation Statistics Detection Bugs
  5. 5. Motivation Statistics Detection Bugs Prediction
  6. 6. Motivation Statistics Detection Bugs Prediction Analysis
  7. 7. Motivation Statistics Detection Bugs Observation Prediction Analysis
  8. 8. So far: Focus on fixes teicher 2003-10-29 16:11:01 fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly
  9. 9. So far: Focus on fixes teicher 2003-10-29 16:11:01 fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly Fixes give only the location of a defect, not when it was introduced.
  10. 10. Bug-introducing changes BUG-INTRODUCING ... if (foo==null) { foo.bar(); ...
  11. 11. Bug-introducing changes BUG-INTRODUCING FIX ... ... later fixed if (foo==null) { if (foo!=null) { foo.bar(); foo.bar(); ... ...
  12. 12. Bug-introducing changes BUG-INTRODUCING FIX ... ... later fixed if (foo==null) { if (foo!=null) { foo.bar(); foo.bar(); ... ... Bug-introducing changes are changes that lead to problems as indicated by later fixes.
  13. 13. Life-cycle of a “bug”
  14. 14. Life-cycle of a “bug” BUG-INTRODUCING CHANGE
  15. 15. Life-cycle of a “bug” BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly BUG-INTRODUCING CHANGE
  16. 16. Life-cycle of a “bug” BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly BUG-INTRODUCING FIX CHANGE CHANGE
  17. 17. The SZZ algorithm 1.18 FIXED BUG 42233
  18. 18. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.18 FIXED BUG 42233
  19. 19. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.18 FIXED BUG 42233
  20. 20. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 FIXED BUG 42233
  21. 21. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 BUG FIXED BUG INTRO 42233
  22. 22. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 BUG BUG FIXED BUG INTRO INTRO 42233
  23. 23. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 BUG BUG BUG FIXED BUG INTRO INTRO INTRO 42233
  24. 24. The SZZ algorithm 1.11 1.14 1.16 1.18 BUG BUG BUG FIXED BUG INTRO INTRO INTRO 42233
  25. 25. The SZZ algorithm submitted closed BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly 1.11 1.14 1.16 1.18 BUG BUG BUG FIXED BUG INTRO INTRO INTRO 42233
  26. 26. The SZZ algorithm submitted closed BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly 1.11 1.14 1.16 1.18 BUGREMOVE BUG BUG FIXED BUG INTRO INTRO POSITIVES INTRO 42233 FALSE
  27. 27. Drawbacks of SZZ Annotation by SCMs is insufficient. (line number in bug-introducing revision is missing)
  28. 28. Drawbacks of SZZ Annotation by SCMs is insufficient. (line number in bug-introducing revision is missing) Not all modifications are fixes. (blank lines, comments, etc.)
  29. 29. False negatives and positives Correct False negatives False positives BUG-INTRODUCING BUGS
  30. 30. An example BUG-INTRODUCING void bar() { if (val==null) { println(val); } ... Revision 7: tom - introduces the defects
  31. 31. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print value println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect
  32. 32. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print valueval println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect
  33. 33. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print valueval println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect
  34. 34. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print valueval println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect The originial SZZ algorithm has too many false positives (rev 23) and false negatives (rev 7).
  35. 35. Our study COLUMBA ECLIPSE (jdt.core) Project Software type Email client IDE Investigated period 11/2002-06/2003 06/2001-03/2002 Number of revisions 500 1000 Number of fixes 143 (29%) 158 (16%) Average LOC 48,135 111,059
  36. 36. Annotation graphs
  37. 37. Annotation graphs fixed revisions
  38. 38. Annotation graphs foo() fixed revisions
  39. 39. Annotation graphs bar() renamed to foo() foo() fixed revisions
  40. 40. Annotation graphs bar() bar() renamed to foo() foo() fixed revisions
  41. 41. Annotation graphs bar() bar() renamed to foo() foo() fixed revisions SZZ reports a bug-introducing change for foo (false positive) but not for bar (false negative).
  42. 42. Annotation graphs bar() bar() renamed to foo() foo() fixed revisions Using annotation graphs we can remove 2% as false positives and identifies further 1%~4%.
  43. 43. Comments & blank lines public void notifySourceElementRequestor() { - + if (reportReferenceInfo) { + notifyAllUnknownReferences(); +} // collect the top level ast nodes int length = 0;
  44. 44. Comments & blank lines public void notifySourceElementRequestor() { - + if (reportReferenceInfo) { + notifyAllUnknownReferences(); +} // collect the top level ast nodes int length = 0; Ignoring comments and blank lines removes 14%~20% as false positives.
  45. 45. Format changes BUG-INTRODUCING if ( a==true ) return;
  46. 46. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return;
  47. 47. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return; FALSE POSITIVE
  48. 48. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return; FALSE FALSE NEGATIVE POSITIVE
  49. 49. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return; FALSE FALSE NEGATIVE POSITIVE Ignoring format changes removes 18%~25% as false positives and identifies further 13%~14%.
  50. 50. Fixes that affect many files Most large fixes are refactoring - public boolean visit(TypeDeclaration - typeDeclaration, BlockScope scope){ + public boolean visit(LocalTypeDeclaration + typeDeclaration, BlockScope scope){
  51. 51. Fixes that affect many files Most large fixes are refactoring - public boolean visit(TypeDeclaration - typeDeclaration, BlockScope scope){ + public boolean visit(LocalTypeDeclaration + typeDeclaration, BlockScope scope){ Ignoring fixes that affect many files (=more than five times the median) removes 7%~16% as false positives
  52. 52. Manual inspection of fixes Two judges check whether a fix is actually a fix. deleteResources(actualNonJavaResources,fForce); - IResource[] remaingFiles; + IResource[] remainingFiles;  try { - remaingFiles=((IFolder)res).members(); + remainingFiles=((IFolder)res).members(); }
  53. 53. Manual inspection of fixes Two judges check whether a fix is actually a fix. deleteResources(actualNonJavaResources,fForce); - IResource[] remaingFiles; + IResource[] remainingFiles;  try { - remaingFiles=((IFolder)res).members(); + remainingFiles=((IFolder)res).members(); } Manual inspection of fixes removes only 4%~5% as false positives.
  54. 54. Putting it together ASE SZZ BUGS
  55. 55. Putting it together False negatives of SZZ: 14% ASE SZZ BUGS
  56. 56. Putting it together False negatives of SZZ: 14% False positives of SZZ: 38%~51% ASE SZZ BUGS
  57. 57. Putting it together False negatives of SZZ: 14% False positives of SZZ: 38%~51% ASE SZZ Automatically we can remove 36%~48% as BUGS false positive.
  58. 58. Don’t program on Fridays ;-) Monday 11.3% Tuesday 10.4% Wednesday 11.1% Thursday 12.1% Friday 12.2% Saturday 11.7% Sunday 11.6% 0% 2.5% 5.0% 7.5% 10.0% 12.5% 15.0% Percentage of bug-introducing changes
  59. 59. Don’t program on Fridays ;-) Monday 11.3% Tuesday 10.4% Wednesday 11.1% Thursday 12.1% Friday 12.2% Saturday 11.7% Sunday 11.6% 0% 2.5% 5.0% 7.5% 10.0% 12.5% 15.0% Percentage of bug-introducing changes
  60. 60. Defect-prone authors Authors (anonymized) Fry Leela Bender Amy 0% 10% 20% 30% 40% 50% 60% 70% 80% Frequency Bug-Introducing changes Fix changes
  61. 61. Risk awareness “Safe” Location (green) Risky Location (dark red)
  62. 62. Change classification
  63. 63. Change classification bug-introducing (“bad”) X X X X
  64. 64. Change classification BUILD A CLASSIFIER bug-introducing (“bad”) X X X X
  65. 65. Change classification BUILD A CLASSIFIER bug-introducing (“bad”) X X X X new change
  66. 66. Change classification BUILD A CLASSIFIER bug-introducing (“bad”) X X X X new change PREDICT QUALITY
  67. 67. Conclusions
  68. 68. Conclusions • Bug-introducing changes tell when a defect was introduced, not only its location.
  69. 69. Conclusions • Bug-introducing changes tell when a defect was introduced, not only its location. • We can automatically identify – 36%~48% of SZZ as false positives and – further 14% of missed bug-introductions.
  70. 70. Conclusions • Bug-introducing changes tell when a defect was introduced, not only its location. • We can automatically identify – 36%~48% of SZZ as false positives and – further 14% of missed bug-introductions. • Bug-introducing changes are useful for defect prediction and software evolution.
  71. 71. More minin g @A Frida SE: y mo Conclusions rning . • Bug-introducing changes tell when a defect was introduced, not only its location. • We can automatically identify – 36%~48% of SZZ as false positives and – further 14% of missed bug-introductions. • Bug-introducing changes are useful for defect prediction and software evolution.

×