Automatic Identification of
  Bug-Introducing Changes

Sunghun Kim
Kai Pan
E. James Whitehead, Jr.
                        ...
Motivation



   Bugs
Motivation
Statistics




             Bugs
Motivation
Statistics
                    Detection


             Bugs
Motivation
Statistics
                     Detection


             Bugs

                    Prediction
Motivation
Statistics
                     Detection


             Bugs

                    Prediction
  Analysis
Motivation
              Statistics
                                   Detection


                           Bugs
Observa...
So far: Focus on fixes
teicher    2003-10-29 16:11:01
fixes issues mentioned in bug 45635: [hovering] rollover hovers
- mous...
So far: Focus on fixes
teicher    2003-10-29 16:11:01
fixes issues mentioned in bug 45635: [hovering] rollover hovers
- mous...
Bug-introducing changes

BUG-INTRODUCING
...
if (foo==null) {
    foo.bar();
...
Bug-introducing changes

BUG-INTRODUCING                 FIX
...                             ...
                   later ...
Bug-introducing changes

BUG-INTRODUCING                    FIX
...                                ...
                   ...
Life-cycle of a “bug”
Life-cycle of a “bug”




BUG-INTRODUCING
CHANGE
Life-cycle of a “bug”

         BUG REPORT
         fixes issues mentioned in bug 45635: [hovering] rollover hovers
       ...
Life-cycle of a “bug”

         BUG REPORT
         fixes issues mentioned in bug 45635: [hovering] rollover hovers
       ...
The SZZ algorithm




                    1.18

                FIXED BUG
                42233
The SZZ algorithm
    $ cvs annotate -r 1.17 Foo.java
    ...
20: 1.11 (john 12-Feb-03):     return i/0;
    ...
40: 1.14 ...
The SZZ algorithm
    $ cvs annotate -r 1.17 Foo.java
    ...
20: 1.11 (john 12-Feb-03):     return i/0;
    ...
40: 1.14 ...
The SZZ algorithm
    $ cvs annotate -r 1.17 Foo.java
    ...
20: 1.11 (john 12-Feb-03):     return i/0;
    ...
40: 1.14 ...
The SZZ algorithm
    $ cvs annotate -r 1.17 Foo.java
    ...
20: 1.11 (john 12-Feb-03):     return i/0;
    ...
40: 1.14 ...
The SZZ algorithm
    $ cvs annotate -r 1.17 Foo.java
    ...
20: 1.11 (john 12-Feb-03):     return i/0;
    ...
40: 1.14 ...
The SZZ algorithm
    $ cvs annotate -r 1.17 Foo.java
    ...
20: 1.11 (john 12-Feb-03):     return i/0;
    ...
40: 1.14 ...
The SZZ algorithm




 1.11        1.14    1.16      1.18

 BUG         BUG     BUG    FIXED BUG
INTRO       INTRO   INTRO...
The SZZ algorithm
         submitted                                                              closed
               BU...
The SZZ algorithm
         submitted                                                              closed
               BU...
Drawbacks of SZZ

    Annotation by SCMs is insufficient.
(line number in bug-introducing revision is missing)
Drawbacks of SZZ

    Annotation by SCMs is insufficient.
(line number in bug-introducing revision is missing)



       No...
False negatives and positives
                  Correct   False negatives

False positives




BUG-INTRODUCING            ...
An example
BUG-INTRODUCING
void bar() {
  if (val==null) {
    println(val);
  }
...



Revision 7: tom
- introduces the d...
An example
BUG-INTRODUCING                                        BUG FIX
void bar() {                void foo() {        ...
An example
BUG-INTRODUCING                                        BUG FIX
void bar() {                void foo() {        ...
An example
BUG-INTRODUCING                                        BUG FIX
void bar() {                void foo() {        ...
An example
BUG-INTRODUCING                                        BUG FIX
void bar() {                void foo() {        ...
Our study

                        COLUMBA          ECLIPSE (jdt.core)
Project
Software type            Email client      ...
Annotation graphs
Annotation graphs



                      fixed
revisions
Annotation graphs


                      foo()

                              fixed
revisions
Annotation graphs

            bar() renamed
               to foo()
                            foo()

                  ...
Annotation graphs
bar()

                    bar() renamed
                       to foo()
                               ...
Annotation graphs
bar()

                                   bar() renamed
                                      to foo()
 ...
Annotation graphs
bar()

                                 bar() renamed
                                    to foo()
     ...
Comments & blank lines
  public void notifySourceElementRequestor() {
-
+ if (reportReferenceInfo) {
+     notifyAllUnknow...
Comments & blank lines
  public void notifySourceElementRequestor() {
-
+ if (reportReferenceInfo) {
+     notifyAllUnknow...
Format changes

BUG-INTRODUCING
if ( a==true ) return;
Format changes

BUG-INTRODUCING                         BUG FIX
if ( a==true ) return;   if (a==true)   if (a==false)
    ...
Format changes

BUG-INTRODUCING                         BUG FIX
if ( a==true ) return;   if (a==true)   if (a==false)
    ...
Format changes

BUG-INTRODUCING                         BUG FIX
if ( a==true ) return;   if (a==true)   if (a==false)
    ...
Format changes

BUG-INTRODUCING                          BUG FIX
if ( a==true ) return;   if (a==true)    if (a==false)
  ...
Fixes that affect many files

Most large fixes are refactoring
- public boolean visit(TypeDeclaration
-   typeDeclaration, B...
Fixes that affect many files

Most large fixes are refactoring
- public boolean visit(TypeDeclaration
-   typeDeclaration, B...
Manual inspection of fixes
Two judges check whether a fix is actually a fix.
    deleteResources(actualNonJavaResources,fForc...
Manual inspection of fixes
Two judges check whether a fix is actually a fix.
    deleteResources(actualNonJavaResources,fForc...
Putting it together




                      ASE

       SZZ

                            BUGS
Putting it together
       False negatives
        of SZZ: 14%




                         ASE

       SZZ

             ...
Putting it together
                  False negatives
                   of SZZ: 14%


False positives
of SZZ: 38%~51%
   ...
Putting it together
                       False negatives
                        of SZZ: 14%


False positives
of SZZ: 3...
Don’t program on Fridays ;-)
  Monday                                           11.3%


  Tuesday                         ...
Don’t program on Fridays ;-)
  Monday                                           11.3%


  Tuesday                         ...
Defect-prone authors
   Authors
(anonymized)

       Fry


      Leela


    Bender


      Amy

              0%    10%  ...
Risk awareness

“Safe” Location
    (green)



Risky Location
  (dark red)
Change classification
Change classification


bug-introducing (“bad”)

X      X       X    X
Change classification
    BUILD A CLASSIFIER
bug-introducing (“bad”)

X       X      X    X
Change classification
    BUILD A CLASSIFIER
bug-introducing (“bad”)

X       X      X    X
                          new c...
Change classification
    BUILD A CLASSIFIER
bug-introducing (“bad”)

X       X      X    X
                               ...
Conclusions
Conclusions

• Bug-introducing changes tell when a defect
  was introduced, not only its location.
Conclusions

• Bug-introducing changes tell when a defect
  was introduced, not only its location.
• We can automatically ...
Conclusions

• Bug-introducing changes tell when a defect
  was introduced, not only its location.
• We can automatically ...
More
                                       minin
                                             g @A
                      ...
Automatic Identification of Bug-Introducing Changes
Upcoming SlideShare
Loading in...5
×

Automatic Identification of Bug-Introducing Changes

1,237

Published on

Presented at ASE 2006.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,237
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
40
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Automatic Identification of Bug-Introducing Changes"

  1. 1. Automatic Identification of Bug-Introducing Changes Sunghun Kim Kai Pan E. James Whitehead, Jr. University of California, Santa Cruz, USA Tom Zimmermann Saarland University, Saarbrücken, Germany
  2. 2. Motivation Bugs
  3. 3. Motivation Statistics Bugs
  4. 4. Motivation Statistics Detection Bugs
  5. 5. Motivation Statistics Detection Bugs Prediction
  6. 6. Motivation Statistics Detection Bugs Prediction Analysis
  7. 7. Motivation Statistics Detection Bugs Observation Prediction Analysis
  8. 8. So far: Focus on fixes teicher 2003-10-29 16:11:01 fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly
  9. 9. So far: Focus on fixes teicher 2003-10-29 16:11:01 fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly Fixes give only the location of a defect, not when it was introduced.
  10. 10. Bug-introducing changes BUG-INTRODUCING ... if (foo==null) { foo.bar(); ...
  11. 11. Bug-introducing changes BUG-INTRODUCING FIX ... ... later fixed if (foo==null) { if (foo!=null) { foo.bar(); foo.bar(); ... ...
  12. 12. Bug-introducing changes BUG-INTRODUCING FIX ... ... later fixed if (foo==null) { if (foo!=null) { foo.bar(); foo.bar(); ... ... Bug-introducing changes are changes that lead to problems as indicated by later fixes.
  13. 13. Life-cycle of a “bug”
  14. 14. Life-cycle of a “bug” BUG-INTRODUCING CHANGE
  15. 15. Life-cycle of a “bug” BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly BUG-INTRODUCING CHANGE
  16. 16. Life-cycle of a “bug” BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly BUG-INTRODUCING FIX CHANGE CHANGE
  17. 17. The SZZ algorithm 1.18 FIXED BUG 42233
  18. 18. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.18 FIXED BUG 42233
  19. 19. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.18 FIXED BUG 42233
  20. 20. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 FIXED BUG 42233
  21. 21. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 BUG FIXED BUG INTRO 42233
  22. 22. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 BUG BUG FIXED BUG INTRO INTRO 42233
  23. 23. The SZZ algorithm $ cvs annotate -r 1.17 Foo.java ... 20: 1.11 (john 12-Feb-03): return i/0; ... 40: 1.14 (kate 23-May-03): return 42; ... 60: 1.16 (mary 10-Jun-03): int i=0; 1.11 1.14 1.16 1.18 BUG BUG BUG FIXED BUG INTRO INTRO INTRO 42233
  24. 24. The SZZ algorithm 1.11 1.14 1.16 1.18 BUG BUG BUG FIXED BUG INTRO INTRO INTRO 42233
  25. 25. The SZZ algorithm submitted closed BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly 1.11 1.14 1.16 1.18 BUG BUG BUG FIXED BUG INTRO INTRO INTRO 42233
  26. 26. The SZZ algorithm submitted closed BUG REPORT fixes issues mentioned in bug 45635: [hovering] rollover hovers - mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation - hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly 1.11 1.14 1.16 1.18 BUGREMOVE BUG BUG FIXED BUG INTRO INTRO POSITIVES INTRO 42233 FALSE
  27. 27. Drawbacks of SZZ Annotation by SCMs is insufficient. (line number in bug-introducing revision is missing)
  28. 28. Drawbacks of SZZ Annotation by SCMs is insufficient. (line number in bug-introducing revision is missing) Not all modifications are fixes. (blank lines, comments, etc.)
  29. 29. False negatives and positives Correct False negatives False positives BUG-INTRODUCING BUGS
  30. 30. An example BUG-INTRODUCING void bar() { if (val==null) { println(val); } ... Revision 7: tom - introduces the defects
  31. 31. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print value println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect
  32. 32. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print valueval println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect
  33. 33. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print valueval println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect
  34. 34. An example BUG-INTRODUCING BUG FIX void bar() { void foo() { void foo() { if (val==null) { // print val // print valueval println(val); if (val==null) if (val!=null) } { { ... println(val); println(val); } } ... ... Revision 7: tom Revision 23: jim Revision 42: kim - introduces the defects - inserts a comment - changes comment - reformats if statement - corrects defect The originial SZZ algorithm has too many false positives (rev 23) and false negatives (rev 7).
  35. 35. Our study COLUMBA ECLIPSE (jdt.core) Project Software type Email client IDE Investigated period 11/2002-06/2003 06/2001-03/2002 Number of revisions 500 1000 Number of fixes 143 (29%) 158 (16%) Average LOC 48,135 111,059
  36. 36. Annotation graphs
  37. 37. Annotation graphs fixed revisions
  38. 38. Annotation graphs foo() fixed revisions
  39. 39. Annotation graphs bar() renamed to foo() foo() fixed revisions
  40. 40. Annotation graphs bar() bar() renamed to foo() foo() fixed revisions
  41. 41. Annotation graphs bar() bar() renamed to foo() foo() fixed revisions SZZ reports a bug-introducing change for foo (false positive) but not for bar (false negative).
  42. 42. Annotation graphs bar() bar() renamed to foo() foo() fixed revisions Using annotation graphs we can remove 2% as false positives and identifies further 1%~4%.
  43. 43. Comments & blank lines public void notifySourceElementRequestor() { - + if (reportReferenceInfo) { + notifyAllUnknownReferences(); +} // collect the top level ast nodes int length = 0;
  44. 44. Comments & blank lines public void notifySourceElementRequestor() { - + if (reportReferenceInfo) { + notifyAllUnknownReferences(); +} // collect the top level ast nodes int length = 0; Ignoring comments and blank lines removes 14%~20% as false positives.
  45. 45. Format changes BUG-INTRODUCING if ( a==true ) return;
  46. 46. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return;
  47. 47. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return; FALSE POSITIVE
  48. 48. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return; FALSE FALSE NEGATIVE POSITIVE
  49. 49. Format changes BUG-INTRODUCING BUG FIX if ( a==true ) return; if (a==true) if (a==false) return; return; FALSE FALSE NEGATIVE POSITIVE Ignoring format changes removes 18%~25% as false positives and identifies further 13%~14%.
  50. 50. Fixes that affect many files Most large fixes are refactoring - public boolean visit(TypeDeclaration - typeDeclaration, BlockScope scope){ + public boolean visit(LocalTypeDeclaration + typeDeclaration, BlockScope scope){
  51. 51. Fixes that affect many files Most large fixes are refactoring - public boolean visit(TypeDeclaration - typeDeclaration, BlockScope scope){ + public boolean visit(LocalTypeDeclaration + typeDeclaration, BlockScope scope){ Ignoring fixes that affect many files (=more than five times the median) removes 7%~16% as false positives
  52. 52. Manual inspection of fixes Two judges check whether a fix is actually a fix. deleteResources(actualNonJavaResources,fForce); - IResource[] remaingFiles; + IResource[] remainingFiles;  try { - remaingFiles=((IFolder)res).members(); + remainingFiles=((IFolder)res).members(); }
  53. 53. Manual inspection of fixes Two judges check whether a fix is actually a fix. deleteResources(actualNonJavaResources,fForce); - IResource[] remaingFiles; + IResource[] remainingFiles;  try { - remaingFiles=((IFolder)res).members(); + remainingFiles=((IFolder)res).members(); } Manual inspection of fixes removes only 4%~5% as false positives.
  54. 54. Putting it together ASE SZZ BUGS
  55. 55. Putting it together False negatives of SZZ: 14% ASE SZZ BUGS
  56. 56. Putting it together False negatives of SZZ: 14% False positives of SZZ: 38%~51% ASE SZZ BUGS
  57. 57. Putting it together False negatives of SZZ: 14% False positives of SZZ: 38%~51% ASE SZZ Automatically we can remove 36%~48% as BUGS false positive.
  58. 58. Don’t program on Fridays ;-) Monday 11.3% Tuesday 10.4% Wednesday 11.1% Thursday 12.1% Friday 12.2% Saturday 11.7% Sunday 11.6% 0% 2.5% 5.0% 7.5% 10.0% 12.5% 15.0% Percentage of bug-introducing changes
  59. 59. Don’t program on Fridays ;-) Monday 11.3% Tuesday 10.4% Wednesday 11.1% Thursday 12.1% Friday 12.2% Saturday 11.7% Sunday 11.6% 0% 2.5% 5.0% 7.5% 10.0% 12.5% 15.0% Percentage of bug-introducing changes
  60. 60. Defect-prone authors Authors (anonymized) Fry Leela Bender Amy 0% 10% 20% 30% 40% 50% 60% 70% 80% Frequency Bug-Introducing changes Fix changes
  61. 61. Risk awareness “Safe” Location (green) Risky Location (dark red)
  62. 62. Change classification
  63. 63. Change classification bug-introducing (“bad”) X X X X
  64. 64. Change classification BUILD A CLASSIFIER bug-introducing (“bad”) X X X X
  65. 65. Change classification BUILD A CLASSIFIER bug-introducing (“bad”) X X X X new change
  66. 66. Change classification BUILD A CLASSIFIER bug-introducing (“bad”) X X X X new change PREDICT QUALITY
  67. 67. Conclusions
  68. 68. Conclusions • Bug-introducing changes tell when a defect was introduced, not only its location.
  69. 69. Conclusions • Bug-introducing changes tell when a defect was introduced, not only its location. • We can automatically identify – 36%~48% of SZZ as false positives and – further 14% of missed bug-introductions.
  70. 70. Conclusions • Bug-introducing changes tell when a defect was introduced, not only its location. • We can automatically identify – 36%~48% of SZZ as false positives and – further 14% of missed bug-introductions. • Bug-introducing changes are useful for defect prediction and software evolution.
  71. 71. More minin g @A Frida SE: y mo Conclusions rning . • Bug-introducing changes tell when a defect was introduced, not only its location. • We can automatically identify – 36%~48% of SZZ as false positives and – further 14% of missed bug-introductions. • Bug-introducing changes are useful for defect prediction and software evolution.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×