Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Sung Kim and Michael D. Ernst {hunkim, mernst}@csail.mit.edu CSAIL  • MIT
Sung Kim and Michael D. Ernst {hunkim, mernst}@csail.mit.edu CSAIL  • MIT
Static analysis I can find  bugs  for you!
Bug-finding tool
Warnings
Warning examples
Bug candidate - Warning Category : Style -  empty try/catch Priority : 3
Are warnings useful? Are all warnings important to be fixed? Does tools’ prioritization help to identify important ones?
Experimental methodology <ul><li>Select source code (a snapshot) </li></ul><ul><li>Mark buggy lines </li></ul><ul><ul><li>...
Subject tools and projects <ul><li>Three bug-finding tools </li></ul><ul><ul><li>FindBugs, JLint, and PMD </li></ul></ul><...
Buggy lines Rev  100 …… Mark as a  buggy   line   if a line is changed in future fixes.
Change logs indicate fixes …… [bugfix]  Fixed bug #3576 fix File 1 Non-fix Non-fix Non-fix Non-fix
Buggy lines Rev 199 Rev 200 Fix If (x=y && Z=x) {
Buggy lines Rev 199 Rev 198 If (x=y && Z=x) { If (x=y && Z=x) { marks Fix
Buggy lines Rev  100 Rev  200 Rev  199 Rev  198 ……
Warning evaluation Rev 100 50% of warnings are fixed later
Evaluate warnings at revision n/2 Rev n/2 Rev n Mark buggy lines Rev n/2+1
Are warnings important? NO only 3~18% are fixed!
Does tools’ prioritization help? NO Tools’ priority 1 warnings  are not better.
Does tools’ prioritization help?
Does tools’ prioritization help?
Does tools’ prioritization help? No, high priority warnings do not have high precision!
…… Our solution: using change  history Warning A File Warning B Foo ……
<ul><li>Compute score of each warning category using history </li></ul><ul><ul><li>Observe warning removals in history </l...
Reprioritization algorithm …… Score(A)  += non-fix-weight (0.1) Score(A) += fix-weight (0.9) Fixed bug #3576 Score(A) += N...
Evaluation Rev n/2-1 Rev n/2 Rev 1 Train warnings Rev n Mark buggy lines Rev n/2+1
Comparison <ul><li>History -based warning prioritization </li></ul><ul><li>Tools ’ warning prioritization  </li></ul><ul><...
Columba
Columba
Lucene
Lucene
Scarab
Scarab
Summary
Sung Kim and Michael D. Ernst {hunkim, mernst}@csail.mit.edu CSAIL  • MIT
Upcoming SlideShare
Loading in …5
×

Which Warnings Should I Fix First?

1,770 views

Published on

2007 European Software Engineering Conference and 2007 Foundations of Software Engineering (ESEC/FSE 2007)

  • Be the first to comment

  • Be the first to like this

Which Warnings Should I Fix First?

  1. 1. Sung Kim and Michael D. Ernst {hunkim, mernst}@csail.mit.edu CSAIL • MIT
  2. 2. Sung Kim and Michael D. Ernst {hunkim, mernst}@csail.mit.edu CSAIL • MIT
  3. 3. Static analysis I can find bugs for you!
  4. 4. Bug-finding tool
  5. 5. Warnings
  6. 6. Warning examples
  7. 7. Bug candidate - Warning Category : Style - empty try/catch Priority : 3
  8. 8. Are warnings useful? Are all warnings important to be fixed? Does tools’ prioritization help to identify important ones?
  9. 9. Experimental methodology <ul><li>Select source code (a snapshot) </li></ul><ul><li>Mark buggy lines </li></ul><ul><ul><li>Lines are changed in later fixes </li></ul></ul><ul><li>Compare with warnings from bug-finding tools </li></ul><ul><li>Measure precision </li></ul><ul><ul><li>High precision indicates important warnings </li></ul></ul>
  10. 10. Subject tools and projects <ul><li>Three bug-finding tools </li></ul><ul><ul><li>FindBugs, JLint, and PMD </li></ul></ul><ul><li>Three open source projects </li></ul>Projects Software type # of revisions Period LOC(K) Columba Email client 1,703 2002/11~2004/06 121 Lucene Search engine 929 2001/10~2006/11 37 Scarab Issue tracker 870 2002/01~2006/11 64
  11. 11. Buggy lines Rev 100 …… Mark as a buggy line if a line is changed in future fixes.
  12. 12. Change logs indicate fixes …… [bugfix] Fixed bug #3576 fix File 1 Non-fix Non-fix Non-fix Non-fix
  13. 13. Buggy lines Rev 199 Rev 200 Fix If (x=y && Z=x) {
  14. 14. Buggy lines Rev 199 Rev 198 If (x=y && Z=x) { If (x=y && Z=x) { marks Fix
  15. 15. Buggy lines Rev 100 Rev 200 Rev 199 Rev 198 ……
  16. 16. Warning evaluation Rev 100 50% of warnings are fixed later
  17. 17. Evaluate warnings at revision n/2 Rev n/2 Rev n Mark buggy lines Rev n/2+1
  18. 18. Are warnings important? NO only 3~18% are fixed!
  19. 19. Does tools’ prioritization help? NO Tools’ priority 1 warnings are not better.
  20. 20. Does tools’ prioritization help?
  21. 21. Does tools’ prioritization help?
  22. 22. Does tools’ prioritization help? No, high priority warnings do not have high precision!
  23. 23. …… Our solution: using change history Warning A File Warning B Foo ……
  24. 24. <ul><li>Compute score of each warning category using history </li></ul><ul><ul><li>Observe warning removals in history </li></ul></ul><ul><ul><li>If a warning is removed in a change </li></ul></ul><ul><ul><ul><li>Score promotes by non-fix-weight (0.1) </li></ul></ul></ul><ul><ul><li>If a warning is removed in a fix-change </li></ul></ul><ul><ul><ul><li>Score promotes by fix-weight (0.9) </li></ul></ul></ul><ul><ul><li>Assume high score warnings are important </li></ul></ul>Reprioritization algorithm
  25. 25. Reprioritization algorithm …… Score(A) += non-fix-weight (0.1) Score(A) += fix-weight (0.9) Fixed bug #3576 Score(A) += Non-fix-weight(0.1) Score(A)=1.1 fix Score(A)=0 Score(A)=0.1 Score(A)=1 Warning A File 1 Warning B Foo …… Score(B)=0
  26. 26. Evaluation Rev n/2-1 Rev n/2 Rev 1 Train warnings Rev n Mark buggy lines Rev n/2+1
  27. 27. Comparison <ul><li>History -based warning prioritization </li></ul><ul><li>Tools ’ warning prioritization </li></ul><ul><li>Compare precision of top 100 warnings </li></ul>
  28. 28. Columba
  29. 29. Columba
  30. 30. Lucene
  31. 31. Lucene
  32. 32. Scarab
  33. 33. Scarab
  34. 34. Summary
  35. 35. Sung Kim and Michael D. Ernst {hunkim, mernst}@csail.mit.edu CSAIL • MIT

×