Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)
Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)
The SZZ algorithm for identifying bug-inducing changes has been widely used to evaluate defect prediction techniques and to empirically investigate when, how, and by whom bugs are introduced. Over the years, researchers have proposed several heuristics to improve the SZZ accuracy, providing various implementations of SZZ. However, fairly evaluating those implementations on a reliable oracle is an open problem: SZZ evaluations usually rely on (i) the manual analysis of the SZZ output to classify the identified bug-inducing commits as true or false positives; or (ii) a golden set linking bug-fixing and bug-inducing commits. In both cases, these manual evaluations are performed by researchers with limited knowledge of the studied subject systems. Ideally, there should be a golden set created by the original developers of the studied systems. We propose a methodology to build a "developer-informed" oracle for the evaluation of SZZ variants. We use Natural Language Processing (NLP) to identify bug-fixing commits in which developers explicitly reference the commit(s) that introduced a fixed bug. This was followed by a manual filtering step aimed at ensuring the quality and accuracy of the oracle. Once built, we used the oracle to evaluate several variants of the SZZ algorithm in terms of their accuracy. Our evaluation helped us to distill a set of lessons learned to further improve the SZZ algorithm. Giovanni Rosa (University of Molise), Luca Pascarella (Università della Svizzera italiana), Simone Scalabrino (University of Molise), Rosalia Tufano (Università della Svizzera Italiana), Gabriele Bavota (Software Institute, USI Università della Svizzera italiana), Michele Lanza (Software Institute, USI Università della Svizzera italiana), Rocco Oliveto (University of Molise), * Live Presentation: https://www.youtube.com/watch?v=ZiuAaysj_Sk * IEEE Digital Library: https://www.computer.org/csdl/proceedings-article/icse/2021/029600a436/1sEXo2zV4OI
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)
Similar to Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)(20)
Recently uploaded
Recently uploaded(20)
Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)
- 1. Evaluating SZZ Implementations Through a Developer-informed Oracle Giovanni Rosa, Luca Pascarella, Simone Scalabrino, Rosalia Tufano, Gabriele Bavota, Michele Lanza, Rocco Oliveto
- 2. Where do bugs come from?
- 3. Find out changes that can lead to a problem and avoid them in future Understanding where bugs are introduced allows to…
- 4. Estimate how much a program is error-prone Understanding where bugs are introduced allows to…
- 5. Better allocate resources in testing activities Understanding where bugs are introduced allows to…
- 7. Step 1 SZZ in a nutshell bug report analysis
- 8. Step 1 (A) Bug-fixing commit (B) git blame (C) Buggy commit SZZ in a nutshell bug report analysis
- 9. Step 1 Step 2 Filtering of resulting commits SZZ in a nutshell (A) Bug-fixing commit (B) git blame (C) Buggy commit bug report analysis
- 10. Step 1 bug-inducing commit Step 2 Step 3 SZZ in a nutshell Filtering of resulting commits (A) Bug-fixing commit (B) git blame (C) Buggy commit bug report analysis
- 11. Different SZZ variants proposed
- 12. There is a problem
- 13. Evaluating and comparing the SZZ variants Da Costa et al. @ TSE 2016
- 14. Evaluating and comparing the SZZ variants Da Costa et al. @ TSE 2016 Small datasets used for evaluation
- 15. Evaluating and comparing the SZZ variants Da Costa et al. @ TSE 2016 Small datasets used for evaluation Validation manually performed by researchers
- 16. Define a dataset validated by the developers The way
- 17. fixes a search bug introduced by 2508e12 and fixes a typo in the README.md
- 19. Mining of commits 2011 2020
- 20. Heuristic approach 1 keyword-based filter AI-powered syntax analysis Duplicate commits removal
- 21. Heuristic approach 2 keyword-based filter AI-powered syntax analysis Duplicate commits removal
- 22. 3 Heuristic approach keyword-based filter AI-powered syntax analysis duplicate commits removal
- 24. Bug report data fixes #1740 quote pov-ray binary on windows this fixes a bug introduced by #3523741… URL Date when the issue is reported https://tracker.freecadweb.org/view.php?id=1740 Commit message
- 25. 19,6M 3,6k 1,9k Analyzed commits: Extracted commits: After manual validation:
- 27. 1,1k 129 Final number of commits: Commits with issue report:
- 28. How do different variants of SZZ perform in identifying bug-inducing changes?
- 29. B-SZZ Śliwerski et al. @ MSR 2005
- 30. R-SZZ e L-SZZ B-SZZ AG-SZZ DJ-SZZ Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008 Kim et al. @ ASE 2006 Davies et al. @ JSE 2013
- 31. R-SZZ e L-SZZ B-SZZ AG-SZZ MA-SZZ DJ-SZZ RA-SZZ Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008 Da Costa et al. @ TSE 2016 Kim et al. @ ASE 2006 Davies et al. @ JSE 2013 Neto et al. @ SANER 2018
- 33. Step 1 bug-inducing commit Step 2 Step 3 Our experiment Filtering of resulting commits (A) Bug-fixing commit (B) git blame (C) Buggy commit bug report analysis
- 35. Results 0.66 (R-SZZ) Precision Recall F1-score 0.72 (SZZ@UNL) 0.61 (R-SZZ) 0.09 (SZZ@UNL) 0.19 (SZZ@OPN) Java only 0.16 (SZZ@UNL)
- 37. What have we learned?
- 38. “ The buggy line is not always impacted in the bug-fix „ Lesson 1
- 39. “ SZZ is sensible to history rewritings „ Lesson 2
- 40. “ Looking at the big picture in code changes „ Lesson 3
- 41. Summary
- 42. Take a look at our SZZ implementation! https://github.com/grosa1/pyszz