Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)

Giovanni Rosa
Giovanni RosaPhD Student in Software Engineering at University of Molise
Evaluating SZZ Implementations
Through a Developer-informed Oracle
Giovanni Rosa, Luca Pascarella, Simone Scalabrino, Rosalia Tufano,
Gabriele Bavota, Michele Lanza, Rocco Oliveto
Where do bugs come from?
Find out changes that can lead to a problem
and avoid them in future
Understanding where bugs are introduced allows to…
Estimate how much a program is error-prone
Understanding where bugs are introduced allows to…
Better allocate resources in testing activities
Understanding where bugs are introduced allows to…
Śliwerski
Zimmermann
Zeller
@ MSR 2005
Step 1
SZZ in a nutshell
bug report
analysis
Step 1
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
SZZ in a nutshell
bug report
analysis
Step 1
Step 2
Filtering of resulting
commits
SZZ in a nutshell
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis
Step 1
bug-inducing
commit
Step 2 Step 3
SZZ in a nutshell
Filtering of resulting
commits
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis
Different SZZ variants proposed
There is a problem
Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016
Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016
Small datasets used for evaluation
Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016
Small datasets used for evaluation
Validation manually performed by
researchers
Define a dataset validated by
the developers
The way
fixes a search bug
introduced by 2508e12
and fixes a typo in the
README.md
Developer-informed
dataset
Mining of commits
2011 2020
Heuristic approach
1
keyword-based filter
AI-powered syntax analysis
Duplicate commits removal
Heuristic approach
2
keyword-based filter
AI-powered syntax analysis
Duplicate commits removal
3 Heuristic approach
keyword-based filter
AI-powered syntax analysis
duplicate commits removal
Manual validation
False
positives
Bug report
data
Bug report data
fixes #1740 quote pov-ray binary on windows
this fixes a bug introduced by #3523741…
URL
Date when the
issue is reported
https://tracker.freecadweb.org/view.php?id=1740
Commit
message
19,6M
3,6k
1,9k
Analyzed commits:
Extracted commits:
After manual validation:
Top programming languages
0
185
370
C
P
y
t
h
o
n
C
+
+
J
S
J
a
v
a
P
H
P
R
u
b
y
C
#
1,1k
129
Final number of commits:
Commits with issue report:
How do different variants of SZZ
perform in identifying
bug-inducing changes?
B-SZZ
Śliwerski et al. @ MSR 2005
R-SZZ e L-SZZ
B-SZZ
AG-SZZ
DJ-SZZ
Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008
Kim et al. @ ASE 2006 Davies et al. @ JSE 2013
R-SZZ e L-SZZ
B-SZZ
AG-SZZ
MA-SZZ
DJ-SZZ
RA-SZZ
Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008 Da Costa et al. @ TSE 2016
Kim et al. @ ASE 2006 Davies et al. @ JSE 2013 Neto et al. @ SANER 2018
Open-Source implementations
SZZ Unleashed
(DJ-SZZ)
OpenSZZ
(B-SZZ)
PyDriller
(AG-SZZ)
RA-SZZ
(RA-SZZ)
Step 1
bug-inducing
commit
Step 2 Step 3
Our experiment
Filtering of resulting
commits
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis
Results
0.66 (R-SZZ)
Precision
Recall
F1-score
0.72 (SZZ@UNL)
0.61 (R-SZZ)
Results
0.66 (R-SZZ)
Precision
Recall
F1-score
0.72 (SZZ@UNL)
0.61 (R-SZZ)
0.09 (SZZ@UNL)
0.19 (SZZ@OPN)
Java only
0.16 (SZZ@UNL)
Qualitative Analysis
What have we learned?
“ The buggy line is
not always impacted
in the bug-fix „
Lesson 1
“ SZZ is sensible to
history rewritings „
Lesson 2
“ Looking at the
big picture in
code changes „
Lesson 3
Summary
Take a look at our SZZ implementation!
https://github.com/grosa1/pyszz
1 of 42

More Related Content

What's hot(20)

Investigating Code Review Practices in Defective FilesInvestigating Code Review Practices in Defective Files
Investigating Code Review Practices in Defective Files
The University of Adelaide1.7K views
Msr17a.pptMsr17a.ppt
Msr17a.ppt
Yann-Gaël Guéhéneuc22 views
Saner16b.pptSaner16b.ppt
Saner16b.ppt
Yann-Gaël Guéhéneuc18 views
Icsm20.pptIcsm20.ppt
Icsm20.ppt
Yann-Gaël Guéhéneuc59 views
Opinion Mining for Software EngineeringOpinion Mining for Software Engineering
Opinion Mining for Software Engineering
Alexander Serebrenik211 views
03. HAMS - Project Scheduling03. HAMS - Project Scheduling
03. HAMS - Project Scheduling
HAMSproject217 views
Plagiarism introductionPlagiarism introduction
Plagiarism introduction
Merin Paul1.1K views
QuaP2P Kickoff Slides 2006QuaP2P Kickoff Slides 2006
QuaP2P Kickoff Slides 2006
Kalman Graffi987 views
Big(ger) Data in Software EngineeringBig(ger) Data in Software Engineering
Big(ger) Data in Software Engineering
Mehdi Mirakhorli1.4K views

Similar to Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)(20)

Mining Software Defects: Should We Consider Affected Releases?Mining Software Defects: Should We Consider Affected Releases?
Mining Software Defects: Should We Consider Affected Releases?
Chakkrit (Kla) Tantithamthavorn311 views
A Tale of Experiments on Bug PredictionA Tale of Experiments on Bug Prediction
A Tale of Experiments on Bug Prediction
Martin Pinzger757 views
A tale of experiments on bug predictionA tale of experiments on bug prediction
A tale of experiments on bug prediction
Martin Pinzger819 views
poster_3.0poster_3.0
poster_3.0
Stefano Sansone219 views
Debug meDebug me
Debug me
Noha Elprince965 views
BH-US-06-Bilar.pdfBH-US-06-Bilar.pdf
BH-US-06-Bilar.pdf
MohammadRazavi172 views

Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)