1.
Evaluating SZZ Implementations
Through a Developer-informed Oracle
Giovanni Rosa, Luca Pascarella, Simone Scalabrino, Rosalia Tufano,
Gabriele Bavota, Michele Lanza, Rocco Oliveto

2.
Where do bugs come from?

3.
Find out changes that can lead to a problem
and avoid them in future
Understanding where bugs are introduced allows to…

4.
Estimate how much a program is error-prone
Understanding where bugs are introduced allows to…

5.
Better allocate resources in testing activities
Understanding where bugs are introduced allows to…

6.
Śliwerski
Zimmermann
Zeller
@ MSR 2005

7.
Step 1
SZZ in a nutshell
bug report
analysis

8.
Step 1
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
SZZ in a nutshell
bug report
analysis

9.
Step 1
Step 2
Filtering of resulting
commits
SZZ in a nutshell
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis

10.
Step 1
bug-inducing
commit
Step 2 Step 3
SZZ in a nutshell
Filtering of resulting
commits
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis

11.
Different SZZ variants proposed

12.
There is a problem

13.
Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016

14.
Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016
Small datasets used for evaluation

15.
Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016
Small datasets used for evaluation
Validation manually performed by
researchers

16.
Define a dataset validated by
the developers
The way

17.
fixes a search bug
introduced by 2508e12
and fixes a typo in the
README.md

18.
Developer-informed
dataset

19.
Mining of commits
2011 2020

20.
Heuristic approach
1
keyword-based filter
AI-powered syntax analysis
Duplicate commits removal

21.
Heuristic approach
2
keyword-based filter
AI-powered syntax analysis
Duplicate commits removal

22.
3 Heuristic approach
keyword-based filter
AI-powered syntax analysis
duplicate commits removal

23.
Manual validation
False
positives
Bug report
data

24.
Bug report data
fixes #1740 quote pov-ray binary on windows
this fixes a bug introduced by #3523741…
URL
Date when the
issue is reported
https://tracker.freecadweb.org/view.php?id=1740
Commit
message

25.
19,6M
3,6k
1,9k
Analyzed commits:
Extracted commits:
After manual validation:

26.
Top programming languages
0
185
370
C
P
y
t
h
o
n
C
+
+
J
S
J
a
v
a
P
H
P
R
u
b
y
C
#

27.
1,1k
129
Final number of commits:
Commits with issue report:

28.
How do different variants of SZZ
perform in identifying
bug-inducing changes?

29.
B-SZZ
Śliwerski et al. @ MSR 2005

30.
R-SZZ e L-SZZ
B-SZZ
AG-SZZ
DJ-SZZ
Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008
Kim et al. @ ASE 2006 Davies et al. @ JSE 2013

31.
R-SZZ e L-SZZ
B-SZZ
AG-SZZ
MA-SZZ
DJ-SZZ
RA-SZZ
Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008 Da Costa et al. @ TSE 2016
Kim et al. @ ASE 2006 Davies et al. @ JSE 2013 Neto et al. @ SANER 2018

32.
Open-Source implementations
SZZ Unleashed
(DJ-SZZ)
OpenSZZ
(B-SZZ)
PyDriller
(AG-SZZ)
RA-SZZ
(RA-SZZ)

33.
Step 1
bug-inducing
commit
Step 2 Step 3
Our experiment
Filtering of resulting
commits
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis

34.
Results
0.66 (R-SZZ)
Precision
Recall
F1-score
0.72 (SZZ@UNL)
0.61 (R-SZZ)

35.
Results
0.66 (R-SZZ)
Precision
Recall
F1-score
0.72 (SZZ@UNL)
0.61 (R-SZZ)
0.09 (SZZ@UNL)
0.19 (SZZ@OPN)
Java only
0.16 (SZZ@UNL)

36.
Qualitative Analysis

37.
What have we learned?

38.
“ The buggy line is
not always impacted
in the bug-fix „
Lesson 1

39.
“ SZZ is sensible to
history rewritings „
Lesson 2

40.
“ Looking at the
big picture in
code changes „
Lesson 3

41.
Summary

42.
Take a look at our SZZ implementation!
https://github.com/grosa1/pyszz