The document discusses the evaluation of various SZZ implementations for identifying bug-inducing changes in software development. It emphasizes the importance of understanding where bugs are introduced to improve resource allocation in testing and presents a developer-informed dataset for thorough analysis. The results indicate differing performances of SZZ variants and provide key lessons regarding the sensitivity of SZZ to history rewritings and the need for a comprehensive view of code changes.

1. Evaluating SZZ Implementations
Through a Developer-informed Oracle
Giovanni Rosa, Luca Pascarella, Simone Scalabrino, Rosalia Tufano,
Gabriele Bavota, Michele Lanza, Rocco Oliveto

2. Where do bugs come from?

3. Find out changes that can lead to a problem
and avoid them in future
Understanding where bugs are introduced allows to…

4. Estimate how much a program is error-prone
Understanding where bugs are introduced allows to…

5. Better allocate resources in testing activities
Understanding where bugs are introduced allows to…

6. Śliwerski
Zimmermann
Zeller
@ MSR 2005

7. Step 1
SZZ in a nutshell
bug report
analysis

8. Step 1
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
SZZ in a nutshell
bug report
analysis

9. Step 1
Step 2
Filtering of resulting
commits
SZZ in a nutshell
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis

10. Step 1
bug-inducing
commit
Step 2 Step 3
SZZ in a nutshell
Filtering of resulting
commits
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis

11. Different SZZ variants proposed

12. There is a problem

13. Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016

14. Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016
Small datasets used for evaluation

15. Evaluating and
comparing the SZZ
variants
Da Costa et al. @ TSE 2016
Small datasets used for evaluation
Validation manually performed by
researchers

16. Define a dataset validated by
the developers
The way

17. fixes a search bug
introduced by 2508e12
and fixes a typo in the
README.md

18. Developer-informed
dataset

19. Mining of commits
2011 2020

20. Heuristic approach
1
keyword-based filter
AI-powered syntax analysis
Duplicate commits removal

21. Heuristic approach
2
keyword-based filter
AI-powered syntax analysis
Duplicate commits removal

22. 3 Heuristic approach
keyword-based filter
AI-powered syntax analysis
duplicate commits removal

23. Manual validation
False
positives
Bug report
data

24. Bug report data
fixes #1740 quote pov-ray binary on windows
this fixes a bug introduced by #3523741…
URL
Date when the
issue is reported
https://tracker.freecadweb.org/view.php?id=1740
Commit
message

25. 19,6M
3,6k
1,9k
Analyzed commits:
Extracted commits:
After manual validation:

26. Top programming languages
0
185
370
C
P
y
t
h
o
n
C
+
+
J
S
J
a
v
a
P
H
P
R
u
b
y
C
#

27. 1,1k
129
Final number of commits:
Commits with issue report:

28. How do different variants of SZZ
perform in identifying
bug-inducing changes?

29. B-SZZ
Śliwerski et al. @ MSR 2005

30. R-SZZ e L-SZZ
B-SZZ
AG-SZZ
DJ-SZZ
Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008
Kim et al. @ ASE 2006 Davies et al. @ JSE 2013

31. R-SZZ e L-SZZ
B-SZZ
AG-SZZ
MA-SZZ
DJ-SZZ
RA-SZZ
Śliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008 Da Costa et al. @ TSE 2016
Kim et al. @ ASE 2006 Davies et al. @ JSE 2013 Neto et al. @ SANER 2018

32. Open-Source implementations
SZZ Unleashed
(DJ-SZZ)
OpenSZZ
(B-SZZ)
PyDriller
(AG-SZZ)
RA-SZZ
(RA-SZZ)

33. Step 1
bug-inducing
commit
Step 2 Step 3
Our experiment
Filtering of resulting
commits
(A)
Bug-fixing
commit
(B)
git blame
(C)
Buggy
commit
bug report
analysis

34. Results
0.66 (R-SZZ)
Precision
Recall
F1-score
0.72 (SZZ@UNL)
0.61 (R-SZZ)

35. Results
0.66 (R-SZZ)
Precision
Recall
F1-score
0.72 (SZZ@UNL)
0.61 (R-SZZ)
0.09 (SZZ@UNL)
0.19 (SZZ@OPN)
Java only
0.16 (SZZ@UNL)

36. Qualitative Analysis

37. What have we learned?

38. “ The buggy line is
not always impacted
in the bug-fix „
Lesson 1

39. “ SZZ is sensible to
history rewritings „
Lesson 2

40. “ Looking at the
big picture in
code changes „
Lesson 3

41. Summary

42. Take a look at our SZZ implementation!
https://github.com/grosa1/pyszz