Presented by Daniel da Costa, Shane McIntosh, Weiyi (Ian) Shang, Uira Kulesza, Roberta Coelho, and Ahmed E. Hassan

1. A Framework for Evaluating
the SZZ Apporach for
Identifying Bug Introducing
Changes
Daniel da Costa Shane McIntosh Weiyi (Ian)
Shang
Uirá Kulesza Roberta Coelho Ahmed E.
Hassan
Queen’s
University
McGill
University
Concordia
University
Federal
University
Federal
University
Queen’s
University

2. After a bug is spotted, one has to find its
root cause
2

3. After a bug is spotted, one has to find its
root cause
2

4. After a bug is spotted, one has to find its
root cause
2
Bug

5. After a bug is spotted, one has to find its
root cause
Bug
The effort to find
the root causes
2

6. The SZZ approach was proposed to find
which changes induce a future fix
3

7. The SZZ approach was proposed to find
which changes induce a future fix
3

8. The SZZ approach was proposed to find
which changes induce a future fix
These induced our later
fix
3

9. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
4

10. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
4

11. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
4

12. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
# mandatory
comment
+ x + y
– x – y
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
Change #3
Fixing a bug
4

13. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
# mandatory
comment
+ x + y
– x – y
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
Change #3
Fixing a bug
4

14. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
# mandatory
comment
+ x + y
– x – y
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
Change #3
Fixing a bug
4

15. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
# mandatory
comment
+ x + y
– x – y
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
Change #3
Fixing a bug
4

16. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
# mandatory
comment
+ x + y
– x – y
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
Change #3
Fixing a bug
4

17. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
# mandatory
comment
+ x + y
– x – y
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
Change #3
Fixing a bug
4

18. SZZ is the foundational piece behind
several empirical findings in recent
studies
5

19. SZZ is the foundational piece behind
several empirical findings in recent
studies
What is the buggiest
day?
(Sliwerski et al., 2005)
5
´

20. SZZ is the foundational piece behind
several empirical findings in recent
studies
What is the buggiest
day?
(Sliwerski et al., 2005)
What is the buggiest
time of day?
(Eyolfson et al., 2011)
5
´

21. SZZ is the foundational piece behind
several empirical findings in recent
studies
Buggy
Not buggy
(Kim et al., 2008)
What is the buggiest
day?
(Sliwerski et al., 2005)
What is the buggiest
time of day?
(Eyolfson et al., 2011)
5
´

22. However… SZZ is not without its
limitations
Are we really finding changes that
induce fixes?
6

23. To make matters harder, no ground truth
is readily available
7

24. To make matters harder, no ground truth
is readily available
7
Can you tell me the
changes that induced
the fix of bug
#50505?

25. To make matters harder, no ground truth
is readily available
7
Can you tell me the
changes that induced
the fix of bug
#50505?

26. To make matters harder, no ground truth
is readily available
7
:-(

27. In spite of this, SZZ-generated results
must be evaluated
8

28. We propose metrics to highlight
suspicious values in SZZ-generated data
9

29. 10
Fix-inducing changes that disagree with
team members are suspicious

30. 10
Fix-inducing changes that disagree with
team members are suspicious

31. 10
Fix-inducing changes that disagree with
team members are suspicious
The day of
this release

32. Time
11
Fix-inducing changes that disagree with
team members are suspicious

33. Time
5.12.3
11
Fix-inducing changes that disagree with
team members are suspicious

34. Time
5.12.3
SZZ
11
Fix-inducing changes that disagree with
team members are suspicious

35. Time
5.12.3
SZZ
Buggy
Change
11
Fix-inducing changes that disagree with
team members are suspicious
Suspicious
!

36. Time
5.12.3
SZZ
Buggy
Change
Buggy
Change
Suspicious
!
OK!
11
Fix-inducing changes that disagree with
team members are suspicious

37. Time
12
Changes that are implicated in several
future fixes are suspicious

38. Time
Buggy
Change
12
Changes that are implicated in several
future fixes are suspicious

39. Time
Buggy
Change
Bug #1
12
Changes that are implicated in several
future fixes are suspicious

40. Time
Buggy
Change
Bug #1 Bug #2
12
Changes that are implicated in several
future fixes are suspicious

41. Time
Buggy
Change
Bug #1 Bug #2 Bug #3
12
Changes that are implicated in several
future fixes are suspicious

42. Time
Buggy
Change
Bug #1 Bug #2 Bug #3…
12
Changes that are implicated in several
future fixes are suspicious
Bug #400

43. Fixes that are induced by changes that
are spaced by years are suspicious
13

44. Time
13
Fixes that are induced by changes that
are spaced by years are suspicious

45. Time
Bug #1
13
Fixes that are induced by changes that
are spaced by years are suspicious

46. Time
Bug #1Buggy
Change
13
Fixes that are induced by changes that
are spaced by years are suspicious

47. Time
Bug #1Buggy
Change
Buggy
Change
13
Fixes that are induced by changes that
are spaced by years are suspicious

48. Time
Bug #1Buggy
Change
Buggy
Change
13
Fixes that are induced by changes that
are spaced by years are suspicious
Buggy
Change

49. Time
Bug #1Buggy
Change
Buggy
Change
Buggy
Change
13
Fixes that are induced by changes that
are spaced by years are suspicious
2 years

50. We evaluate four SZZ variations in our
empirical study
14

51. We evaluate four SZZ variations in our
empirical study
1. if (age > 18) {
2. can_watch = true;
3. }
4. return price;
Review #1
14

52. We evaluate four SZZ variations in our
empirical study
1. if (age > 18) {
2. //check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
1. if (age > 18) {
2. can_watch = true;
3. }
4. return price;
Review #2Review #1
14

53. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. if (age > 18) {
2. //check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
1. if (age > 18) {
2. can_watch = true;
3. }
4. return price;
Review #2Review #1 Review #3
14

54. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
14

55. B-SZZ

56. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
14

57. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

58. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

59. 1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
14

60. 1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
14

61. 1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
14

62. MA-SZZ

63. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

64. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

65. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

66. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

67. R-SZZ

68. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

69. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

70. L-SZZ

71. We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price; 14

72. 1. if (age > 18) {
2. + //check loyalty card
3. + if(loyaltycard){
4. + //give 30% discount
5. + price = price * 0.8;}
6. }
7. can_watch = true;
8. }
9. return price;
We evaluate four SZZ variations in our
empirical study
1. if (age >= 18) {
2. - // check loyalty card
3. if(loyaltycard){
4. //give 30% discount
5. price = price * 0.7;}
6. }
7. can_watch = true;
8. }
9. return price;
1. + if (age > 18) {
2. + can_watch = true;
3. + }
4. + return price;
Review #2Review #1 Review #3
14

73. RESULTS

74. SZZ disagrees with team members
regarding when fixes are induced
15

75. SZZ disagrees with team members
regarding when fixes are induced
0
10
20
30
40
50
B-SZZ MA-SZZ R-SZZ L-SSZ
Disagreement Ratio (%)
15

76. 0
10
20
30
40
50
B-SZZ MA-SZZ R-SZZ L-SSZ
Disagreement Ratio (%)
SZZ disagrees with team members
regarding when fixes are induced
15

77. Changes may induce future fixes that
span several days
16

78. Changes may induce future fixes that
span several days
1,000
10
Days
B-SZZ MA-SZZ R-SZZ L-SZZ 16

79. 1,000
10
Days
B-SZZ MA-SZZ R-SZZ L-SZZ 16
Changes may induce future fixes that
span several days

80. Many fixes take years to be induced
17

81. Many fixes take years to be induced
B-SZZ MA-SZZ
1,000
10
Days
17

82. Many fixes take years to be induced
B-SZZ MA-SZZ
1,000
10
Days
17

83. To what extent should
we trust in SZZ data
in the end of the day?
SZZ
We
18

84. We believe that the extreme values of our
metrics may indicate false positives
19

85. We believe that the extreme values of our
metrics may indicate false positives
B-SZZ MA-SZZ
1,000
10
Days
19

86. We believe that the extreme values of our
metrics may indicate false positives
B-SZZ MA-SZZ
1,000
10
Days
19

87. We believe that the extreme values of our
metrics may indicate false positives
B-SZZ MA-SZZ
1,000
10
Days
19
We
manually
analyze 60
bugs for
each
evaluated
SZZ

88. 38 out of 60 fix-inducing changes were
false positives (MA-SZZ)
20

89. 38 out of 60 fix-inducing changes were
false positives (MA-SZZ)
20
Missed Initial
Code
Imp.
Directory
Renamin
g
Equivale
nt
Backout Low
Likelihoo
d
True
Positive
Disagreement Ratio 5 0 3 1 0 2 9
# of Future Fixes 0 5 4 5 0 0 6
Fix Inducing
Changes Time-span
0 5 3 0 1 4 7
Total 5 10 10 6 1 6 22

90. 38 out of 60 fix-inducing changes were
false positives (MA-SZZ)
20
Missed Initial
Code
Imp.
Directory
Renamin
g
Equivale
nt
Backout Low
Likelihoo
d
True
Positive
Disagreement Ratio 5 0 3 1 0 2 9
# of Future Fixes 0 5 4 5 0 0 6
Fix Inducing
Changes Time-span
0 5 3 0 1 4 7
Total 5 10 10 6 1 6 22

91. Changes that do not change the software
behaviour
22

92. Changes that do not change the software
behaviour
public interface MyAPI {
getMoney(int
howMuch);
}
22

93. Changes that do not change the software
behaviour
public interface MyAPI {
getMoney(int
howMuch);
}
public interface MyAPI {
– getMoney(int howMuch);
+ public getMoney(int
howMuch);
}
22

94. FINAL REMARKS 1

95. SZZ still has opportunities for
improvements
23

96. Despite the lack of a “ground truth”, our
framework helps to evaluate the SZZ-
generated data at hand
24

97. SUMMARY 1

98. SZZ traces historical data from bug-fixing
changes back to fix-inducing changes
function sum(x, y)
begin
# mandatory
comment
+ x + y
– x – y
function sum(x, y)
begin
+ # mandatory
comment
x – y
+ function sum(x, y)
+ begin
+ x – y
+ end
Change #1
Initial
Import
Change #2
We need
documentation
Change #3
Fixing a bug
4

100. Time
5.12.3
SZZ
Buggy
Change
Buggy
Change
Suspicious
!
OK!
11
Fix-inducing changes that disagree with
team members are suspicious

102. 38 out of 60 fix-inducing changes were
false positives
20
Missed Initial
Code
Imp.
Directory
Renamin
g
Equivale
nt
Backout Low
Likelihoo
d
True
Positive
Disagreement Ratio 5 0 3 1 0 2 9
# of Future Fixes 0 5 4 5 0 0 6
Fix Inducing
Changes Time-span
0 5 3 0 1 4 7
Total 5 10 10 6 1 6 22

104. Changes that do not change the software
behaviour
public interface MyAPI {
getMoney(int
howMuch);
}
public interface MyAPI {
– getMoney(int howMuch);
+ public getMoney(int
howMuch);
}
22