The document summarizes research investigating differences in code review practices between defective and clean files. It finds that reviews of files that will eventually become defective (future-defective files) tend to be less rigorous, with less intense discussion, less participation, and faster completion times, compared to reviews of clean files. In contrast, reviews of files that were previously defective (risky files) do not see increased rigor in terms of intensity, participation or duration compared to normal files.

1. Investigating Code Review Practices
in Defective Files
Patanamon (Pick)
Thongtanunam
Shane McIntosh
Ahmed E.
Hassan
Hajimu Iida
May 16-17, 2015.
Firenze, Italy
patanamon-t@is.naist.jp @pamon

2. Modern Code Review: A lightweight, tool-supported
code review process
Code Review Tool

3. Modern Code Review: A lightweight, tool-supported
code review process
Code Review Tool
Code change

4. Modern Code Review: A lightweight, tool-supported
code review process
Code Review Tool
Code change

5. Modern Code Review: A lightweight, tool-supported
code review process
Code Review Tool
Code change
Examine Code

6. Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code Review Tool
Code change
Examine Code

7. Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code change
A lack of code review
activity can increase the
risk of post-release defects
[McIntosh et. al., MSR2014]
My code is awesome!
No needs for a review
Code Review Tool
Code change
Examine Code

8. Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code change
A lack of code review
activity can increase the
risk of post-release defects
[McIntosh et. al., MSR2014]
My code is awesome!
No needs for a review
Code Review Tool
Code change
How should reviewers do a code
review to reduce the risk of
having defects?
Examine Code

9. What is the difference between code review
practices of defective and clean files?

10. What is the difference between code review
practices of defective and clean files?
Review Practice A
Defective
i.e., files that have defects

11. Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects

12. Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects

13. We measure 3 dimensions of review activity metrics
Review Intensity
e.g., #Review Iterations,
Discussion Length
Review Participation
e.g., #Reviewers,
Review Agreement
Reviewing Time
e.g., Review Length,
Code Reading Speed

14. Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects

15. Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects

16. Defective
i.e., files that have defects
We investigate defective files along 2 perspectives

17. Defective
i.e., files that have defects
We investigate defective files along 2 perspectives

18. Risky Files
Files that have historically been defective
Past Defective
i.e., files that have defects
We investigate defective files along 2 perspectives

19. Risky Files
Files that have historically been defective
Past FutureDefective
i.e., files that have defects
Future-Defective Files
Files that will eventually have defects
We investigate defective files along 2 perspectives

20. Risky Files
Files that have historically been defective
Past FutureDefective
i.e., files that have defectsFuture-Defective Files
Files that will eventually have defects
Conjecture: Reviews of Future-
Defective will be
• less intense,
• with less team participation,
• completed with a shorter time
than reviews of clean files
We investigate defective files along 2 perspectives

21. Future-Defective Files: Files that have post-release defects
VCS
Repositories

22. Future-Defective Files: Files that have post-release defects
VCS
Repositories
Release date Bug-fixing commit

23. Future-Defective Files: Files that have post-release defects
VCS
Repositories
Release date Bug-fixing commit
Future-Defective

24. Future-Defective Files: Files that have post-release defects
VCS
Repositories
Release date
No bug-fixing commits
Release date
Bug-fixing commit
Future-Defective

25. Future-Defective Files: Files that have post-release defects
VCS
Repositories
Release date
No bug-fixing commits
Release date
Bug-fixing commit
Future-Defective
Clean

26. Future-Defective Files: Files that have post-release defects
Studied ReviewsVCS
Repositories
Release date
No bug-fixing commits
Release date
Bug-fixing commit
6 months
Future-Defective
Clean

27. Future-Defective Files: Files that have post-release defects
Studied ReviewsVCS
Repositories
Release date
No bug-fixing commits
Release date
Bug-fixing commit
6 months
1,176 Files
3,470 Reviews
10,513 Files
2,727 Reviews
5.0.0
866 Files
2,849 Reviews
11,931 Files
2,690 Reviews
5.1.0
Future-Defective
Clean

28. #Reviewers
#Reviews of Clean files
#Reviewers
#Reviews of Future-Defective files
VS
Review Activity Analysis: Compare code review activity
that has been applied to future-defective and clean files

29. #Reviewers
#Reviews of Clean files
#Reviewers
#Reviews of Future-Defective files
VS
Using a statistical test to determine the difference between
the distributions of code review activity
Review Activity Analysis: Compare code review activity
that has been applied to future-defective and clean files

30. #Reviewers
#Reviews of Clean files
#Reviewers
#Reviews of Future-Defective files
VS
Using a statistical test to determine the difference between
the distributions of code review activity
Raw code review activity metric is
normalized by patch size
Review Activity Analysis: Compare code review activity
that has been applied to future-defective and clean files

31. Findings
Code review activity in
the reviews of future-defective files
Conjecture
Results
Review Intensity
Review Participation
Reviewing Time
Less Intense
Less Team
Participation
Completed with
a shorter time

32. Findings
Code review activity in
the reviews of future-defective files
Conjecture
Less Intense
Results
Review Intensity
Review Participation
Reviewing Time
Less Intense
Less Team
Participation
Completed with
a shorter time

33. Findings
Code review activity in
the reviews of future-defective files
Conjecture
Less Intense
Less Team
Participation
Results
Review Intensity
Review Participation
Reviewing Time
Less Intense
Less Team
Participation
Completed with
a shorter time

34. Findings
Code review activity in
the reviews of future-defective files
Conjecture
Less Intense
Less Team
Participation
Faster Code
Reading Rate
Results
Review Intensity
Review Participation
Reviewing Time
Less Intense
Less Team
Participation
Completed with
a shorter time

35. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Risky Files
Files that have historically been defective
FutureFuture-Defective Files
Files that will eventually have defects
Conjecture: Reviews of Future-
Defective will be
• less intense,
• with less team participation,
• completed with a shorter time
than reviews of clean files

36. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Risky Files
Files that have historically been defective
FutureFuture-Defective Files
Files that will eventually have defects
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files

37. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Risky Files
Files that have historically been defective
FutureFuture-Defective Files
Files that will eventually have defects
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files

38. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Risky Files
Files that have historically been defective
Conjecture: Reviews of risky files
should be
• more intense,
• with more team participation,
• reviewed for a longer time
to reduce the risk of having defects
in the future
FutureFuture-Defective Files
Files that will eventually have defects
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files

39. Risky Files: Files that had post-release defects
in prior release
VCS
Repositories

40. Risky Files: Files that had post-release defects
in prior release
VCS
Repositories
Release date
Prior release date
Bug-fixing
commit

41. Risky Files: Files that had post-release defects
in prior release
VCS
Repositories
Release date
Prior release date
Bug-fixing
commit
Risky

42. Risky Files: Files that had post-release defects
in prior release
VCS
Repositories
No bug-fixing commits
Release date
Prior release
date
Release date
Prior release date
Bug-fixing
commit
Risky

43. Risky Files: Files that had post-release defects
in prior release
VCS
Repositories
No bug-fixing commits
Release date
Prior release
date
Release date
Prior release date
Bug-fixing
commit
Normal
Risky

44. Risky Files: Files that had post-release defects
in prior release
Studied ReviewsVCS
Repositories
No bug-fixing commits
Release date
Prior release
date
Release date
Prior release date
Bug-fixing
commit
6 months
Normal
Risky

45. Risky Files: Files that had post-release defects
in prior release
Studied ReviewsVCS
Repositories
No bug-fixing commits
Release date
Prior release
date
Release date
Prior release date
Bug-fixing
commit
6 months
1,168 Files
2,671 Reviews
11,629 Files
2,868 Reviews
5.1.0
Normal
Risky

46. Findings
Code review activity in
the reviews of risky files
Conjecture
Results
Review Intensity
Review Participation
Reviewing Time
More Intense
More Team
Participation
Completed with
a longer time

47. Findings
Code review activity in
the reviews of risky files
Conjecture
Less Intense
Results
Review Intensity
Review Participation
Reviewing Time
More Intense
More Team
Participation
Completed with
a longer time

48. Findings
Code review activity in
the reviews of risky files
Conjecture
Less Intense
Less Team
Participation
Results
Review Intensity
Review Participation
Reviewing Time
More Intense
More Team
Participation
Completed with
a longer time

49. Findings
Code review activity in
the reviews of risky files
Conjecture
Less Intense
Less Team
Participation
Receive Slow
Feedback &
Faster Code
Reading Rate
Results
Review Intensity
Review Participation
Reviewing Time
More Intense
More Team
Participation
Completed with
a longer time

50. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
FutureFuture-Defective Files
Files that will eventually have defects
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Risky Files
Files that have historically been defective
Conjecture: Reviews of risky files
should be
• more intense,
• with more team participation,
• reviewed for a longer time
to reduce the risk of having defects
in the future

51. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
FutureFuture-Defective Files
Files that will eventually have defects
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Risky Files
Files that have historically been defective
Developers are not as careful
when they review risky files.

52. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Future
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Developers are not as careful
when they review risky files.
Future-Defective Files
Files that will eventually have defects
Risky Files
Files that have historically been defective

53. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Future
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Developers are not as careful
when they review risky files.
Future-Defective Files
Files that will eventually have defects
Risky Files
Files that have historically been defective
Will careless reviews of risky files lead to future defects?

54. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Future
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Developers are not as careful
when they review risky files.
Future-Defective Files
Files that will eventually have defects
Risky Files
Files that have historically been defective
Will careless reviews of risky files lead to future defects?
Investigating code review
practice in risky & future-
defective files

55. Risky & Future-Defective Files: Risky files that will
eventually have defects
VCS
Repositories

56. Risky & Future-Defective Files: Risky files that will
eventually have defects
VCS
Repositories
Bug-fixing commit
Release date
Prior release date
Bug-fixing
commit

57. Risky & Future-Defective Files: Risky files that will
eventually have defects
VCS
Repositories
Bug-fixing commit
Release date
Prior release date
Bug-fixing
commit
Risky & Future-Defective

58. Risky & Future-Defective Files: Risky files that will
eventually have defects
VCS
Repositories
Bug-fixing commit
Release date
Prior release date
Bug-fixing
commit
No bug-fixing
commits
Release datePrior release date
Bug-fixing commit
Risky & Future-Defective

59. Risky & Future-Defective Files: Risky files that will
eventually have defects
VCS
Repositories
Bug-fixing commit
Release date
Prior release date
Bug-fixing
commit
No bug-fixing
commits
Release datePrior release date
Bug-fixing commit
Risky & Clean
Risky & Future-Defective

60. Risky & Future-Defective Files: Risky files that will
eventually have defects
Studied Reviews
VCS
Repositories
Bug-fixing commit
Release date
Prior release date
Bug-fixing
commit
No bug-fixing
commits
Release datePrior release date
Bug-fixing commit
6 months
Risky & Clean
Risky & Future-Defective

61. Risky & Future-Defective Files: Risky files that will
eventually have defects
Studied Reviews
VCS
Repositories
Bug-fixing commit
Release date
Prior release date
Bug-fixing
commit
No bug-fixing
commits
Release datePrior release date
Bug-fixing commit
6 months
206 Files
1,299 Reviews
962 Files
1,372 Reviews
5.1.0
Risky & Clean
Risky & Future-Defective

62. Findings
Code review activity in the reviews of
risky & future-defective files
Conjecture
Less Intense
Less Team
Participation
Receive Slow
Feedback &
Faster Code
Reading Rate
Results
Review Intensity
Review Participation
Reviewing Time
Less Intense
Less Team
Participation
Completed with
a shorter time

63. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Future
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Developers are not as careful
when they review risky files.
Future-Defective Files
Files that will eventually have defects
Risky Files
Files that have historically been defective
Will careless reviews of risky files lead to future defects?
Investigating code review
practice in risky & future-
defective files

64. Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Future
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Developers are not as careful
when they review risky files.
Future-Defective Files
Files that will eventually have defects
Risky Files
Files that have historically been defective
Reviews of files that are both risky & future defective are less
rigorous than files that are risky but clean

65. Evolvability
e.g., Fixing code comments,
Decomposing complex function
Functionality
e.g., Fixing incorrect
program logic
Traceability
e.g., Updating commit
message
We compare concerns that are addressed during
reviews of defective and clean files

66. Evolvability
Functionality Traceability
Proportion of reviews in future-defective files in Qt5.0.0
82%
40%40%
Reviews of defective files often address
evolvability concernsResults
10% higher than
clean files
5% higher than
clean files
10% lower than
clean files
We observe the similar results for the reviews of risky files and
risky & future-defective files

68. Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code change
A lack of code review
activity can increase the
risk of post-release defects
[McIntosh et. al., MSR2014]
My code is awesome!
No needs for a review
Code Review Tool
Code change
How should reviewers do a code
review to reduce the risk of
having defects?
Examine Code

69. Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code change
A lack of code review
activity can increase the
risk of post-release defects
[McIntosh et. al., MSR2014]
My code is awesome!
No needs for a review
Code Review Tool
Code change
How should reviewers do a code
review to reduce the risk of
having defects?
Examine Code
Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects

70. Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code change
A lack of code review
activity can increase the
risk of post-release defects
[McIntosh et. al., MSR2014]
My code is awesome!
No needs for a review
Code Review Tool
Code change
How should reviewers do a code
review to reduce the risk of
having defects?
Examine Code
Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects
We measure 3 dimensions of review activity metrics
Review Intensity
e.g., #Review Iterations,
Discussion Length
Review Participation
e.g., #Reviewers,
Review Agreement
Reviewing Time
e.g., Review Length,
Code Reading Speed

71. Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code change
A lack of code review
activity can increase the
risk of post-release defects
[McIntosh et. al., MSR2014]
My code is awesome!
No needs for a review
Code Review Tool
Code change
How should reviewers do a code
review to reduce the risk of
having defects?
Examine Code
Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects
We measure 3 dimensions of review activity metrics
Review Intensity
e.g., #Review Iterations,
Discussion Length
Review Participation
e.g., #Reviewers,
Review Agreement
Reviewing Time
e.g., Review Length,
Code Reading Speed
Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Future
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Developers are not as careful
when they review risky files.
Future-Defective Files
Files that will eventually have defects
Risky Files
Files that have historically been defective
Reviews of files that are both risky & future defective are less
rigorous than files that are risky but clean

72. Defect-free (Clean)
i.e., files that do not have defects
Review Practice B
What is the difference between code review
practices of defective and clean files?
Review Practice A
VS
Defective
i.e., files that have defects
We measure 3 dimensions of review activity metrics
Review Intensity
e.g., #Review Iterations,
Discussion Length
Review Participation
e.g., #Reviewers,
Review Agreement
Reviewing Time
e.g., Review Length,
Code Reading Speed
Defective
i.e., files that have defects
Past
We investigate defective files along 2 perspectives
Future
Reviews of future-defective files
tend to be less rigorous than
reviews of clean files
Developers are not as careful
when they review risky files.
Future-Defective Files
Files that will eventually have defects
Risky Files
Files that have historically been defective
Reviews of files that are both risky & future defective are less
rigorous than files that are risky but cleanpatanamon-t@is.naist.jp @pamon
Investigating Code Review Practices in Defective Files
Modern Code Review: A lightweight, tool-supported
code review process
Upstream VCS
repositories
Code change
A lack of code review
activity can increase the
risk of post-release defects
[McIntosh et. al., MSR2014]
My code is awesome!
No needs for a review
Code Review Tool
Code change
How should reviewers do a code
review to reduce the risk of
having defects?
Examine Code