This work empirically investigates the factors influence review participation in the MCR process. Through a case study of the Android, Qt, and OpenStack open source projects, we find that the amount of review participation in the past is a significant indicator of patches that will suffer from poor review participation. Moreover, the description length of a patch and the purpose of introducing new features also share a relationship with the likelihood of receiving poor review participation. This full article of this work is published in the Empirical Software Engineering journal. Available online at http://dx.doi.org/10.1007/s10664-016-9452-6

1. Review Participation
in Modern Code Review
— Journal First Paper —
Patanamon (Pick)
Thongtanunam
Shane
McIntosh
Ahmed E.
Hassan
Hajimu Iida
patanamon-t@is.naist.jp @pamonpatanamon.thongtanunam@adelaide.edu.au @pamon
1

2. Software code review is a well-established
software quality practice
Poor Software Quality
(e.g., Defects, Errors)
Poor Design Quality
(e.g., Spaghetti code)
2

3. In large and modern software organizations,
code review is performed through tools
Modern software development practices
Collaboration between
developers is asynchronous
Teams are globally
distributed
Modern Code Review (MCR):
A tool-based code review process
3

4. The value derived from MCR is dependent
on the participation of team members
Upload a
patch
A code review tool
(e.g. Gerrit)
An author
4

5. The value derived from MCR is dependent
on the participation of team members
Upload a
patch
A code review tool
(e.g. Gerrit)
Examine source code &
provide feedback
An author
4

6. The value derived from MCR is dependent
on the participation of team members
Address the
feedback
Upload a
patch
A code review tool
(e.g. Gerrit)
Examine source code &
provide feedback
An author
4

7. The value derived from MCR is dependent
on the participation of team members
Address the
feedback
Upload a
patch
Integrate a clean patch into
software repositories
A code review tool
(e.g. Gerrit)
Examine source code &
provide feedback
An author
4

8. A poor review participation has
an impact on software quality
Address the
feedback
Upload a
patch
Integrate a clean patch into
software repositories
A code review tool
(e.g. Gerrit)
Examine source code &
provide feedback
An author
5

9. A poor review participation has
an impact on software quality
Address the
feedback
Upload a
patch
Integrate a clean patch into
software repositories
A code review tool
(e.g. Gerrit)
Examine source code &
provide feedback
An author
Reviews of files that will be defective in the
future are less rigorous than clean files
[Thongtanunam et al., MSR2015]
5

10. Investigating the characteristics of patches
that suffer from poor review participation
No reviewer
participation
No review
discussion
Slow initial
feedback
6

11. Properties
e.g., Size, Description
length of code changes
History
e.g., Past tendency, #days
since the last modification
Environment
e.g., Review workload
Past Involvement
e.g., Past involvement of an
author and reviewers
Extracting 20 patch characteristics grouped
along 4 dimensions
7

12. Measure patch
characteristics
STEP 1
Training statistical models to explain
the relationship
History EnvironmentProperties Past Involvement
8

13. Measure patch
characteristics
STEP 1
Train a
prediction model
STEP 2
x
Assess the risk of receiving
poor review participation
Relax the linear assumption
of the relationship
Non-linear logistic regression model
Probability
x x
x
x
Training statistical models to explain
the relationship
9

14. Measure patch
characteristics
STEP 1
Train a
prediction model
STEP 2 STEP 3
Assess the fitness
of the model
Discrimination Ability
(AUC)
Stability
(AUC Optimism)
Model performance
Training statistical models to explain
the relationship
10

15. Measure patch
characteristics
STEP 1
Train a
prediction model
STEP 2 STEP 3
Assess the fitness
of the model
Interpret model
STEP 4
Explanatory power
(Wald χ2 statistics)
Plot the relationship with the risk of
receiving poor review collaboration
Observe patch characteristics
Increasing
Relationship
Inverse
Relationship
or
Training statistical models to explain
the relationship
11

16. Using the large-scale data of 196,712 patches spread
across the Android, Qt, and OpenStack projects
2008/10 - 2014/12
(6 Years)
2011/5 - 2014/12
(4 Years)
2011/7 - 2014/12
(4 Years)
51,721 patches 99,286 patches 136,343 patches
12

17. No reviewer
participation
No review
discussion
Slow initial feedback
0.72 0.70 0.74
AUC
Optimism
AUC
0.002 0.001 0.001
0.70 0.72 0.78
AUC
Optimism
AUC
0.002 0.001 0.001
0.66 0.61 0.61
AUC
Optimism
AUC
0.004 0.002 0.001
Our statistical models are accurate and
stable enough for interpretation
13

18. No reviewer
participation
No review
discussion
Slow initial feedback
History and properties metrics contribute a
large amount of explanatory power
History EnvironmentProperties Past Involvement
Description
Length
Past
tendency
Description
Length
Past
tendency
Patch
purpose
Past
tendency
1 3
2
2
1
2 1
Past involvement
of an author &
reviewers
3
#Prior patches
of an author
— —
—
—
14

19. No reviewer
participation
No review
discussion
Slow initial feedback
History and properties metrics contribute a
large amount of explanatory power
History EnvironmentProperties Past Involvement
Description
Length
Past
tendency
Description
Length
Past
tendency
Patch
purpose
Past
tendency
1 3
2
2
1
2 1
Past involvement
of an author &
reviewers
3
#Prior patches
of an author
— —
—
—
15

20. Oddsofreceivingpoor
reviewparticipationPast tendency and description length share an
inverse relationship with poor review participation
Oddsofreceivingpoor
reviewparticipation
History Properties
#Reviewers of prior patches
(Past tendency)
Description Length
16

21. Oddsofreceivingpoor
reviewparticipationPast tendency and description length share an
inverse relationship with poor review participation
Oddsofreceivingpoor
reviewparticipation
History Properties
#Reviewers of prior patches
(Past tendency)
Description Length
High chance that a new
patch will receive poor
review participation
Low chance that a new
patch will receive poor
review participation
17

22. Oddsofreceivingpoor
reviewparticipationPast tendency and description length share an
inverse relationship with poor review participation
Oddsofreceivingpoor
reviewparticipation
History Properties
#Reviewers of prior patches
(Past tendency)
Description Length
The relationship between
the observed characteristic
and the odds
18

23. Oddsofreceivingpoor
reviewparticipationPast tendency and description length share an
inverse relationship with poor review participation
Oddsofreceivingpoor
reviewparticipation
History Properties
#Reviewers of prior patches
(Past tendency)
Description Length
Less data support the
relationship in the broad
confidence interval
Confident
interval
19

24. Oddsofreceivingpoor
reviewparticipationPast tendency and description length share an
inverse relationship with poor review participation
Oddsofreceivingpoor
reviewparticipation
History Properties
#Reviewers of prior patches
(Past tendency)
Description Length
20

25. Oddsofreceivingpoor
reviewparticipationPast tendency and description length share an
inverse relationship with poor review participation
Oddsofreceivingpoor
reviewparticipation
History Properties
#Reviewers of prior patches
(Past tendency)
Description Length
20

26. Oddsofreceivingpoor
reviewparticipationPast tendency and description length share an
inverse relationship with poor review participation
Oddsofreceivingpoor
reviewparticipation
History Properties
#Reviewers of prior patches
(Past tendency)
Description Length
20

27. Investigating the characteristics of patches
that suffer from poor review participation
No reviewer
participation
No review
discussion
Slow initial
feedback
21

28. Investigating the characteristics of patches
that suffer from poor review participation
No reviewer
participation
No review
discussion
Slow initial
feedbackPatches whose prior patches received poor
review participation, and that are not
descriptive are not likely to receive reviewers
nor discussion
22

29. No reviewer
participation
No review
discussion
Slow initial feedback
History and properties metrics contribute a
large amount of explanatory power
History EnvironmentProperties Past Involvement
Description
Length
Past
tendency
Description
Length
Past
tendency
Patch
purpose
Past
tendency
1 3
2
2
1
2 1
Past involvement
of an author &
reviewers
3
#Prior patches
of an author
— —
—
—R
23

30. The purpose of introducing new features increase
the likelihood of receiving slow initial feedback
Oddsofreceivingpoor
reviewparticipation
Feature
introduction
B
ug-fix
Oddsofreceivingpoor
reviewparticipation
Feature
introduction
D
ocum
entation
16-29% 12-40%
24

31. Investigating the characteristics of patches
that suffer from poor review participation
No reviewer
participation
No review
discussion
Slow initial
feedbackPatches whose prior patches received poor
review participation, and that are not
descriptive are not likely to receive reviewers
nor discussion
25

32. Investigating the characteristics of patches
that suffer from poor review participation
No reviewer
participation
No review
discussion
Slow initial
feedback
Patches whose prior patches received poor
review participation, and that are not
descriptive are not likely to receive reviewers
nor discussion
Patches are likely to
receive slow initial
feedback if the purpose
is to introduces new
features
26

33. Wrap-up
27

34. A poor review participation has
an impact on software quality
Address the
feedback
Select reviewers
Upload a
patch
Integrate a clean patch into
software repositories
A code review tool
(e.g. Gerrit)
Examine source code &
provide feedback
An author
Reviews of files that will be defective in the
future are less rigorous than clean files
[Thongtanunam et al., MSR2015]
28

35. Properties
e.g., Size, Description
length of code changes
History
e.g., Past tendency, #days
since the last modification
Environment
e.g., Review workload
Past Involvement
e.g., Past involvement of an
author and reviewers
Extracting 20 patch characteristics grouped
along 4 dimensions
29

36. Investigating the characteristics of patches
that suffer from poor review participation
No reviewer
participation
No review
discussion
Slow initial
feedback
Patches whose prior patches received poor
review participation, and that are not
descriptive are not likely to receive reviewers
nor discussion
Patches are likely to
receive slow initial
feedback if the purpose
is to introduces new
features
30

37. Patanamon (Pick) Thongtanunam
Patches with such characteristics should be
given more attention to increase review
participation, which will likely lead to a more
responsive review process
patanamon.thongtanunam@adelaide.edu.au
http://patanamon.com
Review Participation in Modern Code Review
P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Review Participation in Modern Code
Review,” Empirical Software Engineering (EMSE)., vol. 22, no. 2, pp. 768–817, 2017.
31

38. A poor review participation has
an impact on software quality
Address the
feedback
Select reviewers
Upload a
patch
Integrate a clean patch into
software repositories
A code review tool
(e.g. Gerrit)
Examine source code &
provide feedback
An author
Reviews of files that will be defective in the
future are less rigorous than clean files
[Thongtanunam et al., MSR2015]
Properties
e.g., Size, Description
length of code changes
History
e.g., Past tendency, #days
since the last modification
Environment
e.g., Review workload
Past Involvement
e.g., Past involvement of an
author and reviewers
Extracting 20 patch characteristics grouped
along 4 dimensions
Investigating the characteristics of patches
that suffer from poor review participation
No reviewer
participation
No review
discussion
Slow initial
feedback
Patches whose prior patches received poor
review participation, and that are not
descriptive are not likely to receive reviewers
nor discussion
Patches are likely to
receive slow initial
feedback if the purpose
is to introduces new
features
Patanamon (Pick) Thongtanunam
Patches with such characteristics should be
given more attention to increase review
participation, which will likely lead to a more
responsive review process
patanamon.thongtanunam@adelaide.edu.au
http://patanamon.com
Review Participation in Modern Code Review
P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Review Participation in Modern Code
Review,” Empirical Software Engineering (EMSE)., vol. 22, no. 2, pp. 768–817, 2017.
32