1. Modern Code Reviews in Open-Source Projects:
Which Problems Do They Fix?
Moritz Beller
Delft University of Technology
@Inventitech
Andy Zaidman, Georgios Gousios, Alberto Bacchelli,
Elmar Jürgens, Radjino Bohlanath, Shane McIntosh

2. Modern Code Review Workflow

3. Modern Code Review Workflow
Old

4. Modern Code Review Workflow

5. Modern Code Review Workflow

6. Modern Code Review Workflow

7. Modern Code Review Workflow

8. Modern Code Review Workflow

9. Modern Code Review Workflow

10. Modern Code Review Workflow

11. Modern Code Review Workflow

12. Modern Code Review Workflow

13. Modern Code Review Workflow

14. Modern Code Review Workflow
+

16. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();

17. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

18. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

19. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
public Runnable watchDogUiThread = new
StartupUIThread();

20. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
public Runnable watchDogUiThread = new
StartupUIThread();

21. Review-triggered Change
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
public Runnable watchDogUiThread = new
StartupUIThread();

22. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

23. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

24. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
private Runnable watchDogUiThread = new
StartupUIThread();

25. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
private Runnable watchDogUiThread = new
StartupUIThread();

26. Self-motivated + Review-triggered Change
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
private Runnable watchDogUiThread = new
StartupUIThread();

27. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

28. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

29. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/**
* The ui thread for WatchDog registration. From
* this thread, the WatchDog program is started.
*/
public Runnable watchDog = new StartupUIThread();

30. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/**
* The ui thread for WatchDog registration. From
* this thread, the WatchDog program is started.
*/
public Runnable watchDog = new StartupUIThread();

31. Self-motivated Change
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/**
* The ui thread for WatchDog registration. From
* this thread, the WatchDog program is started.
*/
public Runnable watchDog = new StartupUIThread();

32. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

33. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();

34. /** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();

35. No Change
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();

36. No Change
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();

37. No Change
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
// TODO (GG) Provide a more descriptive name
public Runnable watchDog = new StartupUIThread();
/** The ui thread for WatchDog registration. */
public Runnable watchDog = new StartupUIThread();

38. Does it matter?

39. >1,400 changes
>245 tasks

40. Motivation for Change

41. Motivation for Change

42. 78-90%
Motivation for Change

43. 78-90%
Motivation for Change
22-10%

44. 78-90%
Motivation for Change
22-10%

45. Motivation for No Change

46. Motivation for No Change

47. Motivation for No Change
7-35%

48. Motivation for No Change
7-35%

49. Type of Change

50. Doc
Visual
Structure
Resource
Check
Interface
Logic
Larger
Support
ConQAT
Gromacs
10% 20% 30% 40%

51. 75
Doc
Visual
Structure
Resource
Check
Interface
Logic
Larger
Support
ConQAT
Gromacs
10% 20% 30% 40%

52. 75
:
Doc
Visual
Structure
Resource
Check
Interface
Logic
Larger
Support
ConQAT
Gromacs
10% 20% 30% 40%

53. 75
:
25
Doc
Visual
Structure
Resource
Check
Interface
Logic
Larger
Support
ConQAT
Gromacs
10% 20% 30% 40%

55. 75 : 25

56. What Influences Code Review?

57. What Influences Code Review?

58. What Influences Code Review?

59. What Influences Code Review?
> 2,800 changes
> 970 tasks

60. What Influences Code Review?
> 2,800 changes
> 970 tasks

61. What Influences Code Review?
> 2,800 changes
> 970 tasks

62. What Influences Code Review?
> 2,800 changes
> 970 tasks

63. What Influences Code Review?
> 2,800 changes
> 970 tasks

64. What Influences Code Review?
> 2,800 changes
> 970 tasks

65. What Influences Code Review?
> 2,800 changes
> 970 tasks

66. What Influences Code Review?
> 2,800 changes
> 970 tasks

67. “Continuous” Integration Then (2014)
+

68. “Continuous” Integration Then (2014)
+

69. “Continuous” Integration Then (2014)
+

70. “Continuous” Integration Then (2014)
+

71. Continuous Integration Now (2017)

72. Continuous Integration Now (2017)
CI Services

73. Continuous Integration Now (2017)
a) Static Analysis
CI Services

74. Continuous Integration Now (2017)
a) Static Analysis
b) Dynamic Analysis
CI Services

75. Continuous Integration Now (2017)
+
a) Static Analysis
b) Dynamic Analysis
CI Services

76. a) How Prevalent Is Static Analysis?

77. 122
a) How Prevalent Is Static Analysis?

78. 122
a) How Prevalent Is Static Analysis?

79. 122
a) How Prevalent Is Static Analysis?

80. 122
122
a) How Prevalent Is Static Analysis?

81. 122
122
a) How Prevalent Is Static Analysis?

82. 122
36
122
a) How Prevalent Is Static Analysis?

83. 122
36
122
a) How Prevalent Is Static Analysis?

84. 122
36
122
a) How Prevalent Is Static Analysis?

85. a) How Prevalent Is Static Analysis?

86. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
a) How Prevalent Is Static Analysis?

87. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
a) How Prevalent Is Static Analysis?

88. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
a) How Prevalent Is Static Analysis?

89. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
a) How Prevalent Is Static Analysis?

90. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
a) How Prevalent Is Static Analysis?

91. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
Moritz Beller, Radjino Bholanath, Shane McIntosh, Andy Zaidman:
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source
Software, SANER, Osaka (Japan), 2016
a) How Prevalent Is Static Analysis?

92. TODO: Add background with
Sun
b) How Prevalent Is Dynamic Analysis?

93. TODO: Add background with
Sun
CI is the best practice in which
developers not only integrate their
work into a shared mainline
frequently, but also verify the quality
continuously through testing.
b) How Prevalent Is Dynamic Analysis?

94. TODO: Add background with
Sun
CI is the best practice in which
developers not only integrate their
work into a shared mainline
frequently, but also verify the quality
continuously through testing.
It is known.
b) How Prevalent Is Dynamic Analysis?

96. <TL;DR>...</TL;DR>

97. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

98. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

99. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

100. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

101. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

102. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

103. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

104. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

105. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant
●
Parsing Log files for Ruby: bundler and rake

106. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant
●
Parsing Log files for Ruby: bundler and rake

107. b) How Prevalent Is Dynamic Analysis?

108. ●
69% of Java and 87.5% of the Ruby projects:
test runs
b) How Prevalent Is Dynamic Analysis?

109. ●
69% of Java and 87.5% of the Ruby projects:
test runs
●
Overall, 81% of the projects:
test runs
b) How Prevalent Is Dynamic Analysis?

110. ●
69% of Java and 87.5% of the Ruby projects:
test runs
●
Overall, 81% of the projects:
test runs
●
96% of builds:
test runs
b) How Prevalent Is Dynamic Analysis?

111. b) How Prevalent Is Dynamic Analysis?

112. b) How Prevalent Is Dynamic Analysis?

113. b) How Prevalent Is Dynamic Analysis?

114. Testing is the single most important
reason for broken builds, more
prevalent than compile errors, missing
dependencies, build cancellations and
provisioning problems together.
b) Testing Is Central To CI

115. Testing is the single most important
reason for broken builds, more
prevalent than compile errors, missing
dependencies, build cancellations and
provisioning problems together.
b) Testing Is Central To CI
Moritz Beller, Georgios Gousios, Andy Zaidman:
Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub,
MSR, Buenos Aires (Argentina), 2017

116. What Is TravisTorrent?

117. What Is TravisTorrent?
MSR Mining
Challenge 2017
Public Data Set

118. What Is TravisTorrent?
Free* SQL-like cloud query
interface on Google BQ
MSR Mining
Challenge 2017
Public Data Set

119. What Is TravisTorrent?
.testroots.org
Free* SQL-like cloud query
interface on Google BQ
MSR Mining
Challenge 2017
Public Data Set

120. TODO: Add background with
Sun
Research on Continuous Code Review
Enabled By TravisTorrent

121. TODO: Add background with
Sun
- git_branch
- git_all_built_commits
- ...
Research on Continuous Code Review
Enabled By TravisTorrent

122. TODO: Add background with
Sun
- git_branch
- git_all_built_commits
- ...
- gh_project_name
- gh_is_pr
- gh_team_size
- gh_diff_tests_added
- gh_first_commit_...
Research on Continuous Code Review
Enabled By TravisTorrent

123. TODO: Add background with
Sun
- git_branch
- git_all_built_commits
- ...
- gh_project_name
- gh_is_pr
- gh_team_size
- gh_diff_tests_added
- gh_first_commit_...
- tr_prev_build
- tr_status
- tr_log_tests_ran
- tr_log_tests_failed
- ...
Research on Continuous Code Review
Enabled By TravisTorrent

124. TODO: Add background with
Sun
- git_branch
- git_all_built_commits
- ...
- gh_project_name
- gh_is_pr
- gh_team_size
- gh_diff_tests_added
- gh_first_commit_...
- tr_prev_build
- tr_status
- tr_log_tests_ran
- tr_log_tests_failed
- ...
62columns
Research on Continuous Code Review
Enabled By TravisTorrent

125. TODO: Add background with
Sun
- git_branch
- git_all_built_commits
- ...
- gh_project_name
- gh_is_pr
- gh_team_size
- gh_diff_tests_added
- gh_first_commit_...
- tr_prev_build
- tr_status
- tr_log_tests_ran
- tr_log_tests_failed
- ...
62columns
Research on Continuous Code Review
Enabled By TravisTorrent

126. TODO: Add background with
Sun
- git_branch
- git_all_built_commits
- ...
- gh_project_name
- gh_is_pr
- gh_team_size
- gh_diff_tests_added
- gh_first_commit_...
- tr_prev_build
- tr_status
- tr_log_tests_ran
- tr_log_tests_failed
- ...
62columns
Research on Continuous Code Review
Enabled By TravisTorrent

127. @Inventitech
Moritz Beller, TU Delft
Modern Code Reviews in Open-Source Projects:
Which Problems Do They Fix?

128. @Inventitech
Moritz Beller, TU Delft
Modern Code Reviews in Open-Source Projects:
Which Problems Do They Fix?

129. @Inventitech
Moritz Beller, TU Delft
Modern Code Reviews in Open-Source Projects:
Which Problems Do They Fix?

130. @Inventitech
Moritz Beller, TU Delft
Modern Code Reviews in Open-Source Projects:
Which Problems Do They Fix?