Invited talk at the Leaders of Tomorrow Symposium of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2016). The presentation (and its accompanying paper, see http://mcis.polymtl.ca/publications/2016/fose.pdf) explain the basics of release engineering pipelines, common challenges industry is facing as well as pitfalls software engineering researchers are falling into. Speakers are Bram Adams (MCIS, http://mcis.polymtl.ca) and Shane McIntosh (McGill University, http://shanemcintosh.org). A video-taped version of the talk will be available soon at https://www.youtube.com/channel/UCL8yG6qpHk7V66l1Jt3aZrA/featured.

1. Modern Release
Engineering in a Nutshell
Why Researchers should Care
Bram Adams Shane McIntosh
,
M
C IS

2. Back in 2009 …

4. On average, we
release new code fifty
times a day.

5. Release engineering aims to …
4

6. Release engineering aims to …
4
code
change

7. Release engineering aims to …
4
code
change

8. Release engineering aims to …
4
code
change

9. Release engineering aims to …
4
code
change
FAST

10. Release
Engineering

11. Release
Engineering

12. Release
Engineering
integrating code changes

13. Release
Engineering
integrating code changes building/testing (CI)

14. Release
Engineering
integrating code changes building/testing (CI)
deploying a new release

15. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release

16. Nowadays …

17. Time-boxed
releases
7

18. Time-boxed
releases
6 months
7

19. Time-boxed
releases
6 weeks
6 months
7

20. Time-boxed
releases
2 weeks
(mobile)
6 weeks
6 months
7

21. Time-boxed
releases
twice/day
(web)
2 weeks
(mobile)
6 weeks
6 months
7

22. Time-boxed
releases
twice/day
(web)
daily
2 weeks
(mobile)
6 weeks
6 months
7

23. http://www.informit.com/articles/article.aspx?p=1833567
Jez Humble

24. http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk
of releasing software
if it hurts, do it
more frequently, and
bring the pain
forward
Jez Humble

25. • How quickly can we ship a chemspill release?
• 4-6 weeks 11 hours
• How long to ship a “new feature” release?
• 12-18 months 6 weeks
• How many active code lines?
• 1 1/2 42
• How many checkins per day?
• ~15 per day 325 per day
Before & After
http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9

26. • How quickly can we ship a chemspill release?
• 4-6 weeks 11 hours
• How long to ship a “new feature” release?
• 12-18 months 6 weeks
• How many active code lines?
• 1 1/2 42
• How many checkins per day?
• ~15 per day 325 per day
Before & After
http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9

27. • How quickly can we ship a chemspill release?
• 4-6 weeks 11 hours
• How long to ship a “new feature” release?
• 12-18 months 6 weeks
• How many active code lines?
• 1 1/2 42
• How many checkins per day?
• ~15 per day 325 per day
Before & After
http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9

28. https://xebialabs.com/periodic-table-of-devops-tools/ & https://xebialabs.com/the-ultimate-devops-tool-chest/

29. 11
industry
Release
engineering
innovation
led by
industry!

30. 11
industry
a
c
a
d
e
m
i
a
Release
engineering
innovation
led by
industry!

31. 12
What can you
do for release
engineering?

32. RELENG: International Workshop on
Release Engineering
13

33. RELENG: International Workshop on
Release Engineering
13
230 participants3 editions
dozens of industry
& academic talks

34. RELENG: International Workshop on
Release Engineering
13
230 participants3 editions
dozens of industry
& academic talks

35. RELENG: International Workshop on
Release Engineering
13
230 participants3 editions
next RELENG:
in Fall 2016
http://releng.polymtl.ca
dozens of industry
& academic talks

36. How to predict integration effort?
14

37. How to predict integration effort?
14
git
repo

38. How to predict integration effort?
14
git
repo
myapp
v.1

39. How to predict integration effort?
14
git
repo
myapp
v.1
?

40. How to predict integration effort?
14
git
repo
myapp
v.1

41. How to predict integration effort?
14
feature branch 1
git
repo
myapp
v.1

42. How to predict integration effort?
14
feature branch 1
git
repo
myapp
v.1

43. How to predict integration effort?
14
feature branch 1
git
repo
myapp
v.1

44. master branch
How to predict integration effort?
14
feature branch 1
git
repo
myapp
v.1

45. master branch
How to predict integration effort?
14
feature branch 1
git
repo
MERGE
myapp
v.1

46. master branch
How to predict integration effort?
14
feature branch 1
git
repo
MERGE
myapp
v.1

47. master branch
How to predict integration effort?
14
feature branch 1
git
repo
MERGE
myapp
v.1

48. master branch
How to predict integration effort?
14
feature branch 1
git
repo
release branch
MERGE
myapp
v.1

49. master branch
How to predict integration effort?
14
feature branch 1
git
repo
release branch
MERGE
myapp
v.1

50. master branch
How to predict integration effort?
14
feature branch 1
git
repo
release branch
MERGE
myapp
v.1
feature branch 2

51. master branch
How to predict integration effort?
14
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2

52. master branch
How to predict integration effort?
14
3rd party
dependency
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2

53. master branch
How to predict integration effort?
14
3rd party
dependency
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2

54. master branch
How to predict integration effort?
14
3rd party
dependency
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2 sync

55. 15
https://xkcd.com/303/

56. user acceptance/system tests
performance tests
(selection of) unit tests
all unit tests
integration tests
time to run (hours)
full compilation
incremental compilation
`
Why do CI builds take so long?
16

57. user acceptance/system tests
performance tests
(selection of) unit tests
all unit tests
integration tests
time to run (hours)
full compilation
incremental compilation
`
local developer build
Why do CI builds take so long?
16

58. user acceptance/system tests
performance tests
(selection of) unit tests
all unit tests
integration tests
time to run (hours)
full compilation
incremental compilation
`
local developer build
Why do CI builds take so long?
16

59. user acceptance/system tests
performance tests
(selection of) unit tests
all unit tests
integration tests
time to run (hours)
full compilation
incremental compilation
` CI build of merged change
local developer build
Why do CI builds take so long?
16

60. user acceptance/system tests
performance tests
(selection of) unit tests
all unit tests
integration tests
time to run (hours)
full compilation
incremental compilation
CI build of merged change
local developer build
Why do CI builds take so long?
16

61. user acceptance/system tests
performance tests
(selection of) unit tests
all unit tests
integration tests
time to run (hours)
full compilation
incremental compilation
closer to release
CI build of merged change
local developer build
Why do CI builds take so long?
16

62. user acceptance/system tests
performance tests
(selection of) unit tests
all unit tests
integration tests
time to run (hours)
full compilation
incremental compilation
closer to release
CI build of merged change
local developer build
Why do CI builds take so long?
16

63. Even worse …
17

64. Even worse …
17
build machinery run
on each commit!

65. Even worse …
17
build machinery run
on each commit!

66. Even worse …
17
build machinery run
on each commit!
different feature
configurations

67. Even worse …
17
build machinery run
on each commit!
not just build and
test, also code
quality builds,
nightly builds, etc.
different feature
configurations

68. Even worse …
17
build machinery run
on each commit!
not just build and
test, also code
quality builds,
nightly builds, etc.
not all builds
succeed
different feature
configurations

69. What are deployment best practices?
18

70. What are deployment best practices?
18
v.1 v.1v.1

71. What are deployment best practices?
18
v.1v.2 v.1

72. What are deployment best practices?
18
v.2 v.1v.2

73. What are deployment best practices?
18
v.2 v.1v.1

74. What are deployment best practices?
18
v.1v.1v.1

75. Infrastructure code
19

76. #
Install
PostgreSQL
server
and
client
include_recipe
"postgresql::server"
include_recipe
"postgresql::client"
#
Make
postgresql_database
resource
available
include_recipe
"database::postgresql"
#
Create
database
for
Rails
app
db
=
node["practicingruby"]["database"]
postgresql_database
db["name"]
do
connection(
:host
=>
db["host"],
:port
=>
node["postgresql"]["config"]["port"],
:username
=>
db["username"],
:password
=>
db["password"],
)
end
Infrastructure code
19

77. #
Install
PostgreSQL
server
and
client
include_recipe
"postgresql::server"
include_recipe
"postgresql::client"
#
Make
postgresql_database
resource
available
include_recipe
"database::postgresql"
#
Create
database
for
Rails
app
db
=
node["practicingruby"]["database"]
postgresql_database
db["name"]
do
connection(
:host
=>
db["host"],
:port
=>
node["postgresql"]["config"]["port"],
:username
=>
db["username"],
:password
=>
db["password"],
)
end
Infrastructure code
19

78. #
Install
PostgreSQL
server
and
client
include_recipe
"postgresql::server"
include_recipe
"postgresql::client"
#
Make
postgresql_database
resource
available
include_recipe
"database::postgresql"
#
Create
database
for
Rails
app
db
=
node["practicingruby"]["database"]
postgresql_database
db["name"]
do
connection(
:host
=>
db["host"],
:port
=>
node["postgresql"]["config"]["port"],
:username
=>
db["username"],
:password
=>
db["password"],
)
end
Infrastructure code smells?
19

80. Is the
release in good
shape?

81. Was the release a
success?
Is the
release in good
shape?

82. Was the release a
success?
Is the
release in good
shape?
The release is
botched, how can we
roll back?

83. Was the release a
success?
What’s the optimal cycle
time for us?
Is the
release in good
shape?
The release is
botched, how can we
roll back?

84. Chuck
Rossi
21

85. Continuous delivery
for mobile apps is a serious
challenge!
Chuck
Rossi
21

86. What can you
do for release
engineering?
22

87. What can you
do for release
engineering?
22
What can
release
engineering
do for you?

88. When noble intentions meet reality
23

89. Harmful assumptions about release pipelines
that can impact predictive modelling
24

90. Harmful assumptions about release pipelines
that can impact predictive modelling
1. All releases are equal
24

91. 25

92. Look at Project X!
It had a crazy number of
bugs in the typical six-month
post-release window!
25

93. Look at Project X!
It had a crazy number of
bugs in the typical six-month
post-release window!
Well, Project X releases
every 6 weeks, so
you’re counting several
releases…
25

94. Release cycles vary among
popular studied systems
Daysbetweenreleases
An Empirical Study of Delays in the Integration of
Addressed Issues
D. A. da Costa et al.
[ICSME 2014]

95. Release cycles can even
vary within systems!
1.0
2.0
2.5
3.0
3.5
4.0
5.0
6.0
7.0
8.0
9.0
10.0
0 200 400 600 800
Firefoxrelease
Days since prior release 27

96. The rapid release cycle of modern
software systems
28

97. The rapid release cycle of modern
software systems
Often release several times
in one day!
28

98. 1. All releases are equal
Harmful assumptions about release pipelines
that can impact predictive modelling
29

99. 1. All releases are equal
2. All branches are equal
Harmful assumptions about release pipelines
that can impact predictive modelling
29

100. 30

101. Weird… The size of this
project fluctuates between
50k and 45k lines!
30

102. Weird… The size of this
project fluctuates between
50k and 45k lines!
Hmm, did you select the
relevant branch? Several
are developed in parallel!
30

103. An example of defect prediction
Feature development
Defect repairing
Merge
Commit types
31

104. An example of defect prediction
Feature development
Defect repairing
Merge
Commit types
31

105. An example of defect prediction
v1.0
Feature development
Defect repairing
Merge
Commit types
31

106. An example of defect prediction
v1.0
Feature development
Defect repairing
Merge
Commit types
Pre-
release
31

107. An example of defect prediction
v1.0
Feature development
Defect repairing
Merge
Commit types
Pre-
release
Post-
release
31

108. An example of defect prediction
v1.0
Feature development
Defect repairing
Merge
Commit types
Pre-
release
Post-
release
31

109. Handling the intricacies of a multi-
branch release pipeline
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

110. Handling the intricacies of a multi-
branch release pipeline
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

111. Handling the intricacies of a multi-
branch release pipeline
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

112. Handling the intricacies of a multi-
branch release pipeline
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

113. Handling the intricacies of a multi-
branch release pipeline
v1.0
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

114. Handling the intricacies of a multi-
branch release pipeline
v1.0
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

115. Handling the intricacies of a multi-
branch release pipeline
v1.0
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

116. Handling the intricacies of a multi-
branch release pipeline
v1.0
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

117. Handling the intricacies of a multi-
branch release pipeline
v1.0
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

118. Handling the intricacies of a multi-
branch release pipeline
v1.0
Dev
Stable
Feature development
Defect repairing
Merge
Commit types
32

119. Branching can get really complex…
33

120. 1. All releases are equal
2. All branches are equal
Harmful assumptions about release pipelines
that can impact predictive modelling
34

121. 1. All releases are equal
2. All branches are equal
3. All files are equal
Harmful assumptions about release pipelines
that can impact predictive modelling
34

122. 35

123. This popular web browser
has an enormous codebase!
>30M lines!
35

124. This popular web browser
has an enormous codebase!
>30M lines!
Yes, but the codebase
contains several systems!
The build configuration
decides which one is
produced!
35

125. 36
Many files are conditionally
included in deliverables
Tracing Software Build
Processes to Uncover
License Compliance
Inconsistencies
S. van der Berg et al.
[ASE 2014]
Aterm
Opkg
Bash
CUPS
Xalan
OpenSSL
FFmpeg
% Excluded Files
0% 20% 40% 60% 80%

126. 36
Many files are conditionally
included in deliverables
Fixes in these files may have a
smaller impact (if any) on customers
Tracing Software Build
Processes to Uncover
License Compliance
Inconsistencies
S. van der Berg et al.
[ASE 2014]
Aterm
Opkg
Bash
CUPS
Xalan
OpenSSL
FFmpeg
% Excluded Files
0% 20% 40% 60% 80%

127. Understanding conditionally
included files using the build system
Design recovery and maintenance
of build systems
B. Adams et al.
[ICSM 2007]
37

128. 1. All releases are equal,
2. All branches are equal,
Harmful assumptions about release pipelines
that can impact predictive modelling
3. All files are equal
38

129. but some are more equal than others
1. All releases are equal,
2. All branches are equal,
Harmful assumptions about release pipelines
that can impact predictive modelling
3. All files are equal
38

130. My nightmare
Amassing and indexing a large
sample of version control systems
Audris Mockus
[MSR 2009]
Boa: a language and infrastructure for analyzing
ultra-large-scale software repositories
R. Dyer et al.
[ICSE 2013]
The GHTorrent Dataset and Tool
Suite
G. Gousios
[MSR 2013]

131. My nightmare
Amassing and indexing a large
sample of version control systems
Audris Mockus
[MSR 2009]
Boa: a language and infrastructure for analyzing
ultra-large-scale software repositories
R. Dyer et al.
[ICSE 2013]
The GHTorrent Dataset and Tool
Suite
G. Gousios
[MSR 2013]
We collect all of the data in the world,
but it’s meaningless without context!

133. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release

134. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release

135. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk
of releasing software
if it hurts, do it
more frequently, and
bring the pain
forward
Jez Humble

136. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk
of releasing software
if it hurts, do it
more frequently, and
bring the pain
forward
Jez Humble

137. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk
of releasing software
if it hurts, do it
more frequently, and
bring the pain
forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party
dependency
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2 sync

138. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk
of releasing software
if it hurts, do it
more frequently, and
bring the pain
forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party
dependency
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2
sync

139. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk
of releasing software
if it hurts, do it
more frequently, and
bring the pain
forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party
dependency
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2
sync
but some are more equal than others
1. All releases are equal,
2. All branches are equal,
Harmful assumptions about release pipelines
that can impact predictive modelling
3. All files are equal

140. Release
Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk
of releasing software
if it hurts, do it
more frequently, and
bring the pain
forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party
dependency
feature branch 1
git
repo
release branch
MERGE CONFLICT?
myapp
v.1
feature branch 2
sync
but some are more equal than others
1. All releases are equal,
2. All branches are equal,
Harmful assumptions about release pipelines
that can impact predictive modelling
3. All files are equal