1.
On
Software Release
Engineering
Bram Adams
M
C IS
Latest version: http://mcis.polymtl.ca/~bram/publications/2012/icse_tb.pdf

2.
M (Lab on Maintenance,
C IS Construction and Intelligence
of Software)

3.
On average we
deploy new code fifty times
a day.
M
C IS

4.
Continuous Delivery
15k tests
CVS
test
staging/production
continuous
9 min.
integration 6 http://goo.gl/qPT6

5.
Continuous Delivery
15k tests
CVS
test
staging/production
continuous
9 min.
integration 6 http://goo.gl/qPT6

6.
Continuous Delivery
15k tests
CVS
test
staging/production
continuous
9 min. 6 min.
integration 6 http://goo.gl/qPT6

7.
Work fast and
don’t be afraid to break
things.
http://goo.gl/UlCW

8.
Build a
little and then test it.
Build some more and test
some more.
James Whittaker

9.
Even Desktop Apps Release
More Frequently
40
#releases
http://en.wikipedia.org/wiki/History_of_Firefox
32
?
24
16
8
0
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
9

10.
http://hacks.mozilla.org/wp-content/uploads/2012/05/firefox-releases.jpg
Firefox' New Release
Schedule (1)

11.
http://hacks.mozilla.org/wp-content/uploads/2012/05/firefox-releases.jpg
Firefox' New Release
Schedule (2)

12.
http://hacks.mozilla.org/wp-content/uploads/2012/05/rapid-release.jpg
Release Pipeline

13.
Firefox Release Howto (1)
85+ configuration tag
build
repositories sanity check everything

14.
Firefox Release Howto (2)
generate
incremental
updates for
each supported
old version
i18n signing

15.
http://tims-ideas.blogspot.com/2011/05/live-blog-from-icse-view-of-icses.html
... yet Software Systems
keep on Growing!
<150k tests/ >50k build
>5k developers commit s/day
build cache with >2k projects >50M tests/day
>50TB memory
20 code compilation
ges/min. in the cloud
chan
15

16.
Release Engineering
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
in-house/3rd party integrate
development
red test build
uc
ec
ycl
et
im
e!
16 deployment

17.
Out of Scope (but Highly Related)
release variability
release planning
man agement management
app stores ... software process
i mpact on packaging
project technologies software
ma nagement product lines
17

18.
Release Engineering
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
in-house/3rd party integrate
development
red test build
uc
ec
ycl
et
im
e!
18 deployment

19.
The Build
Process
19

20.
The Build Process
?
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
20

21.
Step 1 - Configuration
Autotools
Features
Tools
21

22.
Step 2 - Construction Make
Prescriptions .c
.o
=
Dependencies .o
.o
.o
.exe
22

23.
+
Source Code Build System
23

24.
24

25.
Our record
so far is a pr
we inherited oject
with an Ant
weighing in a script
t 10,000 line
XML. Needl s of
ess to say, th
project requ is
ired an entir
team devote e
d to keeping
build workin the
g—a comple
waste of r te
KDE 4 is leaving the aging esources.
[Jez Humble
"autotool" build chain behind. & David Farl
ey]
Some developers, not only in
KDE, like to nickname the
autotools as "auto-hell" Build Systems
because of its difficulty to
comprehend architecture.
[http://lwn.net/Articles/188693/] 25
are Complex

26.
>5 MLOC & ~20 Years of History
"Design Recovery and Maintenance of Build Systems" (Adams et al.)
"The Evolution of the Linux Build System" (Adams et al.) 26

27.
Quake 3 client UI
server
game logic
client renderer
quake3.exe
network
d(a)emon
27

28.
Quake 3 conditional compilation
original game
expansion
pack
28

29.
"The Evolution of the Linux Build System" (Adams et al.)
Linux 2.6.16.18
0.01
29

30.
0.01 0.11 1.0
"The Evolution of the Linux Build System" (Adams et al.)
Linux 2.6.16.18
2.0 2.2
1.2
2.4 2.6.0 2.6.21.5
29

31.
0.01 0.11 1.0
"The Evolution of the Linux Build System" (Adams et al.)
Linux 2.6.16.18
2.0 2.2
1.2
Build Systems Grow in
Complexity
2.4 2.6.0 2.6.21.5
29

32.
Build Systems Require 12%
of a Developer’s
Time (on average)
Build maintenance
slows down development!
Software in the DOE: The Hidden Overhead of the “Build” (Kumfert et al.) 30

33.
An Empirical Study of Build Maintenance Effort (McIntosh et al.)
>36 MLOC & ~120 Years of History
31

34.
An Empirical Study of Build Maintenance Effort (McIntosh et al.)
Build Files Change Relatively
More than Source Code Files
5
4.5
#changed source lines/source file
4
#changed build lines/build file
3.5
3
2.5
2
1.5
1
0.5
0
ArgoUML Hibernate Eclipse GCC Git Linux Mozilla PLPlot PostgreSQL
32

35.
The Build System Requires
Significant Maintenance
Low coupling thanks
44%
to higher-level build
Source Build
Test Build
abstraction
27%
20%
16%
12%
8%
4%
33

36.
Our Build Systems need HELP
34

37.
35

38.
A B C
D
35

39.
A D B C
36

40.
all files/pro
stat ic build ducts
stored in c
depe ndencies loud
A D B C
tr ansparent
access via FUSE
36

41.
Build System Quality
SYMake
(Tamrawi et al.)

42.
Build System Quality
Clipper
(Morgenthaler et al.)
Tu et al., Nadi et al., Berger et al., Tartler et al., ...

43.
Release Engineering
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
in-house/3rd party integrate
development
red test build
uc
ec
ycl
et
im
e!
38 deployment

44.
The Integration Process
39

45.
1. The Paradox of Early Integration
40

46.
1. The Paradox of Early Integration
release branch
40

47.
1. The Paradox of Early Integration
release branch
40

48.
1. The Paradox of Early Integration
feature
branches
40

49.
1. The Paradox of Early Integration
40

50.
Cohesive and Isolated Development with Branches (Barr et al.)
Integrations Interrupt Development
once in every 24.4 Commits in DVC
changed files: A, B, C, D and E
changed files: A, E and F
distraction |{A, B, C, D, E}| ∩ |{A, E, F}|
= = 66.6%
on merge |{A, E, F}|
assumption: Pr[real integration conflict] = 90%
41

51.
Proactive Detection of Collaboration Conflicts (Brun et al.)
... and when they Do,
Integrations cause Trouble
on average 16% of on average
all merges had on average 5% 11.7% of all
textual conflicts of all merges had merges had test
(manual resolution build issues issues
needed)
42

52.
The Effect of Branching Strategies on Software Quality (Shihab et al.)
Branching Degrades
Quality of Components
branch
family
file branched/merged
43

53.
The Effect of Branching Strategies on Software Quality (Shihab et al.)
Branching Degrades
Quality of Components
branch
family
file branched/merged
43

54.
The Effect of Branching Strategies on Software Quality (Shihab et al.)
Branching Degrades
Quality of Components
branch
family
file branched/merged
43

55.
Proactive Detection of Collaboration Conflicts (Brun et al.)
Speculative Merge Prevention
B T
S AME A HEAD B EHIND T EXTUAL8 B UILD8 T EST8 T ESTX 44

56.
A Cost-Benefit Approach to Recommending Conflict Resolution for
Parallel Software Development (Niu et al.)
Conflict Resolution Recommendation
method of
interest how hard to resolve conflict?
(a) Using cost-benefit ratio to rank the conflicting procedures
ranking of methods to how much benefit does
resolve conflicts with 45 resolving conflict yield?

57.
2. The Nightmare of 3rd Party
Integration
46 vendor branch

58.
47

59.
Integrators are
Gatekeepers
end users
47

60.
Integrators are
Gatekeepers
maintainers
end users
47

61.
>180k Packages & ~28 Years of History
48

62.
10 20 30 40 50 60
%package versions
0
A B C D E F
New Package Local Patch
Upstream Sync Product-wide
Dependency Concern
Build Change
Management 49

63.
Mozilla Delivers New Necessary?
Version of Firefox –
First Web Browser to Buggy?
Upstream Sync
Support Do Not Track on Secure?
Multiple Platforms!
[Mozilla Blog]
50

64.
Risk Analysis & Cherry-picking
Upstream Sync
UNIX diff
maintainer
51

65.
10 20 30 40 50 60
%package versions
0
A B C D E F
New Package Local Patch
Upstream Sync Product-wide
Dependency Concern
Build Change
Management 52

66.
Library Transitions Ripple
through the whole System
Empir Software Eng (2009) 14:262–285 279
Dependency
Management
xcontrib xfree86–common
xlib6g debianutils
gconv–modules
libgtk1.2
libglib1.2
libc6 ldso
libstdc++2.10
mozilla
libjpeg62
libpng2
zlib1g
libz1
libnspr4
Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of the
Fig. 7 Most popular instance
evolution: a case study for large software compilation
(Gonzalez-Barahona et al.)
two abstract dependencies have only one child

67.
Library Transitions Ripple
through the whole System
Empir Software Eng (2009) 14:262–285 279
Dependency
Management
xcontrib xfree86–common
xlib6g debianutils
gconv–modules
libgtk1.2
libglib1.2
libc6 ldso
libstdc++2.10
mozilla
libjpeg62
libpng2
zlib1g
libz1
libnspr4
Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of the
Fig. 7 Most popular instance
evolution: a case study for large software compilation
(Gonzalez-Barahona et al.)
two abstract dependencies have only one child

68.
Library Transitions Ripple
through the whole System
Empir Software Eng (2009) 14:262–285 279
Dependency
Management
xcontrib xfree86–common
xlib6g debianutils
gconv–modules
libgtk1.2
libglib1.2
libc6 ldso
libstdc++2.10
mozilla
libjpeg62
libpng2
zlib1g
libz1
libnspr4
Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of the
Fig. 7 Most popular instance
evolution: a case study for large software compilation
(Gonzalez-Barahona et al.)
two abstract dependencies have only one child

69.
Library Transitions Ripple
through the whole System
Empir Software Eng (2009) 14:262–285 279
Dependency
Management
xcontrib xfree86–common
xlib6g debianutils
gconv–modules
libgtk1.2
libglib1.2
libc6 ldso
libstdc++2.10
mozilla
libjpeg62
libpng2
zlib1g
libz1
libnspr4
Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of the
Fig. 7 Most popular instance
evolution: a case study for large software compilation
(Gonzalez-Barahona et al.)
two abstract dependencies have only one child

70.
2-way Impact Analysis Dependency
Management
Did Software Eng (2009) 14:262–285
Empir 279
someone Did I break
break my someone's
package?
xcontrib xlib6g
xfree86–common
debianutils package?
gconv–modules
libgtk1.2
libglib1.2
libc6 ldso
libstdc++2.10
mozilla
libjpeg62
libpng2
zlib1g
libz1
libnspr4
Fig. 7 Most popular instance of the Inter-Dependency Graph for mozilla in Debian 2.2. Each of the
two abstract dependencies have only one child 54

71.
If an engineer comes up
with a better way of doing
something, he is tasked
with refactoring all
existing libraries and
assisting dependent
projects to migrate
to the new libraries
James Whittaker

72.
10 20 30 40 50 60
%package versions
0
A B C D E F
New Package Local Patch
Upstream Sync Product-wide
Dependency Concern
Build Change
Management 56

73.
patch
upstream
accepted
Upstream Upstream Upstream
Sync Sync Sync
maintainer
Local Patch Local Patch
57

74.
Package : openssl
Vulnerability : predictable random number generator
Problem type : remote
Debian-specific: yes Local Patch
CVE Id(s) : CVE-2008-0166
Date : 2008-05-13
Luciano Bello discovered that the random number generator in Debian's
openssl package is predictable. This is caused by an incorrect
Debian-specific change to the openssl package (CVE-2008-0166). As a
result, cryptographic key material may be guessable.
It is strongly
>=1.5 years 8-(
recommended that all cryptographic key material which
has been generated by OpenSSL versions starting with 0.9.8c-1 on
Debian systems is recreated from scratch. Furthermore, all DSA keys
ever used on affected Debian systems for signing or authentication
purposes should be considered compromised; the Digital Signature
Algorithm relies on a secret random value used during signature
generation.
>=44 distributions ;-)
The first vulnerable version, 0.9.8c-1, was uploaded to the unstable
distribution on 2006-09-17, and has since propagated to the testing
and current stable (etch) distributions. The old stable distribution
(sarge) is not affected.
58

75.
Release Engineering
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
in-house/3rd party integrate
development
red test build
uc
ec
ycl
et
im
e!
59 deployment

76.
Incremental Testing

77.
Different Test Phases
(selection of) unit tests
integration + build
all unit tests
acceptance/integration tests
capacity tests
user acceptance tests
release
smoke tests
test phase time to run (hours)

78.
Seamless Upgrades (1): Blue/Green
Deployment
old version
of system
users DB
Disk
router
DB
Disk

79.
Seamless Upgrades (1): Blue/Green
Deployment
old version
of system
users DB
Disk
router
DB
Disk
new version
of system

80.
Seamless Upgrades (1): Blue/Green
Deployment
old version
of system
users DB
Disk
router
DB
Disk
new version
of system

81.
Seamless Upgrades (1): Blue/Green
Deployment
old version
of system
users DB
Disk
router
DB
Disk
new version
rollback!! of system

82.
Seamless Upgrades (1): Blue/Green
Deployment
old version
of system
users DB
Disk
router
DB
Disk
new version
rollback!! of system

83.
Seamless Upgrades (2):
Canary Deployment
DB
Disk
users old version
of system
router DB
Disk
old version
of system
DB
Disk
old version
of system

84.
Seamless Upgrades (2):
Canary Deployment
DB
Disk
users old version
new version
of system
router DB
Disk
old version
of system
DB
Disk
old version
of system

85.
Seamless Upgrades (2):
Canary Deployment
DB
Disk
users old version
new version
of system
router DB
Disk
oewveersion
nld v rsion
o system
off system
DB
Disk
old version
of system

86.
Seamless Upgrades (2):
Canary Deployment
DB
Disk
users old version
new version
of system
router DB
Disk
oewveersion
nld v rsion
o system
off system
DB
Disk
oewveersion
nld v rsion
o system
off system

87.
Testing Challenge 1:
Parallelization

88.
Testing Challenge 2:
Incrementalizing Tests

89.
Testing Challenge 3:
Optimizing Test Speed
DB
Disk

90.
Testing Challenge 4: Data
Migration

91.
Testing Challenge 5: Process
Integration

92.
Open Challenges for
Release Engineering
69

93.
? Certificate
= of
High Quality
rapid release
Do Faster Releases Improve Software Quality? - An Empirical Case Study of
Mozilla Firefox (Khomh et al.)

94.
Traditional Release Cycle Rapid Release Cycle
6.0 8.0
1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 7.0 9.0

95.
Rapid Release Cycle
6.0 8.0
3.6 4.0 5.0 7.0 9.0
Certificate
vs. of
High Quality
release&cycle&length
Do Faster Releases Improve Software Quality? - An Empirical Case Study of
Mozilla Firefox (Khomh et al.)

96.
Same # Post-release Bugs
13 20
5
Number of Post Release Bugs Per Day
4
3
2 1.7
1.6
1
0
Traditional Release (TR) Rapid Release (RR)

97.
Crashes Pop Up Earlier
1200
1080
960
Median UpTime in Seconds
840
720
643
600
480
370
360
240
120
0
Traditional Release (TR) Rapid Release (RR)

98.
Rapid Release Cycle
6.0 8.0
3.6 4.0 5.0 7.0 9.0
vs.
bug&fixing
release&cycle&length
Do Faster Releases Improve Software Quality? - An Empirical Case Study of
Mozilla Firefox (Khomh et al.)

99.
Bugs are Fixed Faster
654 159
100
90
80
70
Bug Age in Days
60
50
40
30
20 16
10 6
0
Traditional Release (TR) Rapid Release (RR)

100.
Proportionally Less Bugs Fixed
100%
90%
80%
70% 67%
60% 59%
% of Bugs Fixed
50% 43%
40% 37%
30%
20%
10%
0%
Traditional Rapid Release Traditional Rapid Release
Release (TR) - (RR) - Main Release (TR) - (RR) - Beta
Main Beta

101.
What about Other Projects?
Certificate
of
High Quality

102.
What about Other Projects?
Certificate
of
High Quality

103.
How to Make Software
and Build Comprehension
Pragmatic?
40
79

104.
Where are the Tools?
refactor test your
your makefiles! build!
RELEASE IDE
what did the keep poking those
other team break upstream guys until they
now ;-) give in :-)
80

105.
Work fast and
don’t be afraid to break
things.
We need more
canaries!
http://goo.gl/UlCW

106.
On average we
deploy new code fifty times
a day.
M
C IS

107.
On average we
deploy new code fifty times
a day.
M
C IS

108.
Release Engineering
On average we
deploy new code fifty times
a day.
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
in-house/3rd party integrate
M
development C IS
kee
pi n test build
gc
ycl
et
im
ed
ow
n
7 deployment

109.
Release Engineering
On average we
deploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
a day. in-house/3rd party integrate
development
kee
pi n test build
gc
ycl
et
im
M
ed
C IS ow
n
7 deployment

110.
Release Engineering
The Build System Requires
On average we
deploy new code fifty times
Significant Maintenance
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
a day. in-house/3rd party integrate
development
kee
pi n test build
gc
ycl
et
im
M 44% ed
C IS ow
n
Source Build 7 deployment
Test Build
27%
20%
16%
12%
8%
4%
21

111.
Release Engineering
On average we
deploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
a day. in-house/3rd party integrate
development
kee
pi n test build
gc
ycl
et
im
M
ed
C IS ow
n
7 deployment
The Build System Requires
Significant Maintenance
44%
Source Build
Test Build
27%
20%
16%
12%
8%
4%
21

112.
Release Engineering
Risk Analysis & Cherry-picking
On average we
deploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
a day. in-house/3rd party integrate
development
kee
pi n test build
gc Upstream Sync
ycl
et
im
M
ed
C IS ow
n
7 deployment
The Build System Requires
diff
Significant Maintenance
44%
Source Build
Test Build
27%
12%
16%
20%
maintainer
8%
4%
21

113.
Release Engineering
On average we
deploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg
a day. in-house/3rd party integrate
development
kee
pi n test build
gc
ycl
et
im
M
ed
C IS ow
n
7 deployment
The Build System Requires Risk Analysis & Cherry-picking
Significant Maintenance
Upstream Sync
44%
Source Build
Test Build
27%
diff
20%
16%
12%
8%
4%
maintainer
21

114.
M
C IS
http://mcis.polymtl.ca/
http://google-engtools.blogspot.ca/