So, you are saying that our software
quality was screwed up by ... the

release engineer?!

M

Bram Adams

M

C IS

C IS
Who is
Bram Adams?
Herman Tromp
Ghent University

Wolfgang De Meuter
Vrije Universiteit Brussel
Ahmed E. Hassan
Queen's University
M

C IS

(Lab on Maintenance,
Construction and Intelligence
of Software)
You?

Mini

Parastou
Jojo

M

C IS
Tejinder Dhaliwal Mika Mäntylä Foutse Khomh Ahmed E. Hassan

joint work with

Daniel German Emelie Engström

Kai Petersen
...
So, you are saying that our software
quality was screwed up by ... the

release engineer?!

M

Bram Adams

M

C IS

C IS
M

C IS
On average we
deploy new code fifty times
a day.

M

C IS
Continuous Delivery
15k tests
VCS

test
staging/production
continuous
integration

9 min.
8

http://goo.gl/qPT6
Continuous Delivery
15k tests
VCS

test
staging/production
continuous
integration

9 min.
8

http://goo.gl/qPT6
Continuous Delivery
15k tests
VCS

test
staging/production
continuous
integration

9 min.

6 min.
8

http://goo.gl/qPT6
BENEVOL-ers

IMVU, eh?
Work fast and
don’t be afraid to break
things.

http://goo.gl/UlCW
Build a
little and then test it.
Build some more and test
some more.

James Whittaker
Even Desktop Apps Release
More Frequently
#releases
32

?

24
16
8
0

2002

2003

2004

2005

2006
12

2007

2008

2009

2...
http://hacks.mozilla.org/wp-content/uploads/2012/05/rapid-release.jpg

... although it's
Kind of an Illusion!
Rapid Release
in a Nutshell
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg

integrate

in-house/3rd party
developme...
BENEVOL-ers

Are Rapid Releases
Worth the Effort?
rapid release

?
=
OR

?
=
rapid release

Certificate
of
High Quality
Traditional Release Cycle
1.0

1.5

2.0

Do Faster Releases Improve Software
Quality? - An Empirical Case Study of
Mozilla...
Traditional Release Cycle
1.0

1.5

2.0

Do Faster Releases Improve Software
Quality? - An Empirical Case Study of
Mozilla...
Socorro crash
reports
Bugzilla bug
reports
Rapid Release Cycle
3.6

6.0 8.0
4.0 5.0 7.0 9.0

vs.
release&model
Same # Post-release Bugs
Same # Post-release Bugs
Number of Post Release Bugs Per Day

13

20

1.6

1.7

5
4
3
2
1
0

Traditional Release (TR)

Rap...
We have about 30 of these project branches active at
any one time. That means as a company we can have
30 or more experime...
Crashes Pop Up Earlier
1200

Median UpTime in Seconds

1080
960
840
720

643

600
480
370

360
240
120

0

Traditional Rel...
Firefox 4 was a crisis time for Firefox and Mozilla,
since it took so long to complete and was a painful
engineering effor...
Rapid Release Cycle
3.6

6.0 8.0
4.0 5.0 7.0 9.0

vs.
release&model

bug&fixing
Bugs are Fixed Faster
654

159

100
90
80

Bug Age in Days

70
60
50
40
30
20

16

10

6

0

Traditional Release (TR)

Rap...
Proportionally Less

Bugs Fixed

100%
90%
80%
70%

% of Bugs Fixed

60%

67%
59%

50%

43%

40%

37%

30%
20%
10%
0%

Trad...
I’m not surprised that fewer bugs are fixed: some
defects are trivial and some will take weeks of careful
code inspection t...
can be better interpreted that we are being less
effective at triaging bugs with rapid release, or
that we have more beta ...
same
#bugs

... but faster
crashes

bugs are
fixed faster

... but less
bugs are
being fixed
BENEVOL-ers

Shall we Blame
the Testers?
Rapid Release
in a Nutshell
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg

in-house/3rd party
development

red
uc
...
Did the System
Testing Process
Change?
'06

'07
2.0

'08

'09
3.0

5.0-13.0

'10

'11

3.5 3.6

'12

4.0

Rapid Release Mo...
Different Test Phases
(selection of) unit tests
integration + build
all unit tests
integration tests
capacity tests
user a...
Litmus

testcase id #8410 - all tab options unchecked

get testing!
run tests
view/search tests

reporting
search results
...
Litmus

testcase id #8410 - all tab options unchecked

get testing!
run tests
view/search tests

All Tab options unchecked...
35,000 - 55,000
executions/release

e as many
... but twic
ns per day
est executio
t
6,000 - 9,000
executions/release

Tes...
To survive under the time
constraints of a rapid release
we’ve had to cut the fat and
focus on those test areas
which are ...
700 - 1,100
different test cases

Smaller Scope
of Testing
100 - 170 different
test cases
a fixed set of tests for areas prone to regression
(Flash plugin testing for example)
and
a dynamically changing set of tes...
1,000 - 1,900
testers

Smaller
Testing Team
8 - 34 testers
The weakest point [in rapid release] is that it’s harder
to develop a large community which more
accurately represents the...
1) the core team has remained largely unchanged since
adopting rapid release 2) the contract team has nearly
doubled ... W...
0

2

4

OSs per day

1.0
0.5
0.0

Locales per day

6

1.5

8

2.0

Test Priorities

Traditional

Rapid

less locales bein...
we now distribute across Betas.
For example, we might test Windows 7,
OSX 10.8, and Ubuntu in one Beta then
Windows XP, Ma...
In Other Words ...
Testing has to
become
more
Focused
The greatest strength [of rapid releases] is
that the scope of what needs to be
tested is narrow so we can focus all of
ou...
Release Engineering
http://behrns.files.wordpress.com/2008/03/ikea-car.jpg

in-house/3rd party
development

red
uc
ed

inte...
MSR '13, Jiang et al.

Will My Patch
Make It?
(And How Fast?)

50
I do hold out hope that Google does come
around and works to fix their codebase to
get it merged upstream to stop the huge
...
The Linux Integration Process

contributor 1

linux-usb

contributor 2

lkml

contributor 3

linux-scsi

Reviewing

subsys...
Research Questions

How many
patches are
merged?

What kind of
patch is merged
more likely?

What kind of
patch is accepte...
Case Study Setup

contributor 1

linux-usb

contributor 2

lkml

contributor 3

linux-scsi

Reviewing

subsystem
maintaine...
Case Study Setup

contributor 1

linux-usb

contributor 2

lkml

contributor 3

Linus
Torvalds

Linux 3.5

linux-scsi

Rev...
Case Study Setup

linux-usb

lkml

Linus
Torvalds

linux-scsi

54
Case Study Setup

linux-usb

lkml

Linus
Torvalds

linux-scsi

54
Case Study Setup
email1

email
patch1

email2

email
patch2

email3

...

Linus
Torvalds

email
patch3

54
Case Study Setup
email1

email
patch1

commit
patch1

commit1

email2

email
patch2

commit
patch2

commit2

email3

email...
Case Study Setup
checksum1

commit
patch1

commit1

email2

email
patch2

checksum2

commit
patch2

commit2

email3

email...
Case Study Setup
checksum1

commit
patch1

commit1

email2

email
patch2

checksum2

commit
patch2

commit2

email3

email...
Experience

Commit

5 Dimensions
(29 Patch Metrics)

Patch

Review
55

Email
Building Decision Trees
size: LOC > 50

Is this first patch in thread?

not accepted

Number of reviewers > 3 ?

Number of ...
How many
patches are
merged?

What kind of
patch is merged
more likely?

What kind of
patch is accepted
faster?
33% of Patches Make it ...
accepted/rejected patches
69.26
69.26%
120000
% accepted by linus
% rejected by linus
100000

p...
80

... and it takes 1~6 Months!

40

60

within_half_year
within_year
took_ages

20

Text

%
accepted
patches

within_wee...
70
60

within_half_year
within_year
took_ages

30

40

50

within_week
within_month
within_quarter

0

10

20

percentage ...
60

60
20

30

40

50

within_day

within_week
within_half_year
within_month
within_week within_year
within_half_year
with...
How many
patches are
merged?

33% patch
cceptance
a
integration
becomes slower

What kind of
patch is merged
more likely?
...
How many
patches are
merged?

33% patch
cceptance
a
integration
becomes slower

What kind of
patch is merged
more likely?
...
Experience Matters Most for
Patch Acceptance

precision:73%
recall:68.47%

65
How many
patches are
merged?

What kind of
patch is merged
more likely?

33% patch
cceptance
a

y experienced
b
developers...
How many
patches are
merged?

What kind of
patch is merged
more likely?

33% patch
cceptance
a

y experienced
b
developers...
Experience, Reviewers and Patch
Spread Matter for Reviewing Time

68
How many
patches are
merged?

What kind of
patch is merged
more likely?

What kind of
patch is accepted
faster?

33% patch...
Release
Engineering

?
70

Software
Quality
Rapid Release vs. Quality?
Certificate
of
High Quality
branch-based
integration

vs.

trunk-based
integration

72
0.01

0.11

1.0

"The Evolution of the Linux Build System" (Adams et al.)

Linux 2.6.16.18

1.2

2.2

2.0

How to Evolve B...
How to
Focus
Testing?
How to Keep up with 3rd Party
Dependencies?

75

3rd party branch
Where are the Release
Engineering Tools?
focus your
testing efforts

test your
build!

RELEASE IDE

keep poking those
upst...
How to Align Release Engineers
with the Rest of the Organization?
M

C IS

Testing has to
become
more
Focused

... but faster
crashes

bugs are
fixed faster

On average we
deploy new code fi...
1st International Workshop on Release Engineering
RELENG 2013
6 practition
er
talks

84
participants

2 keynotes

... and ...
M

C IS

Testing has to
become
more
Focused

... but faster
crashes

bugs are
fixed faster

On average we
deploy new code fi...
Upcoming SlideShare
Loading in …5
×

So, you are saying that our software quality was screwed up by ... the release engineer?! (Keynote @ Benevol 2013, Mons, Belgium)

3,594 views

Published on

Software release engineering is the discipline of integrating, building, testing, packaging and delivering qualitative software releases to the end user. Whereas software used to be released in shrink-wrapped form once per year, modern companies like Intuit, Google and Mozilla only need a couple of days or weeks in between releases, while lean start-ups like IMVU release up to 50 times per day! Shortening the release cycle of a software project requires considerable process and development changes in order to safeguard product quality, yet the scope and nature of such changes are unknown to many. For example, while migrating towards rapid releases allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing.

To understand these implications, we empirically studied the development process of Mozilla Firefox in 2010 and 2011, a period during which the project transitioned to a shorter release cycle. By comparing crash rates, median uptime, and the proportion of post-release bugs before and after the migration, we found that (1) with shorter release cycles, users do not experience significantly more post-release bugs and that (2) less bugs are being fixed, but those that are fixed are fixed faster.

In order to validate these findings, we interviewed Mozilla QA personnel and empirically investigated the changes in software testing effort after the migration towards rapid releases. Analysis of the results of 312,502 execution runs of Mozilla Firefox from 2006 to 2012 (5 major traditional and 9 major rapid releases) showed that in rapid releases testing has a narrower scope that enables deeper investigation of the features and regressions with the highest risk, while traditional releases run the whole test suite. Furthermore, rapid releases make it more difficult to build a large testing community, forcing Mozilla to increase contractor resources in order to sustain testing for rapid releases. In other words, rapid releases are able to bring improvements in software quality, yet require changes to bug triaging, fixing and testing processes to avoid unforeseen costs.

Finally, we also discuss a case study on software integration in the Linux kernel using git. We show the important factors that not only impact patch acceptance but also the time until acceptance, which is essential when optimizing the release engineering process for rapid releases.

Published in: Technology, News & Politics
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,594
On SlideShare
0
From Embeds
0
Number of Embeds
1,861
Actions
Shares
0
Downloads
14
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

So, you are saying that our software quality was screwed up by ... the release engineer?! (Keynote @ Benevol 2013, Mons, Belgium)

  1. 1. So, you are saying that our software quality was screwed up by ... the release engineer?! M Bram Adams M C IS C IS
  2. 2. Who is Bram Adams?
  3. 3. Herman Tromp Ghent University Wolfgang De Meuter Vrije Universiteit Brussel
  4. 4. Ahmed E. Hassan Queen's University
  5. 5. M C IS (Lab on Maintenance, Construction and Intelligence of Software)
  6. 6. You? Mini Parastou Jojo M C IS
  7. 7. Tejinder Dhaliwal Mika Mäntylä Foutse Khomh Ahmed E. Hassan joint work with Daniel German Emelie Engström Kai Petersen Ying Zou
  8. 8. So, you are saying that our software quality was screwed up by ... the release engineer?! M Bram Adams M C IS C IS
  9. 9. M C IS
  10. 10. On average we deploy new code fifty times a day. M C IS
  11. 11. Continuous Delivery 15k tests VCS test staging/production continuous integration 9 min. 8 http://goo.gl/qPT6
  12. 12. Continuous Delivery 15k tests VCS test staging/production continuous integration 9 min. 8 http://goo.gl/qPT6
  13. 13. Continuous Delivery 15k tests VCS test staging/production continuous integration 9 min. 6 min. 8 http://goo.gl/qPT6
  14. 14. BENEVOL-ers IMVU, eh?
  15. 15. Work fast and don’t be afraid to break things. http://goo.gl/UlCW
  16. 16. Build a little and then test it. Build some more and test some more. James Whittaker
  17. 17. Even Desktop Apps Release More Frequently #releases 32 ? 24 16 8 0 2002 2003 2004 2005 2006 12 2007 2008 2009 2010 2011 http://en.wikipedia.org/wiki/History_of_Firefox 40
  18. 18. http://hacks.mozilla.org/wp-content/uploads/2012/05/rapid-release.jpg ... although it's Kind of an Illusion!
  19. 19. Rapid Release in a Nutshell http://behrns.files.wordpress.com/2008/03/ikea-car.jpg integrate in-house/3rd party development red uc ec test ycl et build im e! 14 packaging & delivery
  20. 20. BENEVOL-ers Are Rapid Releases Worth the Effort?
  21. 21. rapid release ? = OR ? = rapid release Certificate of High Quality
  22. 22. Traditional Release Cycle 1.0 1.5 2.0 Do Faster Releases Improve Software Quality? - An Empirical Case Study of Mozilla Firefox (MSR '12, Khomh et al.) 3.0 Rapid Release Cycle 3.5 3.6 6.0 8.0 4.0 5.0 7.0 9.0
  23. 23. Traditional Release Cycle 1.0 1.5 2.0 Do Faster Releases Improve Software Quality? - An Empirical Case Study of Mozilla Firefox (MSR '12, Khomh et al.) 3.0 Rapid Release Cycle 3.5 3.6 6.0 8.0 4.0 5.0 7.0 9.0
  24. 24. Socorro crash reports
  25. 25. Bugzilla bug reports
  26. 26. Rapid Release Cycle 3.6 6.0 8.0 4.0 5.0 7.0 9.0 vs. release&model
  27. 27. Same # Post-release Bugs
  28. 28. Same # Post-release Bugs Number of Post Release Bugs Per Day 13 20 1.6 1.7 5 4 3 2 1 0 Traditional Release (TR) Rapid Release (RR)
  29. 29. We have about 30 of these project branches active at any one time. That means as a company we can have 30 or more experiments safely running in parallel....I’m guessing that most of the bugs end up being integration issues [of these branches into the main trunk] otherwise the project branch wouldn’t have merged back in with mainstream devel.
  30. 30. Crashes Pop Up Earlier 1200 Median UpTime in Seconds 1080 960 840 720 643 600 480 370 360 240 120 0 Traditional Release (TR) Rapid Release (RR)
  31. 31. Firefox 4 was a crisis time for Firefox and Mozilla, since it took so long to complete and was a painful engineering effort. We decided to make our development more agile by switching to rapid release at the same time as we were making a bunch of other changes [...], and starting Firefox OS development
  32. 32. Rapid Release Cycle 3.6 6.0 8.0 4.0 5.0 7.0 9.0 vs. release&model bug&fixing
  33. 33. Bugs are Fixed Faster 654 159 100 90 80 Bug Age in Days 70 60 50 40 30 20 16 10 6 0 Traditional Release (TR) Rapid Release (RR)
  34. 34. Proportionally Less Bugs Fixed 100% 90% 80% 70% % of Bugs Fixed 60% 67% 59% 50% 43% 40% 37% 30% 20% 10% 0% Traditional Release (TR) Main Rapid Release (RR) - Main Traditional Release (TR) Beta Rapid Release (RR) - Beta
  35. 35. I’m not surprised that fewer bugs are fixed: some defects are trivial and some will take weeks of careful code inspection to find and fix. All those bugs would have fallen in the same bucket for a 12+ month release cycle, but now the harder ones get pushed out to later releases
  36. 36. can be better interpreted that we are being less effective at triaging bugs with rapid release, or that we have more beta users and so the incoming bug rates are larger
  37. 37. same #bugs ... but faster crashes bugs are fixed faster ... but less bugs are being fixed
  38. 38. BENEVOL-ers Shall we Blame the Testers?
  39. 39. Rapid Release in a Nutshell http://behrns.files.wordpress.com/2008/03/ikea-car.jpg in-house/3rd party development red uc ec integrate test ycl et build im e! 33 packaging & delivery
  40. 40. Did the System Testing Process Change? '06 '07 2.0 '08 '09 3.0 5.0-13.0 '10 '11 3.5 3.6 '12 4.0 Rapid Release Model vs. Software Testing Effort – Study of Mozilla Firefox Project (ICSM '13, Mäntylä et al.)
  41. 41. Different Test Phases (selection of) unit tests integration + build all unit tests integration tests capacity tests user acceptance/system tests release smoke tests test phase time to run (hours)
  42. 42. Litmus testcase id #8410 - all tab options unchecked get testing! run tests view/search tests reporting search results advanced search testdays test runs statistics All Tab options unchecked Product: Firefox Branch: 3.6 Branch Regression Bug ID #: log in 8410 Summary: welcome Testcase ID #: None specified Enabled? Community Enabled? Test Scenario & Expected Outcome Steps to Perform: Expected Results: 1. Open Tool | Options (Preferences) | Tabs. Make sure all check boxes are unchecked. Confirm that all four preferences do not work, since they are disabled: A new window is opened instead of a new tab. You will not be warned when closing multiple tabs. You will not be warned when opening multiple tabs (>15 at once, e.g. the default RSS folder in the toolbar). The tab bar is not shown when only one tab is open. Opening a link in a new tab will open it in the background. help litmus tutorial litmus wiki Testgroup(s) Subgroup(s) 3.6 Full Functional Tests (FFTs) (137) FX 3.6 FFT - Tabbed Browsing (1242) recent test results for testcase id #8410 - all tab options unchecked submission date Recorded Test Results result id# testcase id#: summary 2010-11-02 02:16:00 336691 8410: All Tab options unchecked 2010-07-07 10:07:17 279135 2010-05-12 11:20:12 product platform branch locale Firefox 3.6 Branch en-US 8410: All Tab options unchecked Firefox 3.6 Branch en-US 257854 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2010-01-17 02:21:29 232877 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2010-01-11 20:56:35 230165 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2010-01-07 10:50:21 228669 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2009-10-22 03:22:31 213485 8410: All Tab options unchecked Firefox 3.6 Branch en-US search test results: recent failures | recently unclear | most common failures | testcases most frequently marked as unclear | testgroup popularity | results with comments search testcases: testcases needed! | recently added | recently changed | by id:
  43. 43. Litmus testcase id #8410 - all tab options unchecked get testing! run tests view/search tests All Tab options unchecked Product: Firefox Branch: 3.6 Branch Regression Bug ID #: log in 8410 Summary: welcome Testcase ID #: None specified Enabled? - 06/2012 06/2006 reporting search results advanced search testdays test runs statistics 1,547 test cases Community Enabled? 147 releases Steps to Perform: Expected Results: 1. Open Tool | Options (Preferences) | Tabs. Make sure all check boxes are unchecked. help litmus tutorial litmus wiki 10% of test cases are automated Testgroup(s) Confirm that all four preferences do not work, since they are disabled: A new window is opened instead of a new tab. You will not be warned when closing multiple tabs. You will not be warned when opening multiple tabs (>15 at once, e.g. the default RSS folder in the toolbar). The tab bar is not shown when only one tab is open. Opening a link in a new tab will open it in the background. 312,502 test executions Subgroup(s) 3.6 Full Functional Tests (FFTs) (137) 6,058 different testers FX 3.6 FFT - Tabbed Browsing (1242) recent test results for testcase id #8410 - all tab options unchecked submission date result id# testcase id#: summary 2010-11-02 02:16:00 336691 8410: All Tab options unchecked 2010-07-07 10:07:17 279135 2010-05-12 11:20:12 branch locale Firefox 3.6 Branch en-US 8410: All Tab options unchecked Firefox 3.6 Branch en-US 257854 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2010-01-17 02:21:29 232877 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2010-01-11 20:56:35 230165 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2010-01-07 10:50:21 228669 8410: All Tab options unchecked Firefox 3.6 Branch en-US 2009-10-22 03:22:31 213485 8410: All Tab options unchecked Firefox 3.6 Branch en-US 9 tested 2,00 builds product platform search test results: recent failures | recently unclear | most common failures | testcases most frequently marked as unclear | testgroup popularity | results with comments search testcases: testcases needed! | recently added | recently changed | by id:
  44. 44. 35,000 - 55,000 executions/release e as many ... but twic ns per day est executio t 6,000 - 9,000 executions/release Testing More Intense
  45. 45. To survive under the time constraints of a rapid release we’ve had to cut the fat and focus on those test areas which are prone to failure, less on ensuring legacy support
  46. 46. 700 - 1,100 different test cases Smaller Scope of Testing 100 - 170 different test cases
  47. 47. a fixed set of tests for areas prone to regression (Flash plugin testing for example) and a dynamically changing set of tests to cover recent regressions we chemspilled for and high risk features
  48. 48. 1,000 - 1,900 testers Smaller Testing Team 8 - 34 testers
  49. 49. The weakest point [in rapid release] is that it’s harder to develop a large community which more accurately represents the scale and variability of the general population. Frequently this means that we don’t hear about issues until after release, in effect turning our release early adopters into beta testers
  50. 50. 1) the core team has remained largely unchanged since adopting rapid release 2) the contract team has nearly doubled ... We can scale up our team much faster through contractors than through hiring. The time afforded to us to make the switch to rapid releases left little room for failure which is why we took that approach.
  51. 51. 0 2 4 OSs per day 1.0 0.5 0.0 Locales per day 6 1.5 8 2.0 Test Priorities Traditional Rapid less locales being tested ... Traditional Rapid ... but more operating systems
  52. 52. we now distribute across Betas. For example, we might test Windows 7, OSX 10.8, and Ubuntu in one Beta then Windows XP, Mac OSX 10.7, and Ubuntu in another Beta
  53. 53. In Other Words ...
  54. 54. Testing has to become more Focused
  55. 55. The greatest strength [of rapid releases] is that the scope of what needs to be tested is narrow so we can focus all of our energy on deep diving into a few areas
  56. 56. Release Engineering http://behrns.files.wordpress.com/2008/03/ikea-car.jpg in-house/3rd party development red uc ed integrate test cyc le build tim e! 49 deployment
  57. 57. MSR '13, Jiang et al. Will My Patch Make It? (And How Fast?) 50
  58. 58. I do hold out hope that Google does come around and works to fix their codebase to get it merged upstream to stop the huge blockage that they have now caused in a large number of embedded Linux hardware companies […] But I need the help of the Google developers to make it happen, without them, nothing can change. Greg Kroah-Hartman 51 http://www.kroah.com/log/linux/android-kernel-problems.html
  59. 59. The Linux Integration Process contributor 1 linux-usb contributor 2 lkml contributor 3 linux-scsi Reviewing subsystem maintainer 1 maintainer Linus Torvalds Linux 3.5 subsystem maintainer 2 Integration 52 Staging
  60. 60. Research Questions How many patches are merged? What kind of patch is merged more likely? What kind of patch is accepted faster?
  61. 61. Case Study Setup contributor 1 linux-usb contributor 2 lkml contributor 3 linux-scsi Reviewing subsystem maintainer 1 maintainer Linus Torvalds Linux 3.5 subsystem maintainer 2 Integration 54 Staging
  62. 62. Case Study Setup contributor 1 linux-usb contributor 2 lkml contributor 3 Linus Torvalds Linux 3.5 linux-scsi Reviewing Integration 54 Staging
  63. 63. Case Study Setup linux-usb lkml Linus Torvalds linux-scsi 54
  64. 64. Case Study Setup linux-usb lkml Linus Torvalds linux-scsi 54
  65. 65. Case Study Setup email1 email patch1 email2 email patch2 email3 ... Linus Torvalds email patch3 54
  66. 66. Case Study Setup email1 email patch1 commit patch1 commit1 email2 email patch2 commit patch2 commit2 email3 email patch3 commit patch3 commit3 ... 54 ...
  67. 67. Case Study Setup checksum1 commit patch1 commit1 email2 email patch2 checksum2 commit patch2 commit2 email3 email patch3 checksum3 commit patch3 commit3 email1 email patch1 ... ... 54 ...
  68. 68. Case Study Setup checksum1 commit patch1 commit1 email2 email patch2 checksum2 commit patch2 commit2 email3 email patch3 checksum3 commit patch3 commit3 email1 email patch1 ... ... 54 ...
  69. 69. Experience Commit 5 Dimensions (29 Patch Metrics) Patch Review 55 Email
  70. 70. Building Decision Trees size: LOC > 50 Is this first patch in thread? not accepted Number of reviewers > 3 ? Number of review messages > 3 ? accepted Decision Tree 56 not accepted
  71. 71. How many patches are merged? What kind of patch is merged more likely? What kind of patch is accepted faster?
  72. 72. 33% of Patches Make it ... accepted/rejected patches 69.26 69.26% 120000 % accepted by linus % rejected by linus 100000 percentage of patches 67.21 67.21% # of patches R E J E C T 66.45 66.45% 66.13 66.13% 67.17 67.17% 80000 72.97% 72.97 60000 40000 33.55% 33.55 71.3 71.3% 32.83 32.83% 20000 71.37 71.73% 0 28.27% 32.79 32.79% 33.87 33.87% 2009 2010 30.74 30.74% 27.03 27.03% 28.7 28.7% 28.63 2005 2006 2007 59 2008 2011 2012 A C C E P T
  73. 73. 80 ... and it takes 1~6 Months! 40 60 within_half_year within_year took_ages 20 Text % accepted patches within_week within_month within_quarter 0 percentage of accepted patches of each year instantly within_hour within_day 2005 2006 2007 60 2008 2009 year 2010 2011 2012
  74. 74. 70 60 within_half_year within_year took_ages 30 40 50 within_week within_month within_quarter 0 10 20 percentage of accepted patches of each year 40 30 20 10 instantly within_hour within_day 2005 2006 2007 2008 2009 2010 2011 0 reviewing time (#days) of each year percentage of accepted patches Reviewing Speeds Up ... 2005 2006 2007 2008 2009 year 2010 2011 2012 2012
  75. 75. 60 60 20 30 40 50 within_day within_week within_half_year within_month within_week within_year within_half_year within_quarter took_ages within_month within_year within_quarter took_ages 10 10 20 30 40 50 percentage of accepted patches of each year instantly within_hour instantly within_day within_hour 0 0 percentage of accepted patches of each year integration time (#days) 70 70 ... while Integration Slows Down 2005 2006 2005 2007 2006 2008 2007 2009 2008 2010 2009 2011 2010 2012 2011 2012
  76. 76. How many patches are merged? 33% patch cceptance a integration becomes slower What kind of patch is merged more likely? What kind of patch is accepted faster?
  77. 77. How many patches are merged? 33% patch cceptance a integration becomes slower What kind of patch is merged more likely? What kind of patch is accepted faster?
  78. 78. Experience Matters Most for Patch Acceptance precision:73% recall:68.47% 65
  79. 79. How many patches are merged? What kind of patch is merged more likely? 33% patch cceptance a y experienced b developers integration becomes slower mature patch specific subsystem What kind of patch is accepted faster?
  80. 80. How many patches are merged? What kind of patch is merged more likely? 33% patch cceptance a y experienced b developers integration becomes slower mature patch specific subsystem What kind of patch is accepted faster?
  81. 81. Experience, Reviewers and Patch Spread Matter for Reviewing Time 68
  82. 82. How many patches are merged? What kind of patch is merged more likely? What kind of patch is accepted faster? 33% patch cceptance a y experienced b developers spread out less integration becomes slower mature patch specific subsystem by more experienced developers
  83. 83. Release Engineering ? 70 Software Quality
  84. 84. Rapid Release vs. Quality? Certificate of High Quality
  85. 85. branch-based integration vs. trunk-based integration 72
  86. 86. 0.01 0.11 1.0 "The Evolution of the Linux Build System" (Adams et al.) Linux 2.6.16.18 1.2 2.2 2.0 How to Evolve Build Systems in a Healthy Way? 2.4 2.6.21.5 2.6.0 73
  87. 87. How to Focus Testing?
  88. 88. How to Keep up with 3rd Party Dependencies? 75 3rd party branch
  89. 89. Where are the Release Engineering Tools? focus your testing efforts test your build! RELEASE IDE keep poking those upstream guys until they give in :-) what did the other team break now ;-) 76
  90. 90. How to Align Release Engineers with the Rest of the Organization?
  91. 91. M C IS Testing has to become more Focused ... but faster crashes bugs are fixed faster On average we deploy new code fifty times a day. same #bugs ... but less bugs are being fixed How many patches are merged? What kind of patch is merged more likely? What kind of patch is accepted faster? 33% patch acceptance by experienced developers less spread out integration becomes slower mature patch specific subsystem by more experienced developers
  92. 92. 1st International Workshop on Release Engineering RELENG 2013 6 practition er talks 84 participants 2 keynotes ... and 10 ademic talks ac http://releng.polymtl.ca May 20, 2013, San Francisco, USA
  93. 93. M C IS Testing has to become more Focused ... but faster crashes bugs are fixed faster On average we deploy new code fifty times a day. same #bugs ... but less bugs are being fixed How many patches are merged? What kind of patch is merged more likely? What kind of patch is accepted faster? 33% patch acceptance by experienced developers less spread out integration becomes slower mature patch specific subsystem by more experienced developers

×