On Software Release Engineering (Bram Adams)

2,872 views

Published on

ICSE 2012 technical briefing "On Software Release Engineering" by Bram Adams http://mcis.polymtl.ca/~bram/publications/2012/icse_tb.pdf

Published in: Technology, News & Politics
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,872
On SlideShare
0
From Embeds
0
Number of Embeds
1,165
Actions
Shares
0
Downloads
46
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

On Software Release Engineering (Bram Adams)

  1. 1. On Software Release Engineering Bram Adams M C ISLatest version: http://mcis.polymtl.ca/~bram/publications/2012/icse_tb.pdf
  2. 2. M (Lab on Maintenance,C IS Construction and Intelligence of Software)
  3. 3. On average wedeploy new code fifty times a day. MC IS
  4. 4. Continuous Delivery 15k tests CVS test staging/productioncontinuous 9 min.integration 6 http://goo.gl/qPT6
  5. 5. Continuous Delivery 15k tests CVS test staging/productioncontinuous 9 min.integration 6 http://goo.gl/qPT6
  6. 6. Continuous Delivery 15k tests CVS test staging/productioncontinuous 9 min. 6 min.integration 6 http://goo.gl/qPT6
  7. 7. Work fast and don’t be afraid to break things.http://goo.gl/UlCW
  8. 8. Build a little and then test it. Build some more and test some more.James Whittaker
  9. 9. Even Desktop Apps Release More Frequently40 #releases http://en.wikipedia.org/wiki/History_of_Firefox32 ?2416 8 0 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 9
  10. 10. http://hacks.mozilla.org/wp-content/uploads/2012/05/firefox-releases.jpgFirefox New ReleaseSchedule (1)
  11. 11. http://hacks.mozilla.org/wp-content/uploads/2012/05/firefox-releases.jpgFirefox New ReleaseSchedule (2)
  12. 12. http://hacks.mozilla.org/wp-content/uploads/2012/05/rapid-release.jpg Release Pipeline
  13. 13. Firefox Release Howto (1) 85+ configuration tag buildrepositories sanity check everything
  14. 14. Firefox Release Howto (2) generate incremental updates for each supported old version i18n signing
  15. 15. http://tims-ideas.blogspot.com/2011/05/live-blog-from-icse-view-of-icses.html ... yet Software Systems keep on Growing! <150k tests/ >50k build >5k developers commit s/daybuild cache with >2k projects >50M tests/day>50TB memory 20 code compilation ges/min. in the cloud chan 15
  16. 16. Release Engineeringhttp://behrns.files.wordpress.com/2008/03/ikea-car.jpgin-house/3rd party integrate development red test build uc ec ycl et im e! 16 deployment
  17. 17. Out of Scope (but Highly Related) release variability release planning man agement management app stores ... software process i mpact on packaging project technologies software ma nagement product lines 17
  18. 18. Release Engineeringhttp://behrns.files.wordpress.com/2008/03/ikea-car.jpgin-house/3rd party integrate development red test build uc ec ycl et im e! 18 deployment
  19. 19. The Build Process 19
  20. 20. The Build Process ?http://behrns.files.wordpress.com/2008/03/ikea-car.jpg 20
  21. 21. Step 1 - Configuration AutotoolsFeatures Tools 21
  22. 22. Step 2 - Construction Make Prescriptions .c .o= Dependencies .o .o .o .exe 22
  23. 23. +Source Code Build System 23
  24. 24. 24
  25. 25. Our record so far is a pr we inherited oject with an Ant weighing in a script t 10,000 line XML. Needl s of ess to say, th project requ is ired an entir team devote e d to keeping build workin the g—a comple waste of r teKDE 4 is leaving the aging esources. [Jez Humble"autotool" build chain behind. & David Farl ey]Some developers, not only inKDE, like to nickname theautotools as "auto-hell" Build Systemsbecause of its difficulty tocomprehend architecture.[http://lwn.net/Articles/188693/] 25 are Complex
  26. 26. >5 MLOC & ~20 Years of History"Design Recovery and Maintenance of Build Systems" (Adams et al.)"The Evolution of the Linux Build System" (Adams et al.) 26
  27. 27. Quake 3 client UI server game logic client renderer quake3.exe network d(a)emon27
  28. 28. Quake 3 conditional compilation original game expansion pack28
  29. 29. "The Evolution of the Linux Build System" (Adams et al.) Linux 2.6.16.18 0.01 29
  30. 30. 0.01 0.11 1.0"The Evolution of the Linux Build System" (Adams et al.) Linux 2.6.16.18 2.0 2.2 1.22.4 2.6.0 2.6.21.5 29
  31. 31. 0.01 0.11 1.0"The Evolution of the Linux Build System" (Adams et al.) Linux 2.6.16.18 2.0 2.2 1.2 Build Systems Grow in Complexity2.4 2.6.0 2.6.21.5 29
  32. 32. Build Systems Require 12% of a Developer’s Time (on average) Build maintenance slows down development!Software in the DOE: The Hidden Overhead of the “Build” (Kumfert et al.) 30
  33. 33. An Empirical Study of Build Maintenance Effort (McIntosh et al.)>36 MLOC & ~120 Years of History 31
  34. 34. An Empirical Study of Build Maintenance Effort (McIntosh et al.) Build Files Change Relatively More than Source Code Files 54.5 #changed source lines/source file 4 #changed build lines/build file3.5 32.5 21.5 10.5 0 ArgoUML Hibernate Eclipse GCC Git Linux Mozilla PLPlot PostgreSQL 32
  35. 35. The Build System Requires Significant Maintenance Low coupling thanks 44% to higher-level build Source Build Test Build abstraction 27% 20% 16%12% 8% 4% 33
  36. 36. Our Build Systems need HELP 34
  37. 37. 35
  38. 38. A B C D 35
  39. 39. A D B C 36
  40. 40. all files/pro stat ic build ducts stored in cdepe ndencies loud A D B C tr ansparent access via FUSE 36
  41. 41. Build System Quality SYMake (Tamrawi et al.)
  42. 42. Build System Quality Clipper (Morgenthaler et al.)Tu et al., Nadi et al., Berger et al., Tartler et al., ...
  43. 43. Release Engineeringhttp://behrns.files.wordpress.com/2008/03/ikea-car.jpgin-house/3rd party integrate development red test build uc ec ycl et im e! 38 deployment
  44. 44. The Integration Process 39
  45. 45. 1. The Paradox of Early Integration 40
  46. 46. 1. The Paradox of Early Integration release branch 40
  47. 47. 1. The Paradox of Early Integration release branch 40
  48. 48. 1. The Paradox of Early Integration feature branches 40
  49. 49. 1. The Paradox of Early Integration 40
  50. 50. Cohesive and Isolated Development with Branches (Barr et al.)Integrations Interrupt Development once in every 24.4 Commits in DVC changed files: A, B, C, D and E changed files: A, E and F distraction |{A, B, C, D, E}| ∩ |{A, E, F}| = = 66.6% on merge |{A, E, F}| assumption: Pr[real integration conflict] = 90% 41
  51. 51. Proactive Detection of Collaboration Conflicts (Brun et al.) ... and when they Do, Integrations cause Troubleon average 16% of on average all merges had on average 5% 11.7% of all textual conflicts of all merges had merges had test(manual resolution build issues issues needed) 42
  52. 52. The Effect of Branching Strategies on Software Quality (Shihab et al.) Branching DegradesQuality of Components branch family file branched/merged 43
  53. 53. The Effect of Branching Strategies on Software Quality (Shihab et al.) Branching DegradesQuality of Components branch family file branched/merged 43
  54. 54. The Effect of Branching Strategies on Software Quality (Shihab et al.) Branching DegradesQuality of Components branch family file branched/merged 43
  55. 55. Proactive Detection of Collaboration Conflicts (Brun et al.)Speculative Merge Prevention B T S AME A HEAD B EHIND T EXTUAL8 B UILD8 T EST8 T ESTX 44
  56. 56. A Cost-Benefit Approach to Recommending Conflict Resolution for Parallel Software Development (Niu et al.)Conflict Resolution Recommendation method of interest how hard to resolve conflict? (a) Using cost-benefit ratio to rank the conflicting proceduresranking of methods to how much benefit does resolve conflicts with 45 resolving conflict yield?
  57. 57. 2. The Nightmare of 3rd Party Integration 46 vendor branch
  58. 58. 47
  59. 59. Integrators are Gatekeepers end users47
  60. 60. Integrators are Gatekeepersmaintainers end users 47
  61. 61. >180k Packages & ~28 Years of History 48
  62. 62. 10 20 30 40 50 60 %package versions 0 A B C D E FNew Package Local PatchUpstream Sync Product-wide Dependency Concern Build Change Management 49
  63. 63. Mozilla Delivers New Necessary? Version of Firefox – First Web Browser to Buggy?Upstream Sync Support Do Not Track on Secure? Multiple Platforms! [Mozilla Blog] 50
  64. 64. Risk Analysis & Cherry-picking Upstream Sync UNIX diff maintainer 51
  65. 65. 10 20 30 40 50 60 %package versions 0 A B C D E FNew Package Local PatchUpstream Sync Product-wide Dependency Concern Build Change Management 52
  66. 66. Library Transitions Ripplethrough the whole SystemEmpir Software Eng (2009) 14:262–285 279 Dependency Management xcontrib xfree86–common xlib6g debianutils gconv–modules libgtk1.2 libglib1.2 libc6 ldso libstdc++2.10 mozilla libjpeg62 libpng2 zlib1g libz1 libnspr4 Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of theFig. 7 Most popular instance evolution: a case study for large software compilation (Gonzalez-Barahona et al.)two abstract dependencies have only one child
  67. 67. Library Transitions Ripplethrough the whole SystemEmpir Software Eng (2009) 14:262–285 279 Dependency Management xcontrib xfree86–common xlib6g debianutils gconv–modules libgtk1.2 libglib1.2 libc6 ldso libstdc++2.10 mozilla libjpeg62 libpng2 zlib1g libz1 libnspr4 Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of theFig. 7 Most popular instance evolution: a case study for large software compilation (Gonzalez-Barahona et al.)two abstract dependencies have only one child
  68. 68. Library Transitions Ripplethrough the whole SystemEmpir Software Eng (2009) 14:262–285 279 Dependency Management xcontrib xfree86–common xlib6g debianutils gconv–modules libgtk1.2 libglib1.2 libc6 ldso libstdc++2.10 mozilla libjpeg62 libpng2 zlib1g libz1 libnspr4 Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of theFig. 7 Most popular instance evolution: a case study for large software compilation (Gonzalez-Barahona et al.)two abstract dependencies have only one child
  69. 69. Library Transitions Ripplethrough the whole SystemEmpir Software Eng (2009) 14:262–285 279 Dependency Management xcontrib xfree86–common xlib6g debianutils gconv–modules libgtk1.2 libglib1.2 libc6 ldso libstdc++2.10 mozilla libjpeg62 libpng2 zlib1g libz1 libnspr4 Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of theFig. 7 Most popular instance evolution: a case study for large software compilation (Gonzalez-Barahona et al.)two abstract dependencies have only one child
  70. 70. 2-way Impact Analysis Dependency Management Did Software Eng (2009) 14:262–285 Empir 279someone Did I breakbreak my someonespackage? xcontrib xlib6g xfree86–common debianutils package? gconv–modules libgtk1.2 libglib1.2 libc6 ldso libstdc++2.10 mozilla libjpeg62 libpng2 zlib1g libz1 libnspr4 Fig. 7 Most popular instance of the Inter-Dependency Graph for mozilla in Debian 2.2. Each of the two abstract dependencies have only one child 54
  71. 71. If an engineer comes up with a better way of doing something, he is tasked with refactoring all existing libraries and assisting dependent projects to migrate to the new librariesJames Whittaker
  72. 72. 10 20 30 40 50 60 %package versions 0 A B C D E FNew Package Local PatchUpstream Sync Product-wide Dependency Concern Build Change Management 56
  73. 73. patchupstream accepted Upstream Upstream Upstream Sync Sync Syncmaintainer Local Patch Local Patch 57
  74. 74. Package : opensslVulnerability : predictable random number generatorProblem type : remoteDebian-specific: yes Local PatchCVE Id(s) : CVE-2008-0166Date : 2008-05-13Luciano Bello discovered that the random number generator in Debiansopenssl package is predictable. This is caused by an incorrectDebian-specific change to the openssl package (CVE-2008-0166). As aresult, cryptographic key material may be guessable.It is strongly >=1.5 years 8-( recommended that all cryptographic key material whichhas been generated by OpenSSL versions starting with 0.9.8c-1 onDebian systems is recreated from scratch. Furthermore, all DSA keysever used on affected Debian systems for signing or authenticationpurposes should be considered compromised; the Digital SignatureAlgorithm relies on a secret random value used during signaturegeneration. >=44 distributions ;-)The first vulnerable version, 0.9.8c-1, was uploaded to the unstabledistribution on 2006-09-17, and has since propagated to the testingand current stable (etch) distributions. The old stable distribution(sarge) is not affected. 58
  75. 75. Release Engineeringhttp://behrns.files.wordpress.com/2008/03/ikea-car.jpgin-house/3rd party integrate development red test build uc ec ycl et im e! 59 deployment
  76. 76. Incremental Testing
  77. 77. Different Test Phases (selection of) unit testsintegration + build all unit tests acceptance/integration tests capacity tests user acceptance testsrelease smoke teststest phase time to run (hours)
  78. 78. Seamless Upgrades (1): Blue/Green Deployment old version of systemusers DB Disk router DB Disk
  79. 79. Seamless Upgrades (1): Blue/Green Deployment old version of systemusers DB Disk router DB Disk new version of system
  80. 80. Seamless Upgrades (1): Blue/Green Deployment old version of systemusers DB Disk router DB Disk new version of system
  81. 81. Seamless Upgrades (1): Blue/Green Deployment old version of systemusers DB Disk router DB Disk new version rollback!! of system
  82. 82. Seamless Upgrades (1): Blue/Green Deployment old version of systemusers DB Disk router DB Disk new version rollback!! of system
  83. 83. Seamless Upgrades (2): Canary Deployment DB Disk users old version of system router DB Disk old version of system DB Disk old version of system
  84. 84. Seamless Upgrades (2): Canary Deployment DB Disk users old version new version of system router DB Disk old version of system DB Disk old version of system
  85. 85. Seamless Upgrades (2): Canary Deployment DB Disk users old version new version of system router DB Disk oewveersion nld v rsion o system off system DB Disk old version of system
  86. 86. Seamless Upgrades (2): Canary Deployment DB Disk users old version new version of system router DB Disk oewveersion nld v rsion o system off system DB Disk oewveersion nld v rsion o system off system
  87. 87. Testing Challenge 1: Parallelization
  88. 88. Testing Challenge 2:Incrementalizing Tests
  89. 89. Testing Challenge 3:Optimizing Test Speed DB Disk
  90. 90. Testing Challenge 4: Data Migration
  91. 91. Testing Challenge 5: Process Integration
  92. 92. Open Challenges forRelease Engineering 69
  93. 93. ? Certificate = of High Quality rapid releaseDo Faster Releases Improve Software Quality? - An Empirical Case Study ofMozilla Firefox (Khomh et al.)
  94. 94. Traditional Release Cycle Rapid Release Cycle 6.0 8.01.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 7.0 9.0
  95. 95. Rapid Release Cycle 6.0 8.03.6 4.0 5.0 7.0 9.0 Certificate vs. of High Quality release&cycle&length Do Faster Releases Improve Software Quality? - An Empirical Case Study of Mozilla Firefox (Khomh et al.)
  96. 96. Same # Post-release Bugs 13 20 5Number of Post Release Bugs Per Day 4 3 2 1.7 1.6 1 0 Traditional Release (TR) Rapid Release (RR)
  97. 97. Crashes Pop Up Earlier 1200 1080 960Median UpTime in Seconds 840 720 643 600 480 370 360 240 120 0 Traditional Release (TR) Rapid Release (RR)
  98. 98. Rapid Release Cycle 6.0 8.03.6 4.0 5.0 7.0 9.0 vs. bug&fixing release&cycle&length Do Faster Releases Improve Software Quality? - An Empirical Case Study of Mozilla Firefox (Khomh et al.)
  99. 99. Bugs are Fixed Faster 654 159 100 90 80 70Bug Age in Days 60 50 40 30 20 16 10 6 0 Traditional Release (TR) Rapid Release (RR)
  100. 100. Proportionally Less Bugs Fixed 100% 90% 80% 70% 67% 60% 59%% of Bugs Fixed 50% 43% 40% 37% 30% 20% 10% 0% Traditional Rapid Release Traditional Rapid Release Release (TR) - (RR) - Main Release (TR) - (RR) - Beta Main Beta
  101. 101. What about Other Projects? Certificate of High Quality
  102. 102. What about Other Projects? Certificate of High Quality
  103. 103. How to Make Softwareand Build Comprehension Pragmatic? 40 79
  104. 104. Where are the Tools? refactor test youryour makefiles! build! RELEASE IDE what did the keep poking those other team break upstream guys until they now ;-) give in :-) 80
  105. 105. Work fast and don’t be afraid to break things. We need more canaries!http://goo.gl/UlCW
  106. 106. On average wedeploy new code fifty times a day. MC IS
  107. 107. On average wedeploy new code fifty times a day. MC IS
  108. 108. Release Engineering On average we deploy new code fifty times a day.http://behrns.files.wordpress.com/2008/03/ikea-car.jpgin-house/3rd party integrate M development C ISkee pi n test build gc ycl et im ed ow n 7 deployment
  109. 109. Release Engineering On average wedeploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg a day. in-house/3rd party integrate development kee pi n test build gc ycl et im M edC IS ow n 7 deployment
  110. 110. Release Engineering The Build System Requires On average we deploy new code fifty times Significant Maintenance http://behrns.files.wordpress.com/2008/03/ikea-car.jpg a day. in-house/3rd party integrate development kee pi n test build gc ycl et im M 44% ed C IS ow n Source Build 7 deployment Test Build 27% 20% 16%12% 8% 4% 21
  111. 111. Release Engineering On average we deploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg a day. in-house/3rd party integrate development kee pi n test build gc ycl et im M ed C IS ow n 7 deployment The Build System Requires Significant Maintenance 44% Source Build Test Build 27% 20% 16%12% 8% 4% 21
  112. 112. Release Engineering Risk Analysis & Cherry-picking On average we deploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg a day. in-house/3rd party integrate development kee pi n test build gc Upstream Sync ycl et im M ed C IS ow n 7 deployment The Build System Requires diff Significant Maintenance 44% Source Build Test Build 27%12% 16% 20% maintainer 8% 4% 21
  113. 113. Release Engineering On average we deploy new code fifty times http://behrns.files.wordpress.com/2008/03/ikea-car.jpg a day. in-house/3rd party integrate development kee pi n test build gc ycl et im M ed C IS ow n 7 deployment The Build System Requires Risk Analysis & Cherry-picking Significant Maintenance Upstream Sync 44% Source Build Test Build 27% diff 20% 16%12% 8% 4% maintainer 21
  114. 114. M C IS http://mcis.polymtl.ca/http://google-engtools.blogspot.ca/

×