Software release engineering is the discipline of integrating, building, testing, packaging and delivering qualitative software releases to the end user. Whereas software used to be released in shrink-wrapped form once per year, modern companies like Intuit, Google and Mozilla only need a couple of days or weeks in between releases, while lean start-ups like IMVU release up to 50 times per day! Shortening the release cycle of a software project requires considerable process and development changes in order to safeguard product quality, yet the scope and nature of such changes are unknown to many. For example, while migrating towards rapid releases allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing.
To understand these implications, we empirically studied the development process of Mozilla Firefox in 2010 and 2011, a period during which the project transitioned to a shorter release cycle. By comparing crash rates, median uptime, and the proportion of post-release bugs before and after the migration, we found that (1) with shorter release cycles, users do not experience significantly more post-release bugs and that (2) less bugs are being fixed, but those that are fixed are fixed faster.
In order to validate these findings, we interviewed Mozilla QA personnel and empirically investigated the changes in software testing effort after the migration towards rapid releases. Analysis of the results of 312,502 execution runs of Mozilla Firefox from 2006 to 2012 (5 major traditional and 9 major rapid releases) showed that in rapid releases testing has a narrower scope that enables deeper investigation of the features and regressions with the highest risk, while traditional releases run the whole test suite. Furthermore, rapid releases make it more difficult to build a large testing community, forcing Mozilla to increase contractor resources in order to sustain testing for rapid releases. In other words, rapid releases are able to bring improvements in software quality, yet require changes to bug triaging, fixing and testing processes to avoid unforeseen costs.
Finally, we also discuss a case study on software integration in the Linux kernel using git. We show the important factors that not only impact patch acceptance but also the time until acceptance, which is essential when optimizing the release engineering process for rapid releases.