This document discusses a case study of a software organization that was attempting to adopt continuous delivery (CD) practices while using a stage-gate process for managing releases. The study identified several direct signs of dysfunctional CD, including failing builds, flaky tests, low test coverage, and slow feedback. Through workshops, the researchers analyzed the root causes of these issues, many of which related to lack of time due to the stage-gate process and tight schedules. The stage-gate process affected CD adoption by creating branches, time pressure, and limiting the ability to improve practices like testing. Guidelines are proposed based on providing more time and flexibility to support better CD.

1. Bottom-up Adoption of Continuous Delivery
in a Stage-Gate Managed Software
Organization
Eero Laukkanen, Timo O.A. Lehtinen, Juha Itkonen, Maria Paasivaara,
Casper Lassenius
ACM/IEEE International Symposium on Empirical Software Engineer-
ing and Measurement, September 2016
Department of Computer Science, Aalto University

2. Continuous Delivery (CD)
Continuous Delivery
Continuous Integration
Version
Control
Commit
Stage
Acceptance
Test Stage
Other
Stages
Deploy to
Production
"Continuous Delivery is a software development discipline
where you build software in such a way that the software can
be released to production at any time." (Fowler, 2013)
1

3. CD Adoption
Organization, has
not adopted CD
Organization, has
adopted CD
Continuous delivery adoption:
a set of adoption actions
Problems
CD solves
Problems
of CD
Benefits of
CD
has
CD adoption
problems
has
2

4. Previous Studies of CD Adoption Problems
Debbiche
etal.
Claps
etal.
Leppänen
etal.
Chen
Gm
eineretal.
Laukkanen
etal.
Hukkanen
Identified Problems
Build Design X
System Design X X X X X
Integration X X X X
Testing X X X X X
Release X X
Human&Organizational X X X X X X X
Process X
Resources X X X X X X
3

5. Case Organization
• Nokia Networks software development unit
• Managed by product management unit
• Telecommunications product for managing self-organizing
network infrastructure
• Customers: mobile network operators around the world
• Org. structure: distributed to several sites and countries
• Time difference to one site, other sites on same timezone
"Internally, the target would be continuous deployment,
deploying to target hardware continuously [...] For continuous
deployment, the management is not needed or changes in the
processes."
4

6. Stage-gate Process of the Case Organization
Customer
Trial
Planning Development
System
Verification
General
Availability
Code Freeze
G3 G4G2G1
• Releases go through the stages, gates are milestones
• Planning: feature specification, resource allocation
• Development: implementation, testing (Scrum)
• System Verification: end-to-end system testing
• Customer Trial: trials at customer environments
• General Availability: release to all customers
5

7. Research Goal and Questions
Goal: Understand how the stage-gate process affected CD
adoption in the case organization.
RQ1. What direct signs of a dysfunctional CD practice did the
case organization show?
RQ2. What caused the direct signs of a dysfunctional CD
practice in the case organization?
RQ3. How did the stage-gate process used by the case
organization explain the direct signs of a dysfunctional CD
practice?
6

8. Research Process
• ARCA root cause analysis method (Lehtinen et al., 2011)
• Designed for high-level causal analysis
• Two two-hour workshops on one site of the organization
1. Problem identification: What CD adoption problems the
organization is facing? Which are the most significant?
2. Causal analysis: Why are the identified issues problematic?
• Altogether 15 participants: managers, developers, testers
• Audio recordings transcribed and thematically analyzed
• Limitations
• Perceived causes
• Only one site heard
• Only development unit heard
7

9. Results

10. RQ1. Direct Signs of Dysfunctional CD Practice
Sign Description
Failing builds CD pipeline builds were often failing and not
fixed immediately afterwards.
Flaky tests Some tests were flaky, meaning that they
might fail randomly even when there was no
issue in a code change.
Low test coverage Test coverage was not considered high enough
for having confidence to release the product
after running the tests.
Slow feedback Feedback about the changes made to the
product came slow to the developers.
8

11. RQ2. Mechanisms and Root Causes
Legend
Stage-gate
Process
Failing
Builds
Flaky Tests
Slow
Feedback
Low Test
Coverage
Multiple
Branches
Delayed
Integration
Time
Pressure
Process
Overhead
Tight
Schedule
Lack of
Time To
Improve
Branch
Overhead
Distributed
Organization
Insufficient
Communication
Lack of Testing
Strategy
Duplicate
Testing
Unsuitable
Architecture
Unstable
Architecture
Limited
Hardware
Resources
Sharing Test
Environments
Slow Build
Breaking
Changes
Direct Sign
Other
Mechanism
Process
Mechanism
Root Cause
9

12. RQ3. Lack of Time Mechanisms
Stage-gate
Process
Failing
Builds
Flaky Tests
Low Test
Coverage
Multiple
Branches
Time
Pressure
Process
Overhead
Tight
Schedule
Lack of
Time To
Improve
Branch
Overhead
10

13. RQ3. Tight Schedule Mechanism
Stage-gate
Process
Tight
Schedule
Early Plans Do not Hold
Same Deadlines for Dependencies
and Dependents
Uneven and Unpredictable Workload
Plans not Readjusted after Delays
Problems Hidden to Pass Quality
Gates
Code Freeze not Respected
11

14. RQ3. Branch Mechanisms
Stage-gate
Process
Failing
Builds
Flaky Tests
Slow
Feedback
Low Test
Coverage
Multiple
Branches
Delayed
Integration
Lack of
Time To
Improve
Branch
Overhead
Limited
Hardware
Resources
Sharing Test
Environments
12

15. Guidelines Based on the Case Study
• No time to improve test coverage, fix builds and flaky tests
→ Provide time with slack (Lawson, 2001)
• Branches decided by management caused overhead for
development
→ Avoid branches with feature toggles (Rahman et al., 2016)
→ Give decision-making power to developers
• Stage-gate process substantially affected the CD adoption
→ CD and stage-gate process might be incompatible
→ Get support from other parts of the organization for CD
adoption
13

16. References
L. Chen. Continuous Delivery: Huge Benefits, But Challenges Too. Software, IEEE, 32(2):50--54, 2015.
G. G. Claps, R. B. Svensson, and A. Aurum. On the journey to continuous deployment: Technical and social
challenges along the way. Information and Software Technology, 57(0):21 -- 31, 2015.
A. Debbiche, M. Dienér, and R. Berntsson Svensson. Challenges When Adopting Continuous Integration: A Case
Study. In Product-Focused Software Process Improvement, volume 8892 of Lecture Notes in Computer Science,
pages 17--32. Springer International Publishing, 2014.
M. Fowler. ContinuousDelivery, 2013. URL http://martinfowler.com/bliki/ContinuousDelivery.html.
J. Gmeiner, R. Ramler, and J. Haslinger. Automated testing in the continuous delivery pipeline: A case study of an
online company. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation
Workshops (ICSTW), pages 1--6, 2015.
L. Hukkanen. Adopting Continuous Integration – A Case Study. M.Sc. thesis, Aalto University, 2015.
E. Laukkanen, M. Paasivaara, and T. Arvonen. Stakeholder Perceptions of the Adoption of Continuous Integration – A
Case Study. In 2015 Agile Conference, pages 11--20, Washington, DC, USA, 2015.
M. B. Lawson. In praise of slack: Time is of the essence. The Academy of Management Executive, 15(3):125--135, 2001.
T. O. A. Lehtinen, M. V. Mäntylä, and J. Vanhanen. Development and evaluation of a lightweight root cause analysis
method (ARCA method)–field studies at four software companies. Information and Software Technology, 53(10):
1045--1061, 2011.
M. Leppänen, S. Mäkinen, M. Pagels, V.-P. Eloranta, J. Itkonen, M. V. Mäntylä, and T. Männistö. The Highways and
Country Roads to Continuous Deployment. Software, IEEE, 32(2):64--72, 2015.
M. T. Rahman, L.-P. Querel, P. C. Rigby, and B. Adams. Feature toggles: practitioner practices and a case study. In
Proceedings of the 13th International Conference on Mining Software Repositories, pages 201--211. ACM Press,
2016.
14