Journal-first presentation given at Foundations of Software Engineering 2018. Based on an article published in Empirical Software Engineering: https://link.springer.com/article/10.1007/s10664-018-9616-7

1. Comparison of release
engineering practices in a large
mature company and a startup
Eero Laukkanen, Maria Paasivaara, Juha Itkonen, Casper Lassenius
eero.laukkanen@solita.fi
+

2. Why is this study important?
Context in software engineering (Dybå et al. 2012)
- Software engineering phenomena are essentially context-specific
- Attempts to produce universal results often lead to hasty generalization
- It is important to link research findings to the context they are found in
Adoption of modern release engineering and context (Laukkanen et al. 2017)
- Few studies on the effect of the context on release engineering practices
- Product context: Few reported successes outside web application domain
- Organization context: Few reported successes in large companies and startups

3. What is modern release engineering?
Release engineering: everything from source code change to being used
Modern release engineering (Adams and McIntosh 2016)
- Fast and frequent release cycles
- Extensive version control
- Build, test and deployment automation
- End-to-end organizational collaboration (DevOps)
- Defect detection during and after production deployments, automated rollbacks

4. What did we study?
Goal: Understand how the organizational context affects release engineering practices in
a startup compared to a large mature company
RQ1. What release engineering practices had the case organizations implemented?
RQ2. What outcomes did the implemented release engineering practices have for the case
organizations?
RQ3. What were the reasons for differences in release engineering practices in the case
organizations?

5. Who were studied?
Theme Case BigCorp Case SmallOrg
Organizational context Product development unit in a large mature
company
Startup company
Company size > 20,000 50
Case organization size 180 50
Distribution of the
organization
Two sites in Europe, one site in Asia One site in North
America
Organizations developed competing products
BigCorp acquired SmallOrg after losing new customers to it

6. What research methods were used?

7. What research methods were used?
18 qualitative interviews with a flexible interview guide, inductive coding
Continuous integration data (2600+ builds during 24 weeks)
Preliminary research in BigCorp (1 background interview, 2 group interviews)
Triangulation by
- multiple researchers
- multiple data collection methods
- interviewees from different roles
- case representative agreement

8. Results

9. What kind of product was developed?
Early market, ambiguous requirements, substantial growth expected
Installed to customer premises, integrated with customer-specific production systems
Customer production environments vary significantly
Sales process starts with proof of concept in the customer environment
During proof of concepts, customer-specific changes are often requested
After full production deployment, most customers do not want very frequent releases (6-12
months is OK)
Some of defects can be only discovered in the customer production environments

10. What kind of organizations were
developing the products?
Theme Case BigCorp Case SmallOrg
Customers Tens One, Friendly
Organizational
structure
Large, Distributed, Functional Small, Co-located,
Collaborative
Resources Sufficient workforce and Test
equipment
Limited workforce and Test
equipment

11. What release engineering practices had
the case organizations implemented?

12. What release engineering practices had
the case organizations implemented?

13. What release engineering practices had
the case organizations implemented?
Metric (mean) Case BigCorp Case SmallOrg
Green builds / week 1.6 (min–max: 0–10) 24.3 (min–max: 8–44)
Build success rate 1.9% (weekly min–max: 0–14%) 76.7% (weekly min–max: 38–97%)
CI execution time 2.7 h (weekly min–max: 0.23–5.0 h) 22 min (weekly min–max: 16–41 min)

14. What release engineering practices had
the case organizations implemented?
Case BigCorp Case SmallOrg
No systematic code review All code is reviewed through pull requests
Slow CI, broken most of the time Fast CI, unbroken most of the time
Strict definition of done, dedicated testers,
integration and non-functional testing in test
labs
No definition of done, no dedicated testers, low unit test
coverage, lack of integration and non-functional testing
Pilots with existing customers performed by
other organization than development, low
feedback from production use
Extensive testing in customer labs and production
environment by domain experts and service team
members, fast feedback to development

15. What outcomes did the release
engineering practices have?
Case BigCorp Case SmallOrg
Low feedback from the production use Frequent feedback from the production use
Low number of defect reports High number of defect reports
Elementary programming mistakes revealed with
new customers
Scalability and other issues with new customers

16. What were the reasons for differences in
release engineering practices?

17. What can be learned from the study?
Adopting modern release engineering practices requires different things in different contexts
In large companies, the adoption can require extensive changes in the organization
In startups, extensive automation can be difficult to achieve due to low resources
Internal verification and frequent customer validation seem to have positive effects on the
customer satisfaction

18. References
Adams, Bram, and Shane McIntosh. "Modern release engineering in a nutshell--why researchers
should care." Software Analysis, Evolution, and Reengineering (SANER), 2016 IEEE 23rd
International Conference on. Vol. 5. IEEE, 2016.
Dybå, Tore, Dag IK Sjøberg, and Daniela S. Cruzes. "What works for whom, where, when, and
why?: on the role of context in empirical software engineering." Proceedings of the ACM-IEEE
international symposium on Empirical software engineering and measurement. ACM, 2012.
Laukkanen, Eero, Juha Itkonen, and Casper Lassenius. "Problems, causes and solutions when
adopting continuous delivery—A systematic literature review." Information and Software Technology
82 (2017): 55-79.

19. +
Eero Laukkanen
eero.laukkanen@solita.fi