Development of software-intensive products and services increasingly occurs by continuously deploying product or service increments, such as new features and enhancements, to customers. Product and service developers need to continuously find out what customers want by direct customer feedback and observation of usage behaviour, rather than indirectly through up-front business analyses. This paper examines the preconditions for setting up an experimentation system for continuous customer experiments. It describes the building blocks required for such a system. An initial model for continuous experimentation is analytically derived from prior work. The model is then matched against empirical case study findings from a startup company and adjusted. Building blocks for a continuous experimentation system and infrastructure are presented. A suitable experimentation system requires at least the ability to release minimum viable products or features with suitable instrumentation, design and manage experiment plans, link experiment results with a product roadmap, and manage a flexible business strategy. The main challenges are proper and rapid design of experiments, advanced instrumentation of software to collect, analyse, and store relevant data, and the integration of experiment results in both the product development cycle and the software development process.

1. www.helsinki.fi/yliopisto
Building Blocks for
Continuous Experimentation
Software Systems Engineering Research Group
Alejandro Sánchez Guinea
with Fabian Fagerholm, Hanna Mäenpää, and Jürgen Münch
03/06/14
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation 1
3 June 2014
RCoSE workshop (collocated with ICSE’14)

2. www.helsinki.fi/yliopisto
• Motivation
• Research Goal
• Research Approach
• Context of Case Study
• Continuous Experimentation Model
• Case Study & Continuous Experimentation Model
• Summary and Future work
03/06/14 2
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Overview

3. www.helsinki.fi/yliopisto
• Increasing number of companies are involved in
developing software-intensive products and services
• digitalization in most industry sectors
• startup software companies
• Challenge: delivering value
• technical issues not the only main concern
• most solutions based on guesswork
• some generic approaches such as Lean Startup
methodology (build-measure-learn cycle)
• Framework for Continuous Experimentation
is widely missing
03/06/14 3
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Motivation

4. www.helsinki.fi/yliopisto 03/06/14 4
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Research Goal
RQ How can Continuous Experimentation with
software-intensive products and services
be organized in a systematic way?
Continuous Experimentation model
Case Study

5. www.helsinki.fi/yliopisto
• Research based on design science research
• Initial continuous experimentation model constructed based
on relevant literature and authors’ experience
• Matching model according to empirical observations and
iterative adjustment of the model
• Case study analysis to ground the model in empirical
observations
• Startup company (3 projects)
• Debriefing session after each project
• project team, customer representatives, and researchers analyze
decisions and actions taken during the project
• Semi-structured interviews with company representatives
• gain insights on lessons learned and perceptions concerning the
adoption of customer feedback for decision making
03/06/14 5
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Research approach

6. www.helsinki.fi/yliopisto
• Small Finnish startup that develops a video calling
solution for the home television set (Tellybean Ltd.)
• Single-product strategy: delivering a life-like plug and play
video calling experience
• Main qualities marketplace: product affordability,
accessibility, and ease of use.
• Three case projects
• Duration: 7 weeks each, # developers: 4 – 7 students
• First two projects same business strategy
• Third project after company had modified strategy and
assumptions
03/06/14 6
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Context of Case Study

7. www.helsinki.fi/yliopisto
• In the Software Factory at the Department of Computer
Science of University of Helsinki
 Experimental Research and Development Laboratory
03/06/14 7
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Context of Case Study

8. www.helsinki.fi/yliopisto
• Continuous cycle of experiments
• Repeated Build-Measure-Learn blocks
• Supported by an infrastructure
03/06/14 8
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Continuous Experimentation Model

9. www.helsinki.fi/yliopisto 03/06/14 9
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Build-measure-learn block
…
Continuous Experimentation Model

10. www.helsinki.fi/yliopisto 03/06/14 10Building Blocks for Continuous for Continuous Experimentation
Build-measure-learn block

11. www.helsinki.fi/yliopisto 03/06/14 11Building Blocks for Continuous for Continuous Experimentation
Build-measure-learn block
Build

12. www.helsinki.fi/yliopisto 03/06/14 12Building Blocks for Continuous for Continuous Experimentation
Build-measure-learn block
Measure

13. www.helsinki.fi/yliopisto 03/06/14 13Building Blocks for Continuous for Continuous Experimentation
Build-measure-learn block
Learn

14. www.helsinki.fi/yliopisto 03/06/14 14
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Infrastructure for
Continuous Experimentation
…
Continuous Experimentation Model

15. www.helsinki.fi/yliopisto 03/06/14 15Building Blocks for Continuous for Continuous Experimentation
Infrastructure for
Continuous Experimentation

16. www.helsinki.fi/yliopisto 03/06/14 16Building Blocks for Continuous for Continuous Experimentation
Infrastructure for
Continuous Experimentation

17. www.helsinki.fi/yliopisto 03/06/14 17Building Blocks for Continuous for Continuous Experimentation
Infrastructure for
Continuous Experimentation

18. www.helsinki.fi/yliopisto 03/06/14 18Building Blocks for Continuous for Continuous Experimentation
Infrastructure for
Continuous Experimentation

19. www.helsinki.fi/yliopisto
Build-Measure-Learn Build-Measure-LearnBuild-Measure-Learn …
03/06/14 19
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
pivot
Project 1 Project 2 Project 3
B Software business
analytics system
M Data related to video calls
to present to operators
L Operators requirements
B Software test tool for
service infrastructure
M Operational capacity of
infrastructure
L Infrastructure limitations
B Prototype for rapid deployments
of software updates
M Continuous deployment
capacity
L How to implement continuous
deployment
x Technology
x Architectural solutions
x Development methodology

20. www.helsinki.fi/yliopisto 03/06/14 20Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1
Vision
Business strategy

21. www.helsinki.fi/yliopisto 03/06/14 21Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1 1 Assumption

22. www.helsinki.fi/yliopisto 03/06/14 22Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1
2 Hypothesis
1

23. www.helsinki.fi/yliopisto 03/06/14 23Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1
3 Design
1
2

24. www.helsinki.fi/yliopisto 03/06/14 24Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1
4
MVP / MVF
1
2
3

25. www.helsinki.fi/yliopisto 03/06/14 25Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1
5
Execute
1
2
3
4

26. www.helsinki.fi/yliopisto 03/06/14 26Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1
6
Analyse
1
2
3
4
5

27. www.helsinki.fi/yliopisto 03/06/14 27Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 1
7
Persevere
1
2
3
4
5 6

28. www.helsinki.fi/yliopisto
Build-Measure-Learn Build-Measure-LearnBuild-Measure-Learn …
03/06/14 28
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
pivot
Project 1 Project 2 Project 3
B Software business
analytics system
M Data related to video calls
to present to operators
L Operators requirements
B Software test tool for
service infrastructure
M Operational capacity of
infrastructure
L Infrastructure limitations
B Prototype for rapid deployments
of software updates
M Continuous deployment
capacity
L How to implement continuous
deployment
x Technology
x Architectural solutions
x Development methodology

29. www.helsinki.fi/yliopisto 03/06/14 29Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
1
2
3
4
5 6
7
Persevere
Pivot
Project 2

30. www.helsinki.fi/yliopisto
Build-Measure-Learn Build-Measure-LearnBuild-Measure-Learn …
03/06/14 30
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
pivot
Project 1 Project 2 Project 3
B Software business
analytics system
M Data related to video calls
to present to operators
L Operators requirements
B Software test tool for
service infrastructure
M Operational capacity of
infrastructure
L Infrastructure limitations
B Prototype for rapid deployments
of software updates
M Continuous deployment
capacity
L How to implement continuous
deployment
x Technology
x Architectural solutions
x Development methodology

31. www.helsinki.fi/yliopisto 03/06/14 31Building Blocks for Continuous for Continuous Experimentation
Case Study &
Continuous Experimentation Model
Project 3
1
2
3
4
5 6
7
Implement
8

32. www.helsinki.fi/yliopisto
• Continuous Experimentation model supported by empirical
evidence
• Continuous cycle of experiments
 Experiments are conducted to validate the most important
assumptions
• Infrastructure for the model considers the roles, technical
infrastructure, and information artifacts for running large-scale
continuous experiments.
• Success factors for a Continuous Experimentation system
• e.g., rapid and adequate design of experiments, proper
instrumentation for collection and analysis of relevant data,
feedback loops must be implemented
03/06/14 32Building Blocks for Continuous for Continuous Experimentation
Summary

33. www.helsinki.fi/yliopisto
• Expand the model based on more cases and in regard to
domain-specific variants
• Answer questions related to specific parts of our model
• how to build a back-end system suitable for large-scale
continuous experimentation?
• how to properly design experiments in order to reduce
uncertainty in strategic assumptions?
• how to ensure that experiments are trustworthy?
• Set up experimental cycle in hardware and software co-
design
• synchronization issues between hardware and software
development schedules
• Experiments conducted with several kinds of stakeholders
(e.g., investors, suppliers, etc.)
03/06/14 33
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Challenges and Research Directions

34. www.helsinki.fi/yliopisto
azsanche@cs.helsinki.fi
• Software Systems Engineering Research Group
http://www.sserg.org
• Software Factory
http://softwarefactory.cc
• Software Factory video
https://www.youtube.com/watch?v=uPAE1uRP65Y
• Twitter
@Juergen_Muench
03/06/14 34
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Contact and Further Information

35. www.helsinki.fi/yliopisto
Fabian Fagerholm, Alejandro Sanchez Guinea, Hanna Mäenpää,
Jürgen Münch. Building Blocks for Continuous Experimentation.
In Proceedings of the 1st International Workshop on Rapid
Continuous Software Engineering (RCoSE 2014), Hyderabad,
India, 2014.
The article can be found online in:
http://www.sserg.org/publications/uploads/fedf90b60c9a480f
dff403f127405a248d04ea55.pdf
03/06/14 35
Department of Computer Science
Alejandro Sánchez Guinea
Building Blocks for Continuous for Continuous Experimentation
Reference of the article