Transitioning Towards Continuous Experimentation in a Large Software Product and Service Development Organisation – A Case Study
•Download as PPTX, PDF•
1 like•526 views
Companies need capabilities to evaluate the customer value of software-intensive products and services. One way of systematically acquiring data on customer value is running continuous experiments as part of the overall development process. This presentation describes the first steps of transitioning towards continuous experimentation in a large company, including the challenges faced.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Transitioning Towards Continuous Experimentation in a Large Software Product and Service Development Organisation – A Case Study
- 1. Sezin Gizem Yaman1, Fabian Fagerholm1, Myriam Munezero1, Jürgen Münch1,2, Mika Aaltola3, Christina Palmu3, Tomi Männistö1 1 2 3 Transitioning Towards Continuous Experimentation in a Large Software Product and Service Development Organisation – A Case Study
- 2. Continuous experimentation is a software development approach where research and development activities are driven by constantly conducting experiments with product value*. * Rissanen, O., Münch, J.: Continuous Experimentation in the B2B Domain: A Case Study. * Fagerholm,F.,Guinea,A.S.,Mäenpää,H., Münch, J.: The RIGHT model for Continuous Experimentation.
- 3. HOWEVER, guidence is lacking on how development teams in large organisations with complex business partnership networds can transition towards continious experimentaiton.
- 4. As our aim was to observe the introduction of continuous experimentation in a company, we conducted an actual experiment round with a real product.
- 5. Transition Small teams Small-scale experiments Experimentation Target CollaborationInitial circumstances The experiment Persistence
- 6. Initial Circumstances Multiple stakeholders - Select a focus Evolving product - Allocate few resources Need to limit the risks - Start small
- 7. Small teams Small-scale experiments Champions Brainstorming on possible experiments Simplistic UI experiment as a start [It’s better to] start experimenting with something small. [...] It’s more important to start now. Practice will make it perfect.” (Technical coach)
- 8. Identifying an experimentation target Utilise existing resources Numerous discussions Prioritize and analyse the feature
- 9. Designing and executing the experiment Pilot run Rerun the experiment Expert advice Emphasize “learning” “Fix the experiment [the] best way you can and run it again. You can learn so much with each experiment.” (Technical coach)
- 10. Collaborating with experts Avoid mistakes, biases Faster start Iterative learning
- 11. Persistence Learn from the failures Keep practising Scale up “One should not dwell on temporary failures, but keep practicing.” (The teams)
- 12. Transition Small teams Small-scale experiments Experimentation Target Collaboration Initial circumstances The experiment Persistence
- 13. “Experimentation made it clear to the teams that there is no need to debate between opinions and assumptions as you can quickly test them with an experiment.” (Technical coach)
- 14. Reference Sezin Gizem Yaman, Fabian Fagerholm, Myriam Munzero, Jürgen Münch, Mika Aaltola, Christina Palmu, Tomi Männistö. Transitioning Towards Continuous Experimentation in a Large Software Product and Service Development Organization – A Case Study. In Proceedings of the 17th International Conference on Product- Focused Software Process Improvement (PROFES 2016), LNCS. Springer-Verlag, 2016. Get the article here: http://bit.ly/corporateexperimentation
- 15. sezin.yaman@helsinki.fi www.cs.helsinki.fi/en/people/yaman www.cs.helsinki.fi/group/ese/ Sezin Yaman University of Helsinki Empirical Software Engineering Helsinki Research Group Discussion: What other ways a software company can transition towards continuous experimentation?
Editor's Notes
- Good afternoon everyone, My name is Sezin Gizem Yaman and I come from the University of Helsinki. I’m a PhD student in Computer Science Department. This study has been done together with Reutlingen University and Ericsson Finland.
- To start with the approach, here is one definition given by Rissanen and Munch.. In other words, with CE customers or users can be involved in the decision making process as experiment subjects and decisions can be made based on empirical data rather than opinions. CE is a novel app and there has been increasing attention
- It is unclear how companies can start adopting the approach… especially with....complex.. and therefore, this study investigates how CE can be introduced. In order to address the RQ, we conducted a single case study in which we observed and participated in the introduction of continuous experimentation in a large company.
- Just to remind that Rather than focusing on accomplishing an ambitious experimentation design or reaching the scientific truth I am going to present you how we did initiate this with a company.
- In the process of planning and analyzing the experiment, a number of observations and inferences were made regarding the transition towards continuous experimentation both from practitioners’ and researchers’ points of view. Based on our analysis, these are the themes observed when transitioning towards CE. We will go through them each with respect to challenges faced under each, and how did we mitigate those challenges.
- We observed an initial interest towards CE among company representatives, but also challenges and concerns. One of the challenges was due to the structure. At the time of the study, they were in the middle of reimplementing a product… the product had operators... Selected the focus to be on the portal being developed for operators. Portal service for telecommunication purposes. Secondly, because the transitioning process started in the middle of development with an evolving product with existing deadlines. We used what was available with respect to time and effort Lastly, In order to limit the risks, and limit the scope of the exp
- We observed that beginning with small teams who are interested in the process did facilitate the introduction of a new way of working in the company. One development team (4 developers), one UX team (1.5 persons) A technical coach and a UX designer was very active and they took the lead during the whole study, we call them champions. We held many brainstorming sessions with champions. In those it emerged that champions were hesitant to go ask end users to join to exp, they wanted to LEARN first And, decided to go with a UI exp because the uncertainties about it needed to be determined sooner. Less risky start with small teams less assertive start. And more important to start now, than over planning or trying to test the most important assumptions with inexperience
- We observed that it was not straightforward to identify an experimentation target. During the brainstorming sessions and also in separate meetings, we as the researchers, worked on the existing metarial such as user stories, previously done mockups, bg metarials etc. Assumptions were identified related to the chosen product feature due to initial circumstances, and prioritized as the experiment target. Experiment design was drafted ITERATIVELY which included the assumptions and the hypothesis. Secondly, here we realized that operators were not reachable. Therefore we decided to conduct the experiment with internet users. IT would be suffiennt bc it was an UI experiment.
- Regarding the experiment itself, the case company did a pilot run, But in actual first run they realized that there were some problems with the exp, instructions were not clear. Evaluation was problematic. then run it, then, updated the experiment design and rerun it again. During this stage, lack of experience was observed as a challenge, but.. As the technical coach put it… They were aware that this was a learning process.
- It was beneficial to have expert facilitators guiding the transition and providing support and guidance when needed . 3 months as whole. Some mistakes were avoided through expert opinion. For instance, during the execution stage, guidance was provided on how to achieve more valid results and avoid introducing bias during the experiment As a result, a faster start was enabled and approach was learned iteratively.
- Continuous experimentation may be easy to understand in principle, but actually starting it in a real, large B2B organisation required persistence. The final experiment design was reached after a number of attempts. Similarly, final results were reached after piloting and updating the the design. The teams indicated that when starting, one should not dwell on temporary failures, but keep practising. Important to have the champions. And when they reached to a more mature level, they would like to scale the approach to cover larger portion of the organisation
- In our study, we identified these stages in towards adoptin CE. Initial circ. Caused us to start with small teams and a small scalte experiment. Experimentation target was chosen and experiment was designed and run based on the collaboration with the researchers. And persistence was observed to be required in order to initiate adopting a new approach as CE. This was identified after 3 months of collaboration whereas exp. Itself took only some hours.
- In conclusion, learning was the most important thing. But also… quote They made a product development decision based on empirical data. As future work, We are interested in observing Next experimentation cycles, how to sustain the process and make it continuous, and scale It over.
- We observed that it was not straightforward to identify an experimentation target. During the brainstorming sessions and also in separate meetings, we as the researchers, worked on the existing metarial such as user stories, previously done mockups, bg metarials etc. Assumptions were identified related to the chosen product feature due to initial circumstances, and prioritized as the experiment target. Experiment design was drafted ITERATIVELY which included the assumptions and the hypothesis. Secondly, here we realized that operators were not reachable. Therefore we decided to conduct the experiment with internet users. IT would be suffiennt bc it was an UI experiment.
- This was one potantial way of transitioning as a result of our study. What other ways a software company can transition towards continuous experimentation?