Software Development as an Experiment System: A Qualitative Survey on the State of the Practice
•Download as PPTX, PDF•
1 like•411 views
This document summarizes the findings of a qualitative survey on the state of continuous experimentation in software development companies. The survey involved interviews with experienced practitioners from 10 companies in Finland. It was found that while agile practices like Scrum and continuous integration were common, systematic experimentation was rare. Supporting practices for experimentation existed but were not integrated into a continuous experimentation process. Key challenges identified included organizational culture, product management, data management, and resources. The document concludes that continuous experimentation requires developing organizational capabilities and establishing data-driven decision making.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Software Development as an Experiment System: A Qualitative Survey on the State of the Practice
Similar to Software Development as an Experiment System: A Qualitative Survey on the State of the Practice (20)
Recently uploaded
Recently uploaded (20)
Software Development as an Experiment System: A Qualitative Survey on the State of the Practice
- 1. Software Development as an Experiment System A QualitativeSurvey on the Stateof the Practice Eveliina Lindgren, Jürgen Münch XP2015 May 26, Helsinki
- 2. Outline • Motivation and Related Work • Research Questions • Methods and Data Collection • Findings • Conclusions 1
- 3. Motivation 2 Delivering Value Test the customer value of an idea Learn & improve Repeat
- 4. Related Work 3 Stairway to Heaven: Traditional Agile & Lean Continuous delivery Software development as an experiment system HYPEX Company-specific approaches: Microsoft, Google, etc. Continuous Experimentation Different approaches
- 5. Research Questions 4 How is continuous experimentation applied in software development companies? RQ1 What challenges are associated with continuous experimentation? RQ2
- 6. Methods and Data Collection • Qualitative survey • Semi-structured individual interviews • Thematic coding analysis • Ten ICT companies operating in Finland • Small (50%), medium (20%), and large (30%) • B2B and B2C • Thirteen experienced interviewees 5
- 7. RQ1: Development Practices • Agile methods and practices prevailed • Scrum, Kanban, Lean • Continuous integration • Pursuit of easy deployments • Release cycle < 1 month (56%), < 3 months (33%) • Direct customer feedback in use • Interviews, surveys, prototypes, user testing… 6
- 8. RQ1: Development Practices (cont.) • Product usage data collected by 55% • Often coarse-grained and difficult to analyze • Most wanted to improve data collection • Systematic experimentation rare • Continuous experimentation in only one company • Experimentation considered an appropriate approach 7 Supporting practices exist, but they need to be integrated into a continuous experimentation process
- 9. RQ2: Key Challenges • Organizational Culture • Product Management • Data Management • Resources 8
- 10. 9 Reaching the right customers Breaking down silos (business, BA, PM, R&D) Changing culture and processes Overcoming reluctance to provide feedback Organizational challenges B2B- specific B2B- specific RQ2 Findings (cont.)
- 11. 10 Identifying hypotheses & metrics Accelerating release cycle speed Prioritizing the backlog Identifying an MVP Product management challenges RQ2 Findings (cont.)
- 12. Consent to collect product usage data B2B- specific Availability and sharing of data 11 Linking data and decision making Quality of data analysis Data management challenges RQ2 Findings (cont.)
- 13. Findings: Validity Considerations • External validity • Limited scope Variety of companies, experienced interviewees • Construct validity • Possible misunderstandings Central concept shared with participants, clarifying questions during and after interviews • Reliability • Replication of a qualitative study Interview guide and analytic codebook 12
- 14. Conclusions State of the practice Systematic experimentation is rare Supporting practices exist, but they need to be integrated into a continuous experimentation process Practitioners interested in the approach Challenges Developing organizational capabilities Achieving faster releases Establishing data-driven decision making 13
- 15. Continuous Experimentation is a change of mindset. {eveliina.lindgren, juergen.muench}@cs.helsinki.fi www.juergenmuench.com
- 16. References Holmström Olsson, H., Alahyari, H., Bosch, J.: Climbing the “Stairway to Heaven”: A Multiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software. In: 38th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), pp. 392–399. IEEE Press (2012) Holmström Olsson, H., Bosch, J.: The HYPEX Model: From Opinions to Data-Driven Software Development. In: Continuous Software Engineering Part IV, pp 155-164. Springer (2014) Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online Controlled Experiments at Large Scale. In: 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1168–1176. ACM, New York (2013) Steiber, A., Alänge, S.: A Corporate System for Continuous Innovation: The Case of Google Inc. European Journal of Innovation Management, 16, 2, pp. 243–264 (2013) Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: Building Blocks for Continuous Experimentation. In: 1st International Workshop on Rapid Continuous Software Engineering, pp. 26–35. ACM, New York (2014) Please see conference paper for further references! 15
Editor's Notes
- Good morning also from my part. My name is Eveliina Lindgren, and I am here to introduce a study by myself and Mr. Jürgen Münch from the University of Helsinki. We conducted a qualitative survey on Software Development as an Experiment System To understand whether companies use such an approach.
- First I’ll explain the motivation and the background for the study. Then its on to the research questions and the methods. Next up are the main findings, followed by a quick wrap-up. I have also reserved some time for questions at the end, (but feel free to also ask them along the way.)
- The overall theme of the conference is ”delivering value”, and this is also at the heart of our study. So, the goal is to deliver compelling products, services, and features to the customers. The prerequisite for this is knowing what really is valuable to our customers. (1) In other words, we need to systematically test the value of our development ideas in their marketplace. Relying on pre-defined requirements or assumptions of customer value is too uncertain. (2) Conducting such value tests enables us to learn, make validated decisions, and improve our product. (3) Once we have done this, we are ready for another value test to see if the development efforts took us in the right direction or not. This is, in a tiny nutshell, the experiment system approach we’ll be discussing.
- So, what is the current situation regarding the experiment system approach? (1) The “Stairway to Heaven” model describes the maturation path of companies’ software engineering practices. It progresses from traditional development to agile R&D, continuous integration, and continuous delivery. These increasingly agile practices are the foundation for the final stage, which is (2) R&D as an experiment system. There are naturally many different approaches to experimentation. (3) Many companies, such as Microsoft, Google, and Adobe, have their own methods. There are also more structured approaches, For instance the HYPEX model, which stands for Hypothesis Experiment Data-Driven Development. (It presents a development process based on feature experiments.) A related model is Continuous Experimentation, which is a framework for integrating development and experimentation. It also uses central ideas from the lean startup method. We used this model as a basis for our study. As we can see, there are several approaches to experimentation, But the understanding of the actual state of the practice is somewhat incomplete.
- Therefore our study aimed at finding out whether companies actually use an experiment system approach. And if they do, how do they use it? We also tried to identify if practitioners see any challenges with the approach (, since this has not been widely discussed.) (Most of the current case studies tend to be from major web-facing companies. Not so much is known about whether other types of companies use experimentation.)
- We used a qualitative survey design for the study. Semi-structured individual interviews were conducted to collect data, and the data was analyzed via thematic coding analysis. (1) Ten ICT companies participated in the study. There were companies of different sizes, and three of them can be described as startups. Both business-to-business and business-to-consumer companies were represented. (2) We interviewed a total of thirteen people, most of them currently in a managerial role. (There were companies from various domains, including multimedia, telecom, gaming, and security.) Possible extra material: (A qualitative survey is similar to the multiple case study paradigm, but with less of a focus on individual cases and more on the broader picture.) Interview topics: The interviews covered three main topics: First, current software development practices of the company Second, their practices of collecting and using customer feedback, including product usage data Third, thoughts on future practices. Some examples of interview questions: What kind of software development process do you use? How do you make sure that you are building the right product? How do you collect customer feedback? Do you collect data about customer behavior, for example in the form of product usage data? Are there any obstacles to obtaining deeper customer insights? Rationales: (We selected the qualitative survey approach since the goal was to chart the experiences of practitioners from different company backgrounds.) (We used semi-structured interviews because they combined flexibility with predefined topics. Flexibility was needed to allow for companies’ varying practices and unforeseen information. ) (On average, the interviews lasted 48 minutes. The interviews were recorded and transcribed.) (Thematic analysis is similar to many other qualitative data analysis methods, and was selected for its flexibility. We used inductive coding, as opposed to deductive, because of the explorative research questions. The coding process was iterative.)
- To answer the first research question, we invited the interviewees to describe their company’s software development practices. We did not use a formal questionnaire to evaluate their practices. (1) Everyone stated that they use agile methods such as Scrum, Kanban, and Lean, and practices such as continuous integration. (2) Second, the companies pursued flexible, frequent deployments. Release cycle length was mostly relatively short, And many were trying to further shorten it. On-premises installations of B2B software posed a challenge in this respect. (3) To learn about customer value, companies used many techniques of collecting direct feedback. These included interviews, surveys, prototypes, and usability and user experience testing. (How did companies pursue easy deployments? By always having a deployable product version available, working in a production-like environment to simplify deployments, and by striving towards a DevOps mode of operation.)
- In addition to direct customer feedback, half of the companies collected product usage data to learn about customer value. The data was often quite coarse-grained, and not, for instance, on a feature level. It was also often difficult to analyze. Most companies wanted to improve their practices of collecting product data. (1) We found that systematic experimentation was rare. There was only one startup company which practiced continuous experimentation. Others had tried some experiments but they were not necessarily systematic, Or an integral part of the development process. However, many participants expressed interest in experimentation in the sense that it could be an appropriate approach for them. (For example, some had plans to introduce A/B testing into their development process.) (2) To summarize, There are already many development practices which support experimentation. To improve further, they need to be integrated into a systematic cycle of continuous experimentation.
- So, it seems that experimentation is not yet very common. Our second research question focused on the challenges of the approach. Here are the key topic areas that we identified. I will present the first three in more detail in the following slides. In addition, insufficient resources were often mentioned, But I will not go into it in this presentation (since it is quite self-explanatory.)
- First up are challenges related to organizational culture. (1) First, the culture and the processes should be changed in a way which supports experimentation. Some participants regarded this as the most important obstacle to overcome. (2) A related challenge had to do with breaking down organizational silos. Experimentation requires solid cooperation between parties such as business management, business analysts, product management, and development. So the whole company needs to be attuned to the approach. (Cross-disciplinary teams may be useful.) (3) Finally, in the B2B world, the customer organizations’ culture also had an impact. For example, there were challenges in acquiring enough customer feedback, especially from the end users.
- Next up are challenges related to managing the product and its development in an experimental way. First, in order to learn fast, cycle speeds should be relatively short. Many companies had concerns on this score. (2) The next step is planning the experiment, including identifying the hypotheses to test and the metrics to be used. These were further obstacles. There were also issues with identifying which technique to use to collect customer feedback. (How to evaluate customer value and product success?) (3) Third, to conduct the experiment, we need a suitable product version. However, identifying an MVP was considered “very easy to say, very hard to do”. (4) And finally, the results of the experiment can be used to prioritize the backlog. At the moment this was such a challenge that it was aptly described as “black magic”. (1: A suggested solution was to focus on products and features that create the most customer value.) (2: Quality over quantity was recommended in selecting metrics.)
- As we saw earlier, the companies collected quite a lot of customer feedback and also product data. However, many participants felt that the data was not analyzed thoroughly enough, Or that is was not used efficiently in decision-making. (2) The interviewees also thought that the data needed to be shared more freely with relevant stakeholders. It also needed to be more accessible, meaning that an efficient infrastructure for data management was needed. (3) Finally, some B2B companies had problems with getting a permission to collect product usage data.
- Finally, a few words on validity. (1) The study was limited in scope, so its external validity is debatable. However, a variety of companies and experienced practitioners did contribute to the study. (And we believe that the results reflect actual practice.) (2) As is common in qualitative studies, misunderstandings were possible during data collection and analysis. We tried to minimize this risk by explaining the concept of continuous experimentation to participants before the interviews. We also asked clarifying questions whenever necessary. (3) To improve reliability, we developed the interview guide and the analytic codebook. You can find them online on Figshare.
- So to conclude, (1) our study found that systematic experimentation is currently rare. Continuous experimentation is even more rare. (2) However, many commonly used agile and Lean practices support experimentation. These need to be incorporated into a systematic cycle of continuous experimentation. (3) Many practitioners also seemed to be interested in the approach. Some of the major challenges concerning experimentation have to do with (4) Developing organizational capabilities, (5) Achieving faster releases, (6) And developing the culture and the means for data-driven decision making.
- As we have seen, there are various things to consider in embracing experimentation, (1) but perhaps the main challenge is changing your mindset. That’s all from me, Do you have any questions or comments? You can also contact us after this session. Thank you all!