Uploaded byAli Sarrafi
A/B testing at Spotify
The presentation discusses A/B testing at Spotify, emphasizing its role in improving product functionality and user engagement among millions of subscribers. It outlines the challenges faced by Spotify's autonomous teams and proposes a structured approach using metrics to optimize features to avoid software complications. Key takeaways include the importance of automation and education to enhance the A/B testing process.
More Related Content
![[QE 2018] Paul Gerrard – Automating Assurance: Tools, Collaboration and DevOps](https://cdn.slidesharecdn.com/ss_thumbnails/automatingassurancegerrard-180629130855-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Adria 23]Rok Piltaver Lessons Learned From 10 Years of AB Testing.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/rokpiltaverlessonslearnedfrom10yearsofabtesting-230530180353-baf610f5-thumbnail.jpg?width=640&height=640&fit=bounds)
A/B testing at Spotify
- 1. February 12, 2015 A/Btesting at Spotify Ali Sarrafi & Evan Shrubsole
- 2. Who is this? 2 AliSarrafi Evan Shrubsole Software EngineerProduct Owner
- 3. Section name Source: Agenda 3 Spotify intro HowSpotify works Our challenges How we manage A/B tests How we implement A/B tests Conclusions + Q&A
- 8. February 12, 2015 Theright music for every moment 8
- 9. 9 15 million subscribers 60million monthly active users 30 million songs
- 10. Tribes Provide fast and reliableaccess to all the world's music Enable high product development speed while maintaining a highly available service P P P P Squad Squad Squad Squad Chapter Chapter P P P P Squad Squad Squad Squad Chapter Chapter Our team is growing rapidly 400 30 100 10
- 11. 11 Autonomous fast movingteams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
- 12. February 12, 2015 MoreContext Over 50 autonomous teams working on features Over 7 platforms with multiple features Over 3000 source repositories
- 13. Section name 13 Howto make sure we improve the product?
- 14.
- 15. Do A/B testing! 15 A/Btesting is a simple randomised experiment with at least two variants, which are the control and treatment in the controlled experiment
- 16.
- 17. 17 We don’t wanta Frankenstein software Challenge
- 18. 18 Well at leasta good looking 2014 edition
- 19. Section name 19 Howcan we align teams’ optimisation to avoid the Frankenstein effect?
- 20.
- 21. Source: 21 DAU (Daily ActiveUsers) MAU (Monthly Active Users)
- 22. Section name 22 EvolvingMetrics
- 23. Focus on functionalityas a metric We want people to play music and enjoy it 23
- 24. 24 DAU/MAU Function 1 Function2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
- 25. Section name 25 Howshould we implement this efficiently?
- 26. Code 26 if (shuffle_button =“green_square") show_green_square_shuffle() else show_old_shuffle()
- 27. Managing tests 27 A/B test Name:My test Hypothesis: … …. …. Registry
- 28.
- 29. 29 Making magic happenwith desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -3% Functionality 3 +0.5% 0% Functionality 4 +1% -7%
- 30. Section name 30 Howcan we make the process more successful?
- 31. 31 It’s all aboutautomation
- 32. 32 and of courseeducation
- 33. Copyright Napster pic bybixentro, cc 2.0 http://www.flickr.com/photos/bixentro/ Busy record store pic by gomattolson http://www.flickr.com/photos/gomattolson Empty record store pic by passetti http://www.flickr.com/photos/passetti/ First Frankenstein screenshot form : “Frankenstein” movie by Kenneth Branagh, 1994 Second Frankenstein screenshot form : “I, Frankenstein” movie by Stuart Beattie, 2014 Automation screenshot from: Startrek Next generation, TV show (1987–94)
- 34.