A/B testing at Spotify
•
14 likes•11,405 views
Spotify strives for team autonomy and independence. This means that no team should be blocked by others and they should be able to move as fast as they can. The autonomy has is a challenge for managing a centralised and coordinated experimentation infrastructure and analysis. This a talk about how we approach A/B testing in a fast moving company.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to A/B testing at Spotify
Similar to A/B testing at Spotify (20)
Recently uploaded
Recently uploaded (20)
A/B testing at Spotify
- 1. February 12, 2015 A/B testing at Spotify Ali Sarrafi & Evan Shrubsole
- 2. Who is this? 2 Ali Sarrafi Evan Shrubsole Software EngineerProduct Owner
- 3. Section name Source: Agenda 3 Spotify intro How Spotify works Our challenges How we manage A/B tests How we implement A/B tests Conclusions + Q&A
- 8. February 12, 2015 The right music for every moment 8
- 9. 9 15 million subscribers 60 million monthly active users 30 million songs
- 10. Tribes Provide fast and reliable access to all the world's music Enable high product development speed while maintaining a highly available service P P P P Squad Squad Squad Squad Chapter Chapter P P P P Squad Squad Squad Squad Chapter Chapter Our team is growing rapidly 400 30 100 10
- 11. 11 Autonomous fast moving teams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
- 12. February 12, 2015 More Context Over 50 autonomous teams working on features Over 7 platforms with multiple features Over 3000 source repositories
- 13. Section name 13 How to make sure we improve the product?
- 14. 14 Hippo Approach Highest Paid Person’s Opinion
- 15. Do A/B testing! 15 A/B testing is a simple randomised experiment with at least two variants, which are the control and treatment in the controlled experiment
- 17. 17 We don’t want a Frankenstein software Challenge
- 18. 18 Well at least a good looking 2014 edition
- 19. Section name 19 How can we align teams’ optimisation to avoid the Frankenstein effect?
- 21. Source: 21 DAU (Daily Active Users) MAU (Monthly Active Users)
- 22. Section name 22 Evolving Metrics
- 23. Focus on functionality as a metric We want people to play music and enjoy it 23
- 24. 24 DAU/MAU Function 1 Function 2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
- 25. Section name 25 How should we implement this efficiently?
- 26. Code 26 if (shuffle_button = “green_square") show_green_square_shuffle() else show_old_shuffle()
- 27. Managing tests 27 A/B test Name: My test Hypothesis: … …. …. Registry
- 29. 29 Making magic happen with desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -3% Functionality 3 +0.5% 0% Functionality 4 +1% -7%
- 30. Section name 30 How can we make the process more successful?
- 31. 31 It’s all about automation
- 32. 32 and of course education
- 33. Copyright Napster pic by bixentro, cc 2.0 http://www.flickr.com/photos/bixentro/ Busy record store pic by gomattolson http://www.flickr.com/photos/gomattolson Empty record store pic by passetti http://www.flickr.com/photos/passetti/ First Frankenstein screenshot form : “Frankenstein” movie by Kenneth Branagh, 1994 Second Frankenstein screenshot form : “I, Frankenstein” movie by Stuart Beattie, 2014 Automation screenshot from: Startrek Next generation, TV show (1987–94)
- 34. Thanks