Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
February 12, 2015 A/B testing at Spotify Ali Sarrafi & Evan Shrubsole
Who is this? 2 Ali Sarrafi Evan Shrubsole Software EngineerProduct Owner
Section name Source: Agenda 3 Spotify intro How Spotify works Our challenges How we manage A/B tests How we implement A/B ...
February 12, 2015 The right music for every moment 8
9 15 million subscribers 60 million monthly active users 30 million songs
Tribes Provide fast and reliable access to all the world's music Enable high product development speed while maintaining a...
11 Autonomous fast moving teams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
February 12, 2015 More Context Over 50 autonomous teams working on features Over 7 platforms with multiple features Over 3...
Section name 13 How to make sure we improve the product?
14 Hippo Approach Highest Paid Person’s Opinion
Do A/B testing! 15 A/B testing is a simple randomised experiment with at least two variants, which are the control and tre...
Spotify Example 16
17 We don’t want a Frankenstein software Challenge
18 Well at least a good looking 2014 edition
Section name 19 How can we align teams’ optimisation to avoid the Frankenstein effect?
20 Align on metrics
Source: 21 DAU (Daily Active Users) MAU (Monthly Active Users)
Section name 22 Evolving Metrics
Focus on functionality as a metric We want people to play music and enjoy it 23
24 DAU/MAU Function 1 Function 2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
Section name 25 How should we implement this efficiently?
Code 26 if (shuffle_button = “green_square")    show_green_square_shuffle() else   show_old_shuffle()
Managing tests 27 A/B test Name: My test Hypothesis: … …. …. Registry
Analysing results 28 Exposure logs
29 Making magic happen with desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -...
Section name 30 How can we make the process more successful?
31 It’s all about automation
32 and of course education
Copyright Napster pic by bixentro, cc 2.0 http://www.flickr.com/photos/bixentro/ Busy record store pic by gomattolson http...
Thanks
A/B testing at Spotify
A/B testing at Spotify
A/B testing at Spotify
A/B testing at Spotify
Upcoming SlideShare
Loading in …5
×

A/B testing at Spotify

8,806 views

Published on

Spotify strives for team autonomy and independence. This means that no team should be blocked by others and they should be able to move as fast as they can. The autonomy has is a challenge for managing a centralised and coordinated experimentation infrastructure and analysis. This a talk about how we approach A/B testing in a fast moving company.

Published in: Technology
no profile picture user

  • Login to see the comments

Show More

A/B testing at Spotify

  1. 1. February 12, 2015 A/B testing at Spotify Ali Sarrafi & Evan Shrubsole
  2. 2. Who is this? 2 Ali Sarrafi Evan Shrubsole Software EngineerProduct Owner
  3. 3. Section name Source: Agenda 3 Spotify intro How Spotify works Our challenges How we manage A/B tests How we implement A/B tests Conclusions + Q&A
  4. 4. February 12, 2015 The right music for every moment 8
  5. 5. 9 15 million subscribers 60 million monthly active users 30 million songs
  6. 6. Tribes Provide fast and reliable access to all the world's music Enable high product development speed while maintaining a highly available service P P P P Squad Squad Squad Squad Chapter Chapter P P P P Squad Squad Squad Squad Chapter Chapter Our team is growing rapidly 400 30 100 10
  7. 7. 11 Autonomous fast moving teams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
  8. 8. February 12, 2015 More Context Over 50 autonomous teams working on features Over 7 platforms with multiple features Over 3000 source repositories
  9. 9. Section name 13 How to make sure we improve the product?
  10. 10. 14 Hippo Approach Highest Paid Person’s Opinion
  11. 11. Do A/B testing! 15 A/B testing is a simple randomised experiment with at least two variants, which are the control and treatment in the controlled experiment
  12. 12. Spotify Example 16
  13. 13. 17 We don’t want a Frankenstein software Challenge
  14. 14. 18 Well at least a good looking 2014 edition
  15. 15. Section name 19 How can we align teams’ optimisation to avoid the Frankenstein effect?
  16. 16. 20 Align on metrics
  17. 17. Source: 21 DAU (Daily Active Users) MAU (Monthly Active Users)
  18. 18. Section name 22 Evolving Metrics
  19. 19. Focus on functionality as a metric We want people to play music and enjoy it 23
  20. 20. 24 DAU/MAU Function 1 Function 2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
  21. 21. Section name 25 How should we implement this efficiently?
  22. 22. Code 26 if (shuffle_button = “green_square")    show_green_square_shuffle() else   show_old_shuffle()
  23. 23. Managing tests 27 A/B test Name: My test Hypothesis: … …. …. Registry
  24. 24. Analysing results 28 Exposure logs
  25. 25. 29 Making magic happen with desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -3% Functionality 3 +0.5% 0% Functionality 4 +1% -7%
  26. 26. Section name 30 How can we make the process more successful?
  27. 27. 31 It’s all about automation
  28. 28. 32 and of course education
  29. 29. Copyright Napster pic by bixentro, cc 2.0 http://www.flickr.com/photos/bixentro/ Busy record store pic by gomattolson http://www.flickr.com/photos/gomattolson Empty record store pic by passetti http://www.flickr.com/photos/passetti/ First Frankenstein screenshot form : “Frankenstein” movie by Kenneth Branagh, 1994 Second Frankenstein screenshot form : “I, Frankenstein” movie by Stuart Beattie, 2014 Automation screenshot from: Startrek Next generation, TV show (1987–94)
  30. 30. Thanks

×