Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

A/B testing at Spotify Slide 1 A/B testing at Spotify Slide 2 A/B testing at Spotify Slide 3 A/B testing at Spotify Slide 4 A/B testing at Spotify Slide 5 A/B testing at Spotify Slide 6 A/B testing at Spotify Slide 7 A/B testing at Spotify Slide 8 A/B testing at Spotify Slide 9 A/B testing at Spotify Slide 10 A/B testing at Spotify Slide 11 A/B testing at Spotify Slide 12 A/B testing at Spotify Slide 13 A/B testing at Spotify Slide 14 A/B testing at Spotify Slide 15 A/B testing at Spotify Slide 16 A/B testing at Spotify Slide 17 A/B testing at Spotify Slide 18 A/B testing at Spotify Slide 19 A/B testing at Spotify Slide 20 A/B testing at Spotify Slide 21 A/B testing at Spotify Slide 22 A/B testing at Spotify Slide 23 A/B testing at Spotify Slide 24 A/B testing at Spotify Slide 25 A/B testing at Spotify Slide 26 A/B testing at Spotify Slide 27 A/B testing at Spotify Slide 28 A/B testing at Spotify Slide 29 A/B testing at Spotify Slide 30 A/B testing at Spotify Slide 31 A/B testing at Spotify Slide 32 A/B testing at Spotify Slide 33 A/B testing at Spotify Slide 34
Upcoming SlideShare
A/B Testing Pitfalls and Lessons Learned at Spotify
Next

12 Likes

Share

A/B testing at Spotify

Spotify strives for team autonomy and independence. This means that no team should be blocked by others and they should be able to move as fast as they can. The autonomy has is a challenge for managing a centralised and coordinated experimentation infrastructure and analysis. This a talk about how we approach A/B testing in a fast moving company.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

A/B testing at Spotify

  1. 1. February 12, 2015 A/B testing at Spotify Ali Sarrafi & Evan Shrubsole
  2. 2. Who is this? 2 Ali Sarrafi Evan Shrubsole Software EngineerProduct Owner
  3. 3. Section name Source: Agenda 3 Spotify intro How Spotify works Our challenges How we manage A/B tests How we implement A/B tests Conclusions + Q&A
  4. 4. February 12, 2015 The right music for every moment 8
  5. 5. 9 15 million subscribers 60 million monthly active users 30 million songs
  6. 6. Tribes Provide fast and reliable access to all the world's music Enable high product development speed while maintaining a highly available service P P P P Squad Squad Squad Squad Chapter Chapter P P P P Squad Squad Squad Squad Chapter Chapter Our team is growing rapidly 400 30 100 10
  7. 7. 11 Autonomous fast moving teams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
  8. 8. February 12, 2015 More Context Over 50 autonomous teams working on features Over 7 platforms with multiple features Over 3000 source repositories
  9. 9. Section name 13 How to make sure we improve the product?
  10. 10. 14 Hippo Approach Highest Paid Person’s Opinion
  11. 11. Do A/B testing! 15 A/B testing is a simple randomised experiment with at least two variants, which are the control and treatment in the controlled experiment
  12. 12. Spotify Example 16
  13. 13. 17 We don’t want a Frankenstein software Challenge
  14. 14. 18 Well at least a good looking 2014 edition
  15. 15. Section name 19 How can we align teams’ optimisation to avoid the Frankenstein effect?
  16. 16. 20 Align on metrics
  17. 17. Source: 21 DAU (Daily Active Users) MAU (Monthly Active Users)
  18. 18. Section name 22 Evolving Metrics
  19. 19. Focus on functionality as a metric We want people to play music and enjoy it 23
  20. 20. 24 DAU/MAU Function 1 Function 2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
  21. 21. Section name 25 How should we implement this efficiently?
  22. 22. Code 26 if (shuffle_button = “green_square")    show_green_square_shuffle() else   show_old_shuffle()
  23. 23. Managing tests 27 A/B test Name: My test Hypothesis: … …. …. Registry
  24. 24. Analysing results 28 Exposure logs
  25. 25. 29 Making magic happen with desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -3% Functionality 3 +0.5% 0% Functionality 4 +1% -7%
  26. 26. Section name 30 How can we make the process more successful?
  27. 27. 31 It’s all about automation
  28. 28. 32 and of course education
  29. 29. Copyright Napster pic by bixentro, cc 2.0 http://www.flickr.com/photos/bixentro/ Busy record store pic by gomattolson http://www.flickr.com/photos/gomattolson Empty record store pic by passetti http://www.flickr.com/photos/passetti/ First Frankenstein screenshot form : “Frankenstein” movie by Kenneth Branagh, 1994 Second Frankenstein screenshot form : “I, Frankenstein” movie by Stuart Beattie, 2014 Automation screenshot from: Startrek Next generation, TV show (1987–94)
  30. 30. Thanks
  • FlorianaNicastro

    Sep. 12, 2020
  • UKYCHOI

    Feb. 10, 2020
  • ConnorAguilera

    Sep. 8, 2018
  • rincon.rise

    Sep. 20, 2017
  • IrynaKritsak

    Sep. 7, 2017
  • MosBar

    Jun. 8, 2017
  • yongyuth1

    Jul. 11, 2016
  • beluckyy

    Mar. 27, 2016
  • justinschmidt80

    Jan. 12, 2016
  • coolkdw

    Nov. 25, 2015
  • seubsakol

    Jul. 7, 2015
  • hbflynn

    Mar. 14, 2015

Spotify strives for team autonomy and independence. This means that no team should be blocked by others and they should be able to move as fast as they can. The autonomy has is a challenge for managing a centralised and coordinated experimentation infrastructure and analysis. This a talk about how we approach A/B testing in a fast moving company.

Views

Total views

10,097

On Slideshare

0

From embeds

0

Number of embeds

211

Actions

Downloads

0

Shares

0

Comments

0

Likes

12

×