Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Managing Experiment at Spotify


Published on

Spotify strives for team autonomy and independence. This means that no team should be blocked by others and they should be able to move as fast as they can. The autonomy has is a challenge for managing a centralised and coordinated experimentation infrastructure and analysis. This a talk about how we approach experimentation alignment in a fast moving company.

Managing Experiment at Spotify

  1. May 20, 2014 Managing experiments at Spotify Ali Sarrafi Product Owner A/B testing and Experimentation
  2. May 20, 2014 The right music for every moment 6
  3. May 20, 2014 7 24M active users. 6M paying subscribers. 20Msongs.
  4. Tribes Provide fast and reliable access to all the world's music Enable high product development speed while maintaining a highly available service P P P P Squad Squad Squad Squad Chapter Chapter P P P P Squad Squad Squad Squad Chapter Chapter Our team is growing rapidly 250 30 100 10
  5. 9
  6. 10 Autonomous fast moving teams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
  7. May 20, 2014 Some Statistics Over 40 autonomous teams working on features Over 7 platforms with multiple features Over 3000 source repositories
  8. 12 We don’t want a Frankenstein software
  9. 13 Well at least a good looking 2014 edition
  10. Section name 14 How can we align teams’ optimisation to avoid the frankenstein effect?
  11. 15 Align on metrics rather than actual tests
  12. 16 DAU (Daily Active Users) MAU (Monthly Active Users)
  13. Section name 17 Evolving Metrics
  14. May 20, 2014 Focus on functionality as a metric
  15. 19 DAU/MAU Function 1 Function 2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
  16. Section name 20 Now we know what would work how do we make sure everyone uses it?
  17. 21 It’s all about automation
  18. 22 and of course education
  19. Section name 23
  20. 24 Make magic happen with desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -3% Functionality 3 +0.5% 0% Functionality 4 +1% -7%
  21. 25 People/Infrastructure as a service Feature 1 Feature 5 Feature 4 Feature 2 Feature 3
  22. Summary • Metric alignment is the most effective way of making sure the overall experience does not suffer • Education and Automation are key for adoption • People as a service is a most effective way of reaching out to teams.
  23. May 20, 2014 Copyright stuff Napster pic by bixentro, cc 2.0 Busy record store pic by gomattolson Empty record store pic by passetti First Frankenstein screenshot form : “Frankenstein” movie by Kenneth Branagh, 1994 Second Frankenstein screenshot form : “I, Frankenstein” movie by Stuart Beattie, 2014
  24. May 20, 2014 Thanks @asarrafi