1.
February 12, 2015
A/B testing at Spotify
Ali Sarrafi & Evan Shrubsole

2.
Who is this?
2
Ali Sarrafi Evan Shrubsole
Software EngineerProduct Owner

3.
Section name
Source:
Agenda
3
Spotify intro
How Spotify works
Our challenges
How we manage A/B tests
How we implement A/B tests
Conclusions + Q&A

4.
February 12, 2015
The right music for every moment
8

5.
9
15 million subscribers
60 million monthly active users
30 million songs

6.
Tribes
Provide fast and
reliable access to all
the world's music
Enable high product
development speed
while maintaining a
highly available service
P P P P
Squad Squad Squad Squad
Chapter
Chapter
P P P P
Squad Squad Squad Squad
Chapter
Chapter
Our team is growing rapidly
400
30
100
10

7.
11
Autonomous fast moving teams
Feature 1
Feature 3
Feature 2
Feature 4
Feature 5

8.
February 12, 2015
More Context
Over 50 autonomous teams working on features
Over 7 platforms with multiple features
Over 3000 source repositories

9.
Section name 13
How to make sure we improve the product?

10.
14
Hippo Approach
Highest Paid
Person’s Opinion

11.
Do A/B testing!
15
A/B testing is a simple
randomised experiment
with at least two variants,
which are the control and
treatment in the controlled
experiment

12.
Spotify Example
16

13.
17
We don’t want a Frankenstein software
Challenge

14.
18
Well at least a good looking 2014 edition

15.
Section name 19
How can we align teams’ optimisation to
avoid the Frankenstein effect?

16.
20
Align on metrics

17.
Source:
21
DAU (Daily Active Users)
MAU (Monthly Active Users)

18.
Section name 22
Evolving Metrics

19.
Focus on functionality as a metric
We want people to play music and enjoy it
23

20.
24
DAU/MAU
Function 1 Function 2 Function 3 Function 4
CTR/Team Metrics
Importance
(High to low)
Order of
optimisation

21.
Section name 25
How should we implement this efficiently?

22.
Code
26
if (shuffle_button = “green_square")
show_green_square_shuffle()
else
show_old_shuffle()

23.
Managing tests
27
A/B test
Name: My test
Hypothesis: …
….
….
Registry

24.
Analysing results
28
Exposure logs

25.
29
Making magic happen with desired metrics
Vital Sign Group A Group B
Functionality 1 +1.5% -0.5%
Functionality 2 +2.5% -3%
Functionality 3 +0.5% 0%
Functionality 4 +1% -7%

26.
Section name 30
How can we make the process more
successful?

27.
31
It’s all about automation

28.
32
and of course education

29.
Copyright
Napster pic by bixentro, cc 2.0
http://www.flickr.com/photos/bixentro/
Busy record store pic by gomattolson
http://www.flickr.com/photos/gomattolson
Empty record store pic by passetti
http://www.flickr.com/photos/passetti/
First Frankenstein screenshot form :
“Frankenstein” movie by Kenneth Branagh, 1994
Second Frankenstein screenshot form :
“I, Frankenstein” movie by Stuart Beattie, 2014
Automation screenshot from:
Startrek Next generation, TV show (1987–94)

30.
Thanks