For the first time Spotify data team is sharing how to build a music recommender system from scratch.

1. November 14, 2015
Building
a
Music Recommender
from
Scratch
Vidhya Murali
@vid052

2. Vidhya Murali
Who Am I?
2
•Areas of Interest: Data & Machine Learning
•Data Science Engineer @Spotify
•Masters Student from the University of Wisconsin Madison
aka Happy Badger for life!

3. “Torture the data, and it will
confess!”
3
– Ronald Coase, Nobel Prize Laureate

4. Music Recommendations at Spotify
Features:
Discover
Discover Weekly
Moments
Radio
Related Artists
4

5. 5
30 million tracks…
What to recommend?

6. 6
•Manual Curation by Experts
•Editorial Tagging
•Metadata (e.g. Label provided data, NLP over News,
Blogs)
•Audio Signals
•Collaborative Filtering Model
Approaches

7. 6
•Manual Curation by Experts
•Editorial Tagging
•Metadata (e.g. Label provided data, NLP over News,
Blogs)
•Audio Signals
•Collaborative Filtering Model
Approaches

8. Definition of CF
7
Hey,
I like tracks P, Q, R, S!
Well,
I like tracks Q, R, S, T!
Then you should check out
track P!
Nice! Btw try track T!
Legacy Slide of Erik Bernhardsson

9. Collaborative Filtering Model 8
•Find patterns from user’s past behavior to generate
recommendations
•Domain independent
•Scalable
•Accuracy (Collaborative Model) >= Accuracy (Content
Based Model)

10. Construct Big Matrix!
9
Artists(n)
Users(m)
Vidhya
Ellie Goulding

11. Construct Big Matrix!
9
Artists(n)
Users(m)
Vidhya
Ellie Goulding
Order of Millions!

12. Latent Factor Models 10
Vidhya
Ellie
.. . . . .
.. . . . .
.. . . . .
.. . . . .
.. . . . .
•Use a “small” representation for each user and items(artists): f-dimensional
vectors
.. .
.. .
.. .
.. .
. .
...
...
...
...
..
m m
n
m n

13. Latent Factor Models 10
Vidhya
Ellie
.. . . . .
.. . . . .
.. . . . .
.. . . . .
.. . . . .
•Use a “small” representation for each user and items(artists): f-dimensional
vectors
.. .
.. .
.. .
.. .
. .
...
...
...
...
..
m m
n
m n
User Artist Matrix:
(m x n)

14. Latent Factor Models 10
Vidhya
Ellie
.. . . . .
.. . . . .
.. . . . .
.. . . . .
.. . . . .
•Use a “small” representation for each user and items(artists): f-dimensional
vectors
.. .
.. .
.. .
.. .
. .
...
...
...
...
..
m m
n
m n
User Vector Matrix:
X: (m x f)
User Artist Matrix:
(m x n)

15. Latent Factor Models 10
Vidhya
Ellie
.. . . . .
.. . . . .
.. . . . .
.. . . . .
.. . . . .
•Use a “small” representation for each user and items(artists): f-dimensional
vectors
.. .
.. .
.. .
.. .
. .
...
...
...
...
..
m m
n
m n
User Vector Matrix:
X: (m x f)
Artist Vector Matrix:
Y: (n x f)
User Artist Matrix:
(m x n)

16. Latent Factor Models 10
Vidhya
Ellie
.. . . . .
.. . . . .
.. . . . .
.. . . . .
.. . . . .
•Use a “small” representation for each user and items(artists): f-dimensional
vectors
.. .
.. .
.. .
.. .
. .
...
...
...
...
..
(here, f = 2)
m m
n
m n
User Vector Matrix:
X: (m x f)
Artist Vector Matrix:
Y: (n x f)
User Artist Matrix:
(m x n)

17. Why Vectors? 11
•Vectors encode higher order dependencies
•Users and Items in the same vector space!
•Use vector similarity to compute:
•Item-Item similarities
•User-Item recommendations
•Linear complexity: order of number of latent factors
•Easy to scale up

18. Explicit Matrix Factorization 12
•User explicitly rates a subset of the music catalog
•Goal: Predict how users will rate new music
•How: Approximate ratings matrix by the inner product of 2 smaller matrices
by minimizing the RMSE (root mean squared error)
X YUsers
Artists
• = bias for user
• = bias for item
• = regularization parameter
• = user rating for item
• = user latent factor vector
• = item latent factor vector

19. Matrix Factorization using Implicit Feedback 13

20. Matrix Factorization using Implicit Feedback
User Artist Play
Count Matrix
13

21. Matrix Factorization using Implicit Feedback
User Artist Play
Count Matrix
User Artist
Preference
Matrix
Binary Label:
1 => played
0 => not played
13

22. Matrix Factorization using Implicit Feedback
User Artist Play
Count Matrix
User Artist
Preference
Matrix
Binary Label:
1 => played
0 => not played
Weights
Matrix
Weights based on play count
and smoothing
13

23. Equation(s) Alert!
14

24. Implicit Matrix Factorization 15
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
•Aggregate all (user, artist) streams into a large matrix
•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight
•Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the artist latent factor vectors in Y.
X YUsers
Artists
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed artist else 0
•
• = user latent factor vector
• = item latent factor vector

25. Alternating Least Squares 16
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
X YUsers
Artists
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed artist else 0
•
• = user latent factor vector
• = item latent factor vector
Fix artists
•Aggregate all (user, artist) streams into a large matrix
•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight
•Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the artist latent factor vectors in Y.

26. 17
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
X YUsers
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed artist else 0
•
• = user latent factor vector
• = item latent factor vector
Fix artists
Solve for users
•Aggregate all (user, artist) streams into a large matrix
•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight
•Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the artist latent factor vectors in Y.
Alternating Least Squares
Artists

27. 18
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
X YUsers
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed artist else 0
•
• = user latent factor vector
• = item latent factor vector
Fix users
•Aggregate all (user, artist) streams into a large matrix
•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight
•Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the artist latent factor vectors in Y.
Alternating Least Squares
Artists

28. 19
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
X YUsers
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed artist else 0
•
• = user latent factor vector
• = item latent factor vector
Fix users
Solve for artists
•Aggregate all (user, artist) streams into a large matrix
•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight
•Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the artist latent factor vectors in Y.
Alternating Least Squares
Artists

29. 20
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
X YUsers
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed artist else 0
•
• = user latent factor vector
• = item latent factor vector
Fix users
Solve for artists
Repeat until convergence…
•Aggregate all (user, artist) streams into a large matrix
•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight
•Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the artist latent factor vectors in Y.
Alternating Least Squares
Artists

30. 21
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
X YUsers
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed track else 0
•
• = user latent factor vector
• = item latent factor vector
Fix users
Solve for artists
Repeat until convergence…
•Aggregate all (user, artist) streams into a large matrix
•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight
•Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the artist latent factor vectors in Y.
Alternating Least Squares
Artists

31. Vectors
•“Compact” representation for users and items(artists) in the same space

32. 23
Recommendations via Cosine Similarity

33. 23
Recommendations via Cosine Similarity

34. 24
Annoy
•70 million users, at least 4 million tracks for candidates per user
•Brute Force Approach:
•O(70M x 4M x 10) ~= 0(3 peta-operations)!
• Approximate Nearest Neighbor Oh Yeah!
• Uses Local Sensitive Hashing
• Clone: https://github.com/spotify/annoy

35. 25

36. Thank You!
You can reach me @
Email: vidhya@spotify.com
Twitter: @vid052