Download as PDF, PPTX
![1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
•Aggregate all (user, track) streams into a large matrix
•Goal: Approximate binary preference matrix by inner product of 2 smaller matrices by minimizing the
weighted RMSE (root mean squared error) using a function of plays, context, and recency as weight
X YUsers
Songs
• = bias for user
• = bias for item
• = regularization parameter
• = 1 if user streamed track else 0
•
• = user latent factor vector
• = item latent factor vector
[1] Hu Y. & Koren Y. & Volinsky C. (2008) Collaborative Filtering for Implicit Feedback Datasets 8th IEEE International Conference on Data Mining
Implicit Matrix Factorization](https://image.slidesharecdn.com/discoverweeklydataengconf-151116205329-lva1-app6892/75/From-Idea-to-Execution-Spotify-s-Discover-Weekly-31-2048.jpg)
![1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
•Aggregate all (user, track) streams into a large matrix
•Goal: Model probability of user playing a song as logistic, then maximize log likelihood of binary
preference matrix, weighting positive observations by a function of plays, context, and recency
X YUsers
Songs
• = bias for user
• = bias for item
• = regularization parameter
• = user latent factor vector
• = item latent factor vector
[2] Johnson C. (2014) Logistic Matrix Factorization for Implicit Feedback Data NIPS Workshop on Distributed Matrix Computations
Can also use Logistic Loss!](https://image.slidesharecdn.com/discoverweeklydataengconf-151116205329-lva1-app6892/75/From-Idea-to-Execution-Spotify-s-Discover-Weekly-32-2048.jpg)
![[3] http://benanne.github.io/2014/08/05/spotify-cnns.html
Deep Learning on Audio](https://image.slidesharecdn.com/discoverweeklydataengconf-151116205329-lva1-app6892/75/From-Idea-to-Execution-Spotify-s-Discover-Weekly-35-2048.jpg)
Uploaded byChris Johnson
From Idea to Execution: Spotify's Discover Weekly
The document outlines the development of Spotify's Discover Weekly feature, highlighting its evolution from initial prototype stages in 2013 to its current successful model. Key lessons learned in building scalable recommendation systems include being data-driven, creatively reusing existing infrastructure, and conducting extensive A/B testing. The presentation also emphasizes the importance of user feedback and fostering innovation within the organization.
More Related Content
Similar to From Idea to Execution: Spotify's Discover Weekly
From Idea to Execution: Spotify's Discover Weekly
- 1. From Idea to Execution:Spotify’s Discover Weekly Chris Johnson :: @MrChrisJohnson Edward Newett :: @scaladaze DataEngConf • NYC • Nov 2015 Or: 5 lessons in building recommendation products at scale
- 2. Who are We?? ChrisJohnson Edward Newett
- 3. Spotify in Numbers •Started in 2006, now available in 58 markets • 75+ Million active users, 20 Million paying subscribers • 30+ Million songs, 20,000 new songs added per day • 1.5 Billion user generated playlists • 1 TB user data logged per day • 1,700 node Hadoop cluster • 10,000+ Hadoop jobs run daily
- 4. Challenge: 30M songs…how do we recommend music to users?
- 5.
- 6.
- 7.
- 8. Discover Weekly • Startedin 2006, now available in 58 markets • 75+ Million active users, 20 Million paying subscribers • 30+ Million songs, 20,000 new songs added per day • 1.5 Billion user generated playlists • 1 TB user data logged per day • 1,700 node Hadoop cluster • 10,000+ Hadoop jobs run daily
- 9. The Road to DiscoverWeekly
- 10. 2013 :: DiscoverPage v1.0 • Personalized News Feed of recommendations • Artists, Album Reviews, News Articles, New Releases, Upcoming Concerts, Social Recommendations, Playlists… • Required a lot of attention and digging to engage with recommendations • No organization of content
- 11. 2014 :: DiscoverPage v2.0 • Recommendations grouped into strips (a la Netflix) • Limited to Albums and New Releases • More organized than News-Feed but still requires active interaction
- 12. Insight: users spendingmore time on editorial Browse playlists than Discover.
- 13. Idea: combine the personalizedexperience of Discover with the lean- back ease of Browse
- 14. Meanwhile… 2014 YearIn Music
- 15. Play it forward:Same content as the Discover Page but.. a playlist
- 16. Lesson 1: Be datadriven from start to finish
- 17. Slide from DanMcKinley - Etsy 2008 2012 2015
- 18. • Reach: Howmany users are you reaching • Depth: For the users you reach, what is the depth of reach. • Retention: For the users you reach, how many do you retain? Define success metrics BEFORE you release your test
- 19. • Reach: DWWAU / Spotify WAU • Depth: DW Time Spent / Spotify WAU • Retention: DW week-over-week retention Discover Weekly Key Success Metrics
- 20. 2008 2012 2015 Slidefrom Dan McKinley - Etsy
- 21. Step 1: Prototype(employee test)
- 22. Step 1: Prototype(employee test)
- 23. Results of EmployeeTest were very positive!
- 24. 2008 2012 2015 Slidefrom Dan McKinley - Etsy
- 25. Step 2: ReleaseAB Test to 1% of Users
- 26. Google Form 1%Results
- 27. Personalized image resultedin 10% lift in WAU • Initial 0.5% user test • 1% Spaceman image • 1% Personalized image
- 28. Lesson 2: Reuse existing infrastructurein creative ways
- 29.
- 30.
- 31. 1 0 00 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 •Aggregate all (user, track) streams into a large matrix •Goal: Approximate binary preference matrix by inner product of 2 smaller matrices by minimizing the weighted RMSE (root mean squared error) using a function of plays, context, and recency as weight X YUsers Songs • = bias for user • = bias for item • = regularization parameter • = 1 if user streamed track else 0 • • = user latent factor vector • = item latent factor vector [1] Hu Y. & Koren Y. & Volinsky C. (2008) Collaborative Filtering for Implicit Feedback Datasets 8th IEEE International Conference on Data Mining Implicit Matrix Factorization
- 32. 1 0 00 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 •Aggregate all (user, track) streams into a large matrix •Goal: Model probability of user playing a song as logistic, then maximize log likelihood of binary preference matrix, weighting positive observations by a function of plays, context, and recency X YUsers Songs • = bias for user • = bias for item • = regularization parameter • = user latent factor vector • = item latent factor vector [2] Johnson C. (2014) Logistic Matrix Factorization for Implicit Feedback Data NIPS Workshop on Distributed Matrix Computations Can also use Logistic Loss!
- 33. NLP Models onNews and Blogs
- 34. Playlist itself isa document Songs in playlist are words NLP Models work great on Playlists!
- 35.
- 36. •normalized item-vectors Songs ina Latent Space representation
- 37. •user-vector in samespace Songs in a Latent Space representation
- 38. Lesson 3: Don’t scaleuntil you need to
- 39. Scaling to 100%:Rollout Challenges ‣Create and publish 75M playlists every week ‣Downloading and processing Facebook images ‣Language translations
- 40. Scaling to 100%:Weekly refresh ‣Time sensitive updates ‣Refresh 75M playlists every Sunday night ‣Take timezones into account
- 41.
- 45. What’s next? Iterating oncontent quality and interface enhancements
- 46. Iterating on qualityand adding a feedback loop.
- 47. DW feedback comesat the expense of presentation bias.
- 48. Lesson 4: Users knowbest. In the end, AB Test everything!
- 49. Lesson 5 (finallesson!): Empower bottom-up innovation in your org and amazing things will happen.
- 50. Thank You! (btw, we’rehiring Machine Learning and Data Engineers, come chat with us!)