Upcoming SlideShare
From Idea to Execution: Spotify's Discover Weekly
- From Idea to Execution: Spotify’s Discover Weekly Chris Johnson :: @MrChrisJohnson Edward Newett :: @scaladaze DataEngConf • NYC • Nov 2015 Or: 5 lessons in building recommendation products at scale
- Who are We?? Chris Johnson Edward Newett
- Spotify in Numbers • Started in 2006, now available in 58 markets • 75+ Million active users, 20 Million paying subscribers • 30+ Million songs, 20,000 new songs added per day • 1.5 Billion user generated playlists • 1 TB user data logged per day • 1,700 node Hadoop cluster • 10,000+ Hadoop jobs run daily
- Challenge: 30M songs… how do we recommend music to users?
- Discover
- Radio
- Related Artists
- Discover Weekly • Started in 2006, now available in 58 markets • 75+ Million active users, 20 Million paying subscribers • 30+ Million songs, 20,000 new songs added per day • 1.5 Billion user generated playlists • 1 TB user data logged per day • 1,700 node Hadoop cluster • 10,000+ Hadoop jobs run daily
- The Road to Discover Weekly
- 2013 :: Discover Page v1.0 • Personalized News Feed of recommendations • Artists, Album Reviews, News Articles, New Releases, Upcoming Concerts, Social Recommendations, Playlists… • Required a lot of attention and digging to engage with recommendations • No organization of content
- 2014 :: Discover Page v2.0 • Recommendations grouped into strips (a la Netflix) • Limited to Albums and New Releases • More organized than News-Feed but still requires active interaction
- Insight: users spending more time on editorial Browse playlists than Discover.
- Idea: combine the personalized experience of Discover with the lean- back ease of Browse
- Meanwhile… 2014 Year In Music
- Play it forward: Same content as the Discover Page but.. a playlist
- Lesson 1: Be data driven from start to finish
- Slide from Dan McKinley - Etsy 2008 2012 2015
- • Reach: How many users are you reaching • Depth: For the users you reach, what is the depth of reach. • Retention: For the users you reach, how many do you retain? Define success metrics BEFORE you release your test
- • Reach: DW WAU / Spotify WAU • Depth: DW Time Spent / Spotify WAU • Retention: DW week-over-week retention Discover Weekly Key Success Metrics
- 2008 2012 2015 Slide from Dan McKinley - Etsy
- Step 1: Prototype (employee test)
- Step 1: Prototype (employee test)
- Results of Employee Test were very positive!
- 2008 2012 2015 Slide from Dan McKinley - Etsy
- Step 2: Release AB Test to 1% of Users
- Google Form 1% Results
- Personalized image resulted in 10% lift in WAU • Initial 0.5% user test • 1% Spaceman image • 1% Personalized image
- Lesson 2: Reuse existing infrastructure in creative ways
- Discover Weekly Data Flow
- Recommendation Models
- 1 0 0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 •Aggregate all (user, track) streams into a large matrix •Goal: Approximate binary preference matrix by inner product of 2 smaller matrices by minimizing the weighted RMSE (root mean squared error) using a function of plays, context, and recency as weight X YUsers Songs • = bias for user • = bias for item • = regularization parameter • = 1 if user streamed track else 0 • • = user latent factor vector • = item latent factor vector [1] Hu Y. & Koren Y. & Volinsky C. (2008) Collaborative Filtering for Implicit Feedback Datasets 8th IEEE International Conference on Data Mining Implicit Matrix Factorization
- 1 0 0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 •Aggregate all (user, track) streams into a large matrix •Goal: Model probability of user playing a song as logistic, then maximize log likelihood of binary preference matrix, weighting positive observations by a function of plays, context, and recency X YUsers Songs • = bias for user • = bias for item • = regularization parameter • = user latent factor vector • = item latent factor vector [2] Johnson C. (2014) Logistic Matrix Factorization for Implicit Feedback Data NIPS Workshop on Distributed Matrix Computations Can also use Logistic Loss!
- NLP Models on News and Blogs
- Playlist itself is a document Songs in playlist are words NLP Models work great on Playlists!
- [3] http://benanne.github.io/2014/08/05/spotify-cnns.html Deep Learning on Audio
- •normalized item-vectors Songs in a Latent Space representation
- •user-vector in same space Songs in a Latent Space representation
- Lesson 3: Don’t scale until you need to
- Scaling to 100%: Rollout Challenges ‣Create and publish 75M playlists every week ‣Downloading and processing Facebook images ‣Language translations
- Scaling to 100%: Weekly refresh ‣Time sensitive updates ‣Refresh 75M playlists every Sunday night ‣Take timezones into account
- Discover Weekly publishing flow
- What’s next? Iterating on content quality and interface enhancements
- Iterating on quality and adding a feedback loop.
- DW feedback comes at the expense of presentation bias.
- Lesson 4: Users know best. In the end, AB Test everything!
- Lesson 5 (final lesson!): Empower bottom-up innovation in your org and amazing things will happen.
- Thank You! (btw, we’re hiring Machine Learning and Data Engineers, come chat with us!)
Login to see the comments