Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spotify's Music Recommendations Lambda Architecture

3,051 views

Published on

Spotify's Collaborative Filtering platform powers our Discover Page. In light of the fact, that we have been adding one new user every three seconds, it is paramount that we do things in real-time. We redesigned our recommendation system and added a Storm based real-time platform.

Published in: Software, Internet
  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi,@Esh : Thank you for the great presentation. Can you please let me know if you posted the recorded video anywhere? Thanks
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • @建 徐 They are words that appear in the context of star. If you went through a piece of text about the sun, it is likely that you might similar words. Indicating similarity between star and sun.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • hi,thanks for sharing. in page 8,what's the purpose of the blue words?are the blue words represent the similay words of "star"
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Spotify's Music Recommendations Lambda Architecture

  1. 1. Spotify’s Music Recommendations Lambda Architecture Esh Kumar @eshvk Emily Samuels @emilymsa
  2. 2. Overview ‣ Why Lambda? ‣ Use Case: Discover Recommendations • Batch Architecture • Real-time Architecture • Challenges ‣ Future Work
  3. 3. Why Lambda? • 1 new user every 3 seconds. • Contextual, time based recs more & more important
  4. 4. Discover Recs
  5. 5. The Discover Page Algorithmically generated fresh recs for users.
  6. 6. The Discover Batch Pipeline
  7. 7. Machine Learning Deep Dive
  8. 8. Word2Vec Words with similar contexts have similar meaning
  9. 9. Word2Vec King – Man + Woman = Queen
  10. 10. Annoy • Approximate Nearest Neighbors Oh Yeah! • https://github.com/s potify/annoy
  11. 11. Batch Architecture Strengths
  12. 12. Intro to Storm
  13. 13. Storm • Distributed real-time computation system
  14. 14. Storm @ Spotify
  15. 15. Real-time Architecture
  16. 16. • Workers die -> Cascading JVM Process death • Memcache flakiness • Cassandra JVM problems due to write/overwrite pattern Challenges
  17. 17. Future/Ongoing Work • Simplify the topology • Keep listens for 24 hours • Ongoing work on other real time personalization features.
  18. 18. Questions Esh Kumar eshvk@spotify.com Emily Samuels esamuels@spotify.com

×