Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Recommending for the World
Yves Raimond (@moustaki)
03/16
Research/Engineering Manager
Search & Recommendations Algorithm ...
Some background
● > 75M members
● > 190 countries
● > 3.7B hours of content streamed every
month
● > 1000 device types
● 36% of peak US do...
Recommendations @ Netflix
Goal
Help members find content to watch and enjoy
to maximize satisfaction and retention
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪ …
Models & Algorithms
Going global
● How do we make sure all these
algorithms are ready to work on a
global scale?
● Led us to investigate many
...
Challenge 1: Uneven Video Availability
US
FR
US
FR
1,000 users
100 users
0
...
...Co-occurrences
! =
?????
R ≈ UM
! =
What would have happened if the two
videos were available to the same
members?
US
FR
1,000 users
100 users
100,000 users
10 users
≈
US
FR
2016-01-01 2016-01-02
Newly available
What would have happened if the two
videos were available to the same
members for the same amount of time?
Challenge 2: Cultural Awareness
Two questions
1) Similar users, in two different countries. Should they get similar
recommendations?
2) Overall, should recommendations be different for users in Japan vs users
in Argentina? What about new users?
Regional models
Group countries into regions, and train
individual models on each region.
Pros
● Easy!
● Catalog can be co...
Sparsity and global models
Only a small fraction of users from all countries
would be interested in these titles. Models t...
Global communities - Anime
Global communities - Bollywood
Local taste vs personal taste
● Personal taste benefits from global algorithms
○ Taste patterns travel globally
● Local ta...
Challenge 3: Language
Instant search
● Ranking entities for partial queries
● Optimizing for the minimum number of interactions needed to find s...
Hangul alphabet, 3 syllables but
requires 7 (2 + 3 + 2) interactions
One interaction
Language & Recommendations
≈+
US US/AU FR
?
Challenge 4: Does it even work?
Tracking quality
● Objective: build algorithms that work equally well for all our members
● Looking at global metrics migh...
Conclusion
● Catalog differences, cultural awareness, language and metrics
● Worldwide communities of interest for better recommendat...
Questions?
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Recommending for the World
Upcoming SlideShare
Loading in …5
×

Recommending for the World

The Netflix experience is driven by a number of Machine Learning algorithms: personalized ranking, page generation, search, similarity, ratings, etc. On the 6th of January, we simultaneously launched Netflix in 130 new countries around the world, which brings the total to over 190 countries. Preparing for such a rapid expansion while ensuring each algorithm was ready to work seamlessly created new challenges for our recommendation and search teams. In this post, we highlight the four most interesting challenges we’ve encountered in making our algorithms operate globally and, most importantly, how this improved our ability to connect members worldwide with stories they'll love.

  • Be the first to comment

Recommending for the World

  1. 1. Recommending for the World Yves Raimond (@moustaki) 03/16 Research/Engineering Manager Search & Recommendations Algorithm Engineering Netflix
  2. 2. Some background
  3. 3. ● > 75M members ● > 190 countries ● > 3.7B hours of content streamed every month ● > 1000 device types ● 36% of peak US downstream traffic Netflix scale
  4. 4. Recommendations @ Netflix
  5. 5. Goal Help members find content to watch and enjoy to maximize satisfaction and retention
  6. 6. ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ … Models & Algorithms
  7. 7. Going global ● How do we make sure all these algorithms are ready to work on a global scale? ● Led us to investigate many challenges, leading to many rollouts of new algorithms, over the last year ○ Tech blog post ○ Company blog post
  8. 8. Challenge 1: Uneven Video Availability
  9. 9. US FR
  10. 10. US FR 1,000 users 100 users
  11. 11. 0 ... ...Co-occurrences
  12. 12. ! = ?????
  13. 13. R ≈ UM
  14. 14. ! =
  15. 15. What would have happened if the two videos were available to the same members?
  16. 16. US FR 1,000 users 100 users 100,000 users 10 users
  17. 17.
  18. 18. US FR 2016-01-01 2016-01-02 Newly available
  19. 19. What would have happened if the two videos were available to the same members for the same amount of time?
  20. 20. Challenge 2: Cultural Awareness
  21. 21. Two questions
  22. 22. 1) Similar users, in two different countries. Should they get similar recommendations?
  23. 23. 2) Overall, should recommendations be different for users in Japan vs users in Argentina? What about new users?
  24. 24. Regional models Group countries into regions, and train individual models on each region. Pros ● Easy! ● Catalog can be constrained to be relatively uniform ● Solves question 2 Cons ● Doesn’t solve question 1 ● How to define groupings? ● Algorithms x A/B model variants x regions ● Biggest country in the region will dominate ● Sparsity
  25. 25. Sparsity and global models Only a small fraction of users from all countries would be interested in these titles. Models trained locally perform poorly -- lack of data. Pooling data from all countries discovers a worldwide community of interest, making recommendations better for these users.
  26. 26. Global communities - Anime
  27. 27. Global communities - Bollywood
  28. 28. Local taste vs personal taste ● Personal taste benefits from global algorithms ○ Taste patterns travel globally ● Local taste still needs to be taken into account in order to solve 2) ● Incorporate signals and priors capturing local taste patterns (e.g. country and language)
  29. 29. Challenge 3: Language
  30. 30. Instant search ● Ranking entities for partial queries ● Optimizing for the minimum number of interactions needed to find something ● Different languages involve very different interaction patterns ● How to automatically detect and adapt to such patterns in newly introduced languages?
  31. 31. Hangul alphabet, 3 syllables but requires 7 (2 + 3 + 2) interactions
  32. 32. One interaction
  33. 33. Language & Recommendations ≈+ US US/AU FR ?
  34. 34. Challenge 4: Does it even work?
  35. 35. Tracking quality ● Objective: build algorithms that work equally well for all our members ● Looking at global metrics might hide issues with small subsets of members ● How to identify sub-optimality for a subset of our members? ○ Language, country, device, … ○ Slicing on all dimensions lead to sparsity and noisiness ○ Automatically grouping observations for the purpose of automatically detecting outliers ● Metrics, instrumentation and monitoring ○ Detect problems ○ Highlight areas of improvement
  36. 36. Conclusion
  37. 37. ● Catalog differences, cultural awareness, language and metrics ● Worldwide communities of interest for better recommendations ○ Thinking about global actually led us to test and release better algorithms ○ But also need to capture signals and priors related to cultural preferences ● Quickly finding entities in any language ● Detecting issues at a finer grain ● … Still a lot of work to do! ○ Better global algorithms… (Now that we have data) ○ Better cultural/language awareness ○ Better user and item cold start ○ Reactiveness ○ Better algorithms for anomaly detection Conclusion
  38. 38. Questions?

×