Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- Large Graph Mining – Patterns, tool... by BigMine 6050 views
- Processing Reachability Queries wit... by BigMine 3599 views
- Challenging Problems for Scalable M... by BigMine 4434 views
- Netflix Case Study by Kikuyu Daniels 172566 views
- Netflix Case Study by Julien Guitton 39454 views
- The Netflix Way to deal with Big Da... by Monal Daxini 2496 views

5,642 views

Published on

In this talk I will discuss the different approaches we follow to deal with these large streams of data in order to extract information for personalizing our service. I will describe some of the machine learning models used, as well as the architectures that allow us to combine complex offline batch processes with real-time data streams.

No Downloads

Total views

5,642

On SlideShare

0

From Embeds

0

Number of Embeds

2,510

Shares

0

Downloads

196

Comments

0

Likes

13

No embeds

No notes for slide

- 1. Big & Personal: the data and the models behind Netflix recommendations
- 2. Outline 1. The Netflix Prize & the Recommendation Problem 2. Anatomy of Netflix Personalization 3. Data & Models 4. More data or better Models?
- 3. What we were interested in: ■ High quality recommendations Proxy question: ■ Accuracy in predicted rating ■ Improve by 10% = $1million! ● Top 2 algorithms still in production Results SVD RBM
- 4. What about the final prize ensembles? ■ Our offline studies showed they were too computationally intensive to scale ■ Expected improvement not worth the engineering effort ■ Plus…. Focus had already shifted to other issues that had more impact than rating prediction.
- 5. Change of focus 2006 2013
- 6. Anatomy of Netflix Personalization Everything is a Recommendation
- 7. Everything is personalized Note: Recommendations are per household, not individual user Ranking
- 8. Top 10 Personalization awareness Diversity DadAll SonDaughterDad&Mom MomAll Daughter MomAll?
- 9. Support for Recommendations Social Support
- 10. Social Recommendations
- 11. Genre rows ■ Personalized genre rows focus on user interest ■ Also provide context and “evidence” ■ Important for member satisfaction – moving personalized rows to top on devices increased retention ■ How are they generated? ■ Implicit: based on user’s recent plays, ratings, & other interactions ■ Explicit taste preferences ■ Hybrid:combine the above ■ Also take into account: ■ Freshness - has this been shown before? ■ Diversity– avoid repeating tags and genres, limit number of TV genres, etc.
- 12. Genres - personalization
- 13. ■ Displayed in many different contexts ■ In response to user actions/context (search, queue add…) ■ More like… rows Similars
- 14. Data & Models
- 15. Big Data @Netflix ■ Almost 40M subscribers ■ Ratings: 4M/day ■ Searches: 3M/day ■ Plays: 30M/day ■ 2B hours streamed in Q4 2011 ■ 1B hours in June 2012 ■ > 4B hours in Q1 2013 Member Behavior Geo-informationTime Impressions Device Info Metadata Social
- 16. Smart Models ■ Logistic/linear regression ■ Elastic nets ■ SVD and other MF models ■ Factorization Machines ■ Restricted Boltzmann Machines ■ Markov Chains ■ Different clustering approaches ■ LDA ■ Association Rules ■ Gradient Boosted Decision Trees/Random Forests ■ …
- 17. SVD X[n x m] = U[n x r] S [ r x r] (V[m x r] )T ■ X: m x n matrix (e.g., m users, n videos) ■ U: m x r matrix (m users, r factors) ■ S: r x r diagonal matrix (strength of each ‘factor’) (r: rank of the matrix) ■ V: r x n matrix (n videos, r factor)
- 18. SVD for Rating Prediction ■ User factor vectors and item-factors vector ■ Baseline (bias) (user & item deviation from average) ■ Predict rating as ■ SVD++ (Koren et. Al) asymmetric variation w. implicit feedback ■ Where ■ are three item factor vectors ■ Users are not parametrized, but rather represented by: ■ R(u): items rated by user u ■ N(u): items for which the user has given implicit preference (e.g. rated vs. not rated)
- 19. Simon Funk’s SVD ■ One of the most interesting findings during the Netflix Prize came out of a blog post ■ Incremental, iterative, and approximate way to compute the SVD using gradient descent
- 20. Restricted Boltzmann Machines ■ Restrict the connectivity in ANN to make learning easier. ■ Only one layer of hidden units. ■ Although multiple layers are possible ■ No connections between hidden units. ■ Hidden units are independent given the visible states.. ■ RBMs can be stacked to form Deep Belief Networks (DBN) – 4th generation of ANNs hidden i j visible
- 21. RBM for the Netflix Prize
- 22. Ranking Key algorithm, sorts titles in most contexts
- 23. Ranking ■ Ranking = Scoring + Sorting + Filtering bags of movies for presentation to a user ■ Goal: Find the best possible ordering of a set of videos for a user within a specific context in real-time ■ Objective: maximize consumption ■ Aspirations: Played & “enjoyed” titles have best score ■ Akin to CTR forecast for ads/search results ■ Factors ■ Accuracy ■ Novelty ■ Diversity ■ Freshness ■ Scalability ■ …
- 24. Example: Two features, linear model
- 25. Example: Two features, linear model
- 26. Ranking
- 27. Ranking
- 28. Ranking Novelty Diversity Freshness Accuracy Scalability
- 29. Learning to rank ■ Machine learning problem: goal is to construct ranking model from training data ■ Training data can have partial order or binary judgments (relevant/not relevant). ■ Resulting order of the items typically induced from a numerical score ■ Learning to rank is a key element for personalization ■ You can treat the problem as a standard supervised classification problem
- 30. Learning to Rank Approaches 1. Pointwise ■ Ranking function minimizes loss function defined on individual relevance judgment ■ Ranking score based on regression or classification ■ Ordinal regression, Logistic regression, SVM, GBDT, … 2. Pairwise ■ Loss function is defined on pair-wise preferences ■ Goal: minimize number of inversions in ranking ■ Ranking problem is then transformed into the binary classification problem ■ RankSVM, RankBoost, RankNet, FRank…
- 31. Learning to rank - metrics ■ Quality of ranking measured using metrics as ■ Normalized Discounted Cumulative Gain ■ Mean Reciprocal Rank (MRR) ■ Fraction of Concordant Pairs (FCP) ■ Others… ■ But, it is hard to optimize machine-learned models directly on these measures (they are not differentiable) ■ Recent research on models that directly optimize ranking measures
- 32. Learning to Rank Approaches 3. Listwise a. Indirect Loss Function ■ RankCosine: similarity between ranking list and ground truth as loss function ■ ListNet: KL-divergence as loss function by defining a probability distribution ■ Problem: optimization of listwise loss function may not optimize IR metrics b. Directly optimizing IR measures (difficult since they are not differentiable) ■ Directly optimize IR measures through Genetic Programming or Simulated Annealing ■ Gradient descent on smoothed version of objective function (e.g. CLiMF at Recsys 2012 or TFMAP at SIGIR 2012) ■ SVM-MAP relaxes the MAP metric by adding it to the SVM constraints ■ AdaRank uses boosting to optimize NDCG
- 33. Other research questions we are interested on ● Row selection ○ How to select and rank lists of “related” items imposing inter- group diversity, avoiding duplicates... ● Diversity ○ Can we increase diversity while preserving relevance in a way that we optimize user response? ● Similarity ○ How to compute optimal and personalized similarity between items by using different data that can range from play histories to item metadata ● Context-aware recommendations ● Mood and session intent inference ● ...
- 34. More data or better models?
- 35. More data or better models? Really? Anand Rajaraman: Stanford & Senior VP at Walmart Global eCommerce (former Kosmix)
- 36. Sometimes, it’s not about more data More data or better models?
- 37. [Banko and Brill, 2001] Norvig: “Google does not have better Algorithms, only more Data” Many features/ low-bias models More data or better models?
- 38. More data or better models? Sometimes, it’s not about more data
- 39. X More data or better models?
- 40. Data without a sound approach = noise
- 41. Conclusions
- 42. The Personalization Problem ■ The Netflix Prize simplified the recommendation problem to predicting ratings ■ But… ■ User ratings are only one of the many data inputs we have ■ Rating predictions are only part of our solution ■ Other algorithms such as ranking or similarity are very important ■ We can reformulate the recommendation problem ■ Function to optimize: probability a user chooses something and enjoys it enough to come back to the service
- 43. More data + Better models + More accurate metrics + Better approaches & architectures Lots of room for improvement!
- 44. Thanks! Xavier Amatriain (@xamat) xavier@netflix.com We’re hiring!

No public clipboards found for this slide

Be the first to comment