SlideShare a Scribd company logo
1 of 47
Download to read offline
Emmys, Oscars, and

Machine Learning Algorithms

@ Netflix Scale
November,	
  2013	
  
	
  

Xavier	
  Amatriain	
  
Director	
  -­‐	
  Algorithms	
  Engineering	
  -­‐	
  Ne<lix	
  

@xamat	
  
“In a simple Netlfix-style item recommender, we would
simply apply some form of matrix factorization (i.e NMF)”
What we were interested in:
§  High quality recommendations

Proxy question:
§  Accuracy in predicted rating
§  Improve by 10% = $1million!
From the Netflix Prize
to today

2006

2013
Big Data @Netflix

§  > 40M subscribers
§  Ratings: ~5M/day

Time

§  Searches: >3M/day

Geo-information

§  Plays: > 50M/day

Impressions Device Info
§  Streamed hours:
Metadata
Social

Ratings o  5B hours in Q3 2013
Member Behavior

Demographics
Smart Models

§  Regression models (Logistic, Linear,
Elastic nets)
§  SVD & other MF models
§  Factorization Machines
§  Restricted Boltzmann Machines
§  Markov Chains & other graph models
§  Clustering (from k-means to HDP)
§  Deep ANN
§  LDA
§  Association Rules
§  GBDT/RF
§  …
Deep-dive into
Netflix Algorithms
Rating Prediction
2007 Progress Prize
▪  Top 2 algorithms
▪  SVD - Prize RMSE: 0.8914
▪  RBM - Prize RMSE: 0.8990

▪  Linear blend Prize RMSE: 0.88
▪  Currently in use as part of Netflix’ rating prediction component
▪  Limitations
▪  Designed for 100M ratings, we have 5B ratings
▪  Not adaptable as users add ratings
▪  Performance issues
SVD - Definition

A[n x m] = U[n x r] Λ [ r x r] (V[m x r])T
▪ 

A: n x m matrix (e.g., n documents, m terms)

▪ 

U: n x r matrix (n documents, r concepts)

▪ 

▪ 

Λ: r x r diagonal matrix (strength of each ‘concept’) (r:
rank of the matrix)
V: m x r matrix (m terms, r concepts)
SVD - Properties
▪  ‘spectral decomposition’ of the matrix:

= u1

u2

x λ1

λ2

▪  ‘documents’, ‘terms’ and ‘concepts’:
§  U: document-to-concept similarity matrix
§  V: term-to-concept similarity matrix
§  Λ: its diagonal elements: ‘strength’ of each concept

x

v1
v2
SVD for Rating Prediction
§  User factor vectors
§  Baseline (bias)

and item-factors vectors
(user & item deviation

from average)

§  Predict rating as
§  SVD++ (Koren et. Al) asymmetric variation w.
implicit feedback

§  Where
§ 
are three item factor vectors
§  Users are not parametrized
Restricted Boltzmann Machines
§  Restrict the connectivity in ANN
to make learning easier.

§ 

Only one layer of hidden units.

§ 

§ 

Although multiple layers are
possible

No connections between
hidden units.

§  Hidden units are independent
given the visible states..
§  RBMs can be stacked to form
Deep Belief Networks (DBN) – 4th
generation of ANNs
What about the final prize ensembles?
▪  Our offline studies showed they were too
computationally intensive to scale
▪  Expected improvement not worth the
engineering effort

▪  Plus, focus had already shifted to other
issues that had more impact than rating
prediction...
Ranking

Ranking
Ranking
§  Ranking = Scoring + Sorting +
Filtering bags of movies for
presentation to a user

§  Factors
§  Accuracy
§  Novelty

Key algorithm, sorts titles in most
contexts

§  Diversity
§  Freshness

§  Goal: Find the best possible ordering of
a set of videos for a user within a
specific context in real-time

§  Scalability

§ 

§  Objective: maximize consumption &

“enjoyment”

§  …
Example: Two features, linear model

2	
  
3	
  
4	
  
5	
  

Popularity	
  

Linear	
  Model:	
  
frank(u,v)	
  =	
  w1	
  p(v)	
  +	
  w2	
  r(u,v)	
  +	
  b	
  

Final	
  Ranking	
  

Predicted	
  RaEng	
  

1	
  
Example: Two features, linear model

2	
  
3	
  
4	
  
5	
  

Popularity	
  

Final	
  Ranking	
  

Predicted	
  RaEng	
  

1	
  
Ranking
Learning to Rank
▪  Ranking is a very important problem in many contexts
(search, advertising, recommendations)
▪  Quality of ranking is measured using ranking metrics
such as NDCG, MRR, MAP, FPC…
▪  It is hard to optimize machine-learned models directly on
these measures
▪  They are not differentiable

▪  We would like to use the same measure for optimizing
and for the final evaluation
Learning to Rank Approaches
§  ML problem: construct ranking model from training data
1. 

Pointwise (Ordinal regression, Logistic regression, SVM, GBDT, …)
Loss function defined on individual relevance judgment

§ 

2. 

Pairwise (RankSVM, RankBoost, RankNet, FRank…)
§ 
§ 

3. 

Loss function defined on pair-wise preferences
Goal: minimize number of inversions in ranking

Listwise
§ 

Indirect Loss Function (RankCosine, ListNet…)

§ 

Directly optimize IR measures (NDCG, MRR, FCP…)
§ 

Genetic Programming or Simulated Annealing

§ 

Use boosting to optimize NDCG (Adarank)

§ 

Gradient descent on smoothed version (CLiMF, TFMAP, GAPfm @cikm13)

§ 

Iterative Coordinate Ascent (Direct Rank @kdd13)
Similarity
Similars
What is similarity?
§ 

Similarity can refer to different dimensions
§  Similar in metadata/tags
§  Similar in user play behavior
§  Similar in user rating behavior
§  …

§ 

You can learn a model for each of them and finally learn
an ensemble
Graph-based similarities

0.8
0.3
0.3
0.4
0.2

0.3
0.7
Example of graph-based similarity: SimRank
▪  SimRank (Jeh & Widom, 02): “two objects are
similar if they are referenced by similar objects.”
Final similarity ensemble
§ 

Come up with a score of play similarity, rating similarity,
tag-based similarity…

§ 

Combine them using an ensemble
§  Weights are learned using regression over existing response

§ 

The final concept of “similarity” responds to what users
vote as similar
Row Selection
Row	
  SelecEon	
  Inputs	
  
§  Visitor	
  data	
  
§  Video	
  history	
  
§  Queue	
  adds	
  
§  RaEngs	
  
§  Taste	
  preferences	
  
§  Predicted	
  raEngs	
  
§  PVR	
  ranking	
  
§  etc.	
  

§  Global data
§  Metadata
§  Popularity
§  etc.
Row	
  SelecEon	
  Core	
  Algorithm	
  
SelecEon	
  
constraints	
  
Candidate	
  
data	
  

Groupify	
  

Score/Select	
  
ParEal	
  
selecEon	
  

Group	
  
selecEon	
  
SelecEon	
  Constraints	
  
•  Many	
  business	
  rules	
  and	
  heurisEcs	
  
–  RelaEve	
  posiEon	
  of	
  groups	
  
–  Presence/absence	
  of	
  groups	
  
–  Deduping	
  rules	
  
–  Number	
  of	
  groups	
  of	
  a	
  given	
  type	
  

•  Determined	
  by	
  past	
  A/B	
  tests	
  
•  Evolved	
  over	
  Eme	
  from	
  a	
  simple	
  template	
  
GroupificaEon	
  
•  Ranking	
  of	
  Etles	
  according	
  to	
  
ranking	
  algo	
  
•  Filtering	
  
•  Deduping	
  
•  Forma^ng	
  

Candidate	
  
data	
  

•  SelecEon	
  of	
  evidence	
  
•  ...	
  

First	
  n	
  
selected	
  
groups	
  

Group	
  for	
  
posiEon	
  n
+1	
  
Scoring	
  and	
  SelecEon	
  
•  Features	
  
considered	
  

–  ARO	
  scorer	
  
–  Greedy	
  algorithm	
  

–  Quality	
  of	
  items	
  

–  Framework	
  supports	
  
other	
  methods	
  

–  Diversity	
  of	
  tags	
  
–  Quality	
  of	
  evidence	
  

§  Backtracking	
  

–  Freshness	
  

§  Approximate	
  integer	
  
programming	
  

–  Recency	
  
–  ...	
  

§  etc.	
  
	
  
Search Recommendations
Search Recommendation
CombinaEon	
  of	
  Different	
  models	
  
Play	
  Aaer	
  Search:	
  TransiEon	
  on	
  Play	
  aaer	
  Query	
  	
  
Play	
  Related	
  to	
  Search:	
  Similarity	
  between	
  user’s	
  (query,	
  
play)	
  	
  
…	
  

• 
• 
• 

Read this paper for a description of a similar system at LinkedIn
Unavailable Title Recommendations
Gamification Algorithms
rating gameGame
Rating
Mood Selection Algorithm
1.  Editorially selected moods
2.  Sort them according to users consumption
3.  Take into account freshness, diversity...
More data or
better models?
More data or better models?

Really?

Anand Rajaraman: Former Stanford Prof. &
Senior VP at Walmart
More data or better models?
Sometimes, it’s not
about more data
More data or better models?
[Banko and Brill, 2001]

Norvig: “Google does not
have better Algorithms,
only more Data”

Many features/
low-bias models
More data or better models?

Sometimes, it’s not
about more data
“Data without a sound approach = noise”
Conclusion
More data +
Smarter models +
More accurate metrics +
Better system architectures
Lots of room for improvement!
Xavier Amatriain (@xamat)
xavier@netflix.com

Thanks!

We’re hiring!

More Related Content

What's hot

What's hot (20)

Machine Learning to Grow the World's Knowledge
Machine Learning to Grow  the World's KnowledgeMachine Learning to Grow  the World's Knowledge
Machine Learning to Grow the World's Knowledge
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
 
Building Large-scale Real-world Recommender Systems - Recsys2012 tutorial
Building Large-scale Real-world Recommender Systems - Recsys2012 tutorialBuilding Large-scale Real-world Recommender Systems - Recsys2012 tutorial
Building Large-scale Real-world Recommender Systems - Recsys2012 tutorial
 
MLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@QuoraMLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@Quora
 
Machine learning the high interest credit card of technical debt [PWL]
Machine learning the high interest credit card of technical debt [PWL]Machine learning the high interest credit card of technical debt [PWL]
Machine learning the high interest credit card of technical debt [PWL]
 
Replicable Evaluation of Recommender Systems
Replicable Evaluation of Recommender SystemsReplicable Evaluation of Recommender Systems
Replicable Evaluation of Recommender Systems
 
Past present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry PerspectivePast present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry Perspective
 
Barcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons LearnedBarcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons Learned
 
Recommendation at Netflix Scale
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix Scale
 
Recommending the world's knowledge
Recommending the world's knowledgeRecommending the world's knowledge
Recommending the world's knowledge
 
Recsys 2014 Tutorial - The Recommender Problem Revisited
Recsys 2014 Tutorial - The Recommender Problem RevisitedRecsys 2014 Tutorial - The Recommender Problem Revisited
Recsys 2014 Tutorial - The Recommender Problem Revisited
 
kdd2015
kdd2015kdd2015
kdd2015
 
Machine Learning at Netflix Scale
Machine Learning at Netflix ScaleMachine Learning at Netflix Scale
Machine Learning at Netflix Scale
 
Efficient Top-N Recommendation by Linear Regression
Efficient Top-N Recommendation by Linear RegressionEfficient Top-N Recommendation by Linear Regression
Efficient Top-N Recommendation by Linear Regression
 
Netflix Recommendations - Beyond the 5 Stars
Netflix Recommendations - Beyond the 5 StarsNetflix Recommendations - Beyond the 5 Stars
Netflix Recommendations - Beyond the 5 Stars
 
Recsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and DeepakRecsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and Deepak
 
Aiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversionAiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversion
 
Scaling Recommendations at Quora (RecSys talk 9/16/2016)
Scaling Recommendations at Quora (RecSys talk 9/16/2016)Scaling Recommendations at Quora (RecSys talk 9/16/2016)
Scaling Recommendations at Quora (RecSys talk 9/16/2016)
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
 

Viewers also liked

Рекоменд. письмо_МинСпорт
Рекоменд. письмо_МинСпортРекоменд. письмо_МинСпорт
Рекоменд. письмо_МинСпорт
Natalia Sidorkina
 
Repurpose Your Career - 5 Steps to your Next Career Pivot
Repurpose Your Career - 5 Steps to your Next Career PivotRepurpose Your Career - 5 Steps to your Next Career Pivot
Repurpose Your Career - 5 Steps to your Next Career Pivot
Career Pivot
 

Viewers also liked (19)

The Top 10 Mistakes on Local Websites & How to Fix Them
The Top 10 Mistakes on Local Websites & How to Fix ThemThe Top 10 Mistakes on Local Websites & How to Fix Them
The Top 10 Mistakes on Local Websites & How to Fix Them
 
Gesture drawing
Gesture drawingGesture drawing
Gesture drawing
 
Social Media are serious or maybe #not
Social Media are serious or maybe #notSocial Media are serious or maybe #not
Social Media are serious or maybe #not
 
Trucs et astuces de recherche: voir les mots-clés en contexte
Trucs et astuces de recherche: voir les mots-clés en contexteTrucs et astuces de recherche: voir les mots-clés en contexte
Trucs et astuces de recherche: voir les mots-clés en contexte
 
Рекоменд. письмо_МинСпорт
Рекоменд. письмо_МинСпортРекоменд. письмо_МинСпорт
Рекоменд. письмо_МинСпорт
 
50 Culinary Hacks to make you a Kitchen Master
50 Culinary Hacks to make you a Kitchen Master50 Culinary Hacks to make you a Kitchen Master
50 Culinary Hacks to make you a Kitchen Master
 
The Importance of Being Awesome: From Words and Pictures to Code and Cyborgs
The Importance of Being Awesome: From Words and Pictures to Code and CyborgsThe Importance of Being Awesome: From Words and Pictures to Code and Cyborgs
The Importance of Being Awesome: From Words and Pictures to Code and Cyborgs
 
Professor Joanne Goodell presents Performance Based Funding
Professor Joanne Goodell presents Performance Based FundingProfessor Joanne Goodell presents Performance Based Funding
Professor Joanne Goodell presents Performance Based Funding
 
Intro
IntroIntro
Intro
 
Your Number One Fan: Turning Employees into Recruiters and Brand Ambassadors
Your Number One Fan: Turning Employees into Recruiters and Brand AmbassadorsYour Number One Fan: Turning Employees into Recruiters and Brand Ambassadors
Your Number One Fan: Turning Employees into Recruiters and Brand Ambassadors
 
Olympics team usa bsktball rio 2016 defenders
Olympics team usa bsktball rio 2016 defendersOlympics team usa bsktball rio 2016 defenders
Olympics team usa bsktball rio 2016 defenders
 
Hướng dẫn chuẩn bị cho kì thi JLPT 2014
Hướng dẫn chuẩn bị cho kì thi JLPT 2014Hướng dẫn chuẩn bị cho kì thi JLPT 2014
Hướng dẫn chuẩn bị cho kì thi JLPT 2014
 
SEO & Social Media Explained
SEO & Social Media ExplainedSEO & Social Media Explained
SEO & Social Media Explained
 
Introduction to CQRS
Introduction to CQRSIntroduction to CQRS
Introduction to CQRS
 
Conozca la bvc
Conozca la bvcConozca la bvc
Conozca la bvc
 
El diario
El diarioEl diario
El diario
 
Using Social Media As A Takeoff Point For Campaigns
Using Social Media As A Takeoff Point For CampaignsUsing Social Media As A Takeoff Point For Campaigns
Using Social Media As A Takeoff Point For Campaigns
 
Repurpose Your Career - 5 Steps to your Next Career Pivot
Repurpose Your Career - 5 Steps to your Next Career PivotRepurpose Your Career - 5 Steps to your Next Career Pivot
Repurpose Your Career - 5 Steps to your Next Career Pivot
 
Lire avec le Numérique
Lire avec le NumériqueLire avec le Numérique
Lire avec le Numérique
 

Similar to MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix

Webpage Personalization and User Profiling
Webpage Personalization and User ProfilingWebpage Personalization and User Profiling
Webpage Personalization and User Profiling
yingfeng
 
Download
DownloadDownload
Download
butest
 
Download
DownloadDownload
Download
butest
 
Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...
Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...
Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...
OpenSource Connections
 
Rated Ranking Evaluator: An Open Source Approach for Search Quality Evaluation
Rated Ranking Evaluator: An Open Source Approach for Search Quality EvaluationRated Ranking Evaluator: An Open Source Approach for Search Quality Evaluation
Rated Ranking Evaluator: An Open Source Approach for Search Quality Evaluation
Alessandro Benedetti
 

Similar to MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix (20)

acmsigtalkshare-121023190142-phpapp01.pptx
acmsigtalkshare-121023190142-phpapp01.pptxacmsigtalkshare-121023190142-phpapp01.pptx
acmsigtalkshare-121023190142-phpapp01.pptx
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
Dataiku at SF DataMining Meetup - Kaggle Yandex Challenge
Dataiku at SF DataMining Meetup - Kaggle Yandex ChallengeDataiku at SF DataMining Meetup - Kaggle Yandex Challenge
Dataiku at SF DataMining Meetup - Kaggle Yandex Challenge
 
Rokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptxRokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptx
 
Rokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptxRokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptx
 
Webpage Personalization and User Profiling
Webpage Personalization and User ProfilingWebpage Personalization and User Profiling
Webpage Personalization and User Profiling
 
Download
DownloadDownload
Download
 
Download
DownloadDownload
Download
 
Recommender systems in practice
Recommender systems in practiceRecommender systems in practice
Recommender systems in practice
 
Buidling large scale recommendation engine
Buidling large scale recommendation engineBuidling large scale recommendation engine
Buidling large scale recommendation engine
 
Weka bike rental
Weka bike rentalWeka bike rental
Weka bike rental
 
JavaScript and Artificial Intelligence by Aatman & Sagar - AhmedabadJS
JavaScript and Artificial Intelligence by Aatman & Sagar - AhmedabadJSJavaScript and Artificial Intelligence by Aatman & Sagar - AhmedabadJS
JavaScript and Artificial Intelligence by Aatman & Sagar - AhmedabadJS
 
Nose Dive into Apache Spark ML
Nose Dive into Apache Spark MLNose Dive into Apache Spark ML
Nose Dive into Apache Spark ML
 
Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...
Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...
Haystack 2019 - Rated Ranking Evaluator: an Open Source Approach for Search Q...
 
Rated Ranking Evaluator: An Open Source Approach for Search Quality Evaluation
Rated Ranking Evaluator: An Open Source Approach for Search Quality EvaluationRated Ranking Evaluator: An Open Source Approach for Search Quality Evaluation
Rated Ranking Evaluator: An Open Source Approach for Search Quality Evaluation
 
Florian Douetteau @ Dataiku
Florian Douetteau @ DataikuFlorian Douetteau @ Dataiku
Florian Douetteau @ Dataiku
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing Recommendations
 
Rated Ranking Evaluator: an Open Source Approach for Search Quality Evaluation
Rated Ranking Evaluator: an Open Source Approach for Search Quality EvaluationRated Ranking Evaluator: an Open Source Approach for Search Quality Evaluation
Rated Ranking Evaluator: an Open Source Approach for Search Quality Evaluation
 
Lucene/Solr Revolution 2015: Where Search Meets Machine Learning
Lucene/Solr Revolution 2015: Where Search Meets Machine LearningLucene/Solr Revolution 2015: Where Search Meets Machine Learning
Lucene/Solr Revolution 2015: Where Search Meets Machine Learning
 
Lucene/Solr Revolution 2015: Where Search Meets Machine Learning
Lucene/Solr Revolution 2015: Where Search Meets Machine LearningLucene/Solr Revolution 2015: Where Search Meets Machine Learning
Lucene/Solr Revolution 2015: Where Search Meets Machine Learning
 

More from Xavier Amatriain

More from Xavier Amatriain (18)

Data/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealthData/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealth
 
AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19
 
AI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 updateAI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 update
 
AI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approachAI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approach
 
Lessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systemsLessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systems
 
AI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for EveryoneAI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for Everyone
 
Towards online universal quality healthcare through AI
Towards online universal quality healthcare through AITowards online universal quality healthcare through AI
Towards online universal quality healthcare through AI
 
From one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategyFrom one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategy
 
Learning to speak medicine
Learning to speak medicineLearning to speak medicine
Learning to speak medicine
 
ML to cure the world
ML to cure the worldML to cure the world
ML to cure the world
 
Medical advice as a Recommender System
Medical advice as a Recommender SystemMedical advice as a Recommender System
Medical advice as a Recommender System
 
Machine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora ExampleMachine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora Example
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
 
Lean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems
 
Kdd 2014 Tutorial - the recommender problem revisited
Kdd 2014 Tutorial -  the recommender problem revisitedKdd 2014 Tutorial -  the recommender problem revisited
Kdd 2014 Tutorial - the recommender problem revisited
 

Recently uploaded

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 

Recently uploaded (20)

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 

MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix

  • 1. Emmys, Oscars, and Machine Learning Algorithms @ Netflix Scale November,  2013     Xavier  Amatriain   Director  -­‐  Algorithms  Engineering  -­‐  Ne<lix   @xamat  
  • 2. “In a simple Netlfix-style item recommender, we would simply apply some form of matrix factorization (i.e NMF)”
  • 3. What we were interested in: §  High quality recommendations Proxy question: §  Accuracy in predicted rating §  Improve by 10% = $1million!
  • 4. From the Netflix Prize to today 2006 2013
  • 5. Big Data @Netflix §  > 40M subscribers §  Ratings: ~5M/day Time §  Searches: >3M/day Geo-information §  Plays: > 50M/day Impressions Device Info §  Streamed hours: Metadata Social Ratings o  5B hours in Q3 2013 Member Behavior Demographics
  • 6. Smart Models §  Regression models (Logistic, Linear, Elastic nets) §  SVD & other MF models §  Factorization Machines §  Restricted Boltzmann Machines §  Markov Chains & other graph models §  Clustering (from k-means to HDP) §  Deep ANN §  LDA §  Association Rules §  GBDT/RF §  …
  • 9. 2007 Progress Prize ▪  Top 2 algorithms ▪  SVD - Prize RMSE: 0.8914 ▪  RBM - Prize RMSE: 0.8990 ▪  Linear blend Prize RMSE: 0.88 ▪  Currently in use as part of Netflix’ rating prediction component ▪  Limitations ▪  Designed for 100M ratings, we have 5B ratings ▪  Not adaptable as users add ratings ▪  Performance issues
  • 10. SVD - Definition A[n x m] = U[n x r] Λ [ r x r] (V[m x r])T ▪  A: n x m matrix (e.g., n documents, m terms) ▪  U: n x r matrix (n documents, r concepts) ▪  ▪  Λ: r x r diagonal matrix (strength of each ‘concept’) (r: rank of the matrix) V: m x r matrix (m terms, r concepts)
  • 11. SVD - Properties ▪  ‘spectral decomposition’ of the matrix: = u1 u2 x λ1 λ2 ▪  ‘documents’, ‘terms’ and ‘concepts’: §  U: document-to-concept similarity matrix §  V: term-to-concept similarity matrix §  Λ: its diagonal elements: ‘strength’ of each concept x v1 v2
  • 12. SVD for Rating Prediction §  User factor vectors §  Baseline (bias) and item-factors vectors (user & item deviation from average) §  Predict rating as §  SVD++ (Koren et. Al) asymmetric variation w. implicit feedback §  Where §  are three item factor vectors §  Users are not parametrized
  • 13. Restricted Boltzmann Machines §  Restrict the connectivity in ANN to make learning easier. §  Only one layer of hidden units. §  §  Although multiple layers are possible No connections between hidden units. §  Hidden units are independent given the visible states.. §  RBMs can be stacked to form Deep Belief Networks (DBN) – 4th generation of ANNs
  • 14. What about the final prize ensembles? ▪  Our offline studies showed they were too computationally intensive to scale ▪  Expected improvement not worth the engineering effort ▪  Plus, focus had already shifted to other issues that had more impact than rating prediction...
  • 16. Ranking §  Ranking = Scoring + Sorting + Filtering bags of movies for presentation to a user §  Factors §  Accuracy §  Novelty Key algorithm, sorts titles in most contexts §  Diversity §  Freshness §  Goal: Find the best possible ordering of a set of videos for a user within a specific context in real-time §  Scalability §  §  Objective: maximize consumption & “enjoyment” §  …
  • 17. Example: Two features, linear model 2   3   4   5   Popularity   Linear  Model:   frank(u,v)  =  w1  p(v)  +  w2  r(u,v)  +  b   Final  Ranking   Predicted  RaEng   1  
  • 18. Example: Two features, linear model 2   3   4   5   Popularity   Final  Ranking   Predicted  RaEng   1  
  • 20. Learning to Rank ▪  Ranking is a very important problem in many contexts (search, advertising, recommendations) ▪  Quality of ranking is measured using ranking metrics such as NDCG, MRR, MAP, FPC… ▪  It is hard to optimize machine-learned models directly on these measures ▪  They are not differentiable ▪  We would like to use the same measure for optimizing and for the final evaluation
  • 21. Learning to Rank Approaches §  ML problem: construct ranking model from training data 1.  Pointwise (Ordinal regression, Logistic regression, SVM, GBDT, …) Loss function defined on individual relevance judgment §  2.  Pairwise (RankSVM, RankBoost, RankNet, FRank…) §  §  3.  Loss function defined on pair-wise preferences Goal: minimize number of inversions in ranking Listwise §  Indirect Loss Function (RankCosine, ListNet…) §  Directly optimize IR measures (NDCG, MRR, FCP…) §  Genetic Programming or Simulated Annealing §  Use boosting to optimize NDCG (Adarank) §  Gradient descent on smoothed version (CLiMF, TFMAP, GAPfm @cikm13) §  Iterative Coordinate Ascent (Direct Rank @kdd13)
  • 24. What is similarity? §  Similarity can refer to different dimensions §  Similar in metadata/tags §  Similar in user play behavior §  Similar in user rating behavior §  … §  You can learn a model for each of them and finally learn an ensemble
  • 26. Example of graph-based similarity: SimRank ▪  SimRank (Jeh & Widom, 02): “two objects are similar if they are referenced by similar objects.”
  • 27. Final similarity ensemble §  Come up with a score of play similarity, rating similarity, tag-based similarity… §  Combine them using an ensemble §  Weights are learned using regression over existing response §  The final concept of “similarity” responds to what users vote as similar
  • 29. Row  SelecEon  Inputs   §  Visitor  data   §  Video  history   §  Queue  adds   §  RaEngs   §  Taste  preferences   §  Predicted  raEngs   §  PVR  ranking   §  etc.   §  Global data §  Metadata §  Popularity §  etc.
  • 30. Row  SelecEon  Core  Algorithm   SelecEon   constraints   Candidate   data   Groupify   Score/Select   ParEal   selecEon   Group   selecEon  
  • 31. SelecEon  Constraints   •  Many  business  rules  and  heurisEcs   –  RelaEve  posiEon  of  groups   –  Presence/absence  of  groups   –  Deduping  rules   –  Number  of  groups  of  a  given  type   •  Determined  by  past  A/B  tests   •  Evolved  over  Eme  from  a  simple  template  
  • 32. GroupificaEon   •  Ranking  of  Etles  according  to   ranking  algo   •  Filtering   •  Deduping   •  Forma^ng   Candidate   data   •  SelecEon  of  evidence   •  ...   First  n   selected   groups   Group  for   posiEon  n +1  
  • 33. Scoring  and  SelecEon   •  Features   considered   –  ARO  scorer   –  Greedy  algorithm   –  Quality  of  items   –  Framework  supports   other  methods   –  Diversity  of  tags   –  Quality  of  evidence   §  Backtracking   –  Freshness   §  Approximate  integer   programming   –  Recency   –  ...   §  etc.    
  • 35. Search Recommendation CombinaEon  of  Different  models   Play  Aaer  Search:  TransiEon  on  Play  aaer  Query     Play  Related  to  Search:  Similarity  between  user’s  (query,   play)     …   •  •  •  Read this paper for a description of a similar system at LinkedIn
  • 38. rating gameGame Rating Mood Selection Algorithm 1.  Editorially selected moods 2.  Sort them according to users consumption 3.  Take into account freshness, diversity...
  • 40. More data or better models? Really? Anand Rajaraman: Former Stanford Prof. & Senior VP at Walmart
  • 41. More data or better models? Sometimes, it’s not about more data
  • 42. More data or better models? [Banko and Brill, 2001] Norvig: “Google does not have better Algorithms, only more Data” Many features/ low-bias models
  • 43. More data or better models? Sometimes, it’s not about more data
  • 44. “Data without a sound approach = noise”
  • 46. More data + Smarter models + More accurate metrics + Better system architectures Lots of room for improvement!