SlideShare a Scribd company logo
1 of 57
Recent Trends in
Personalization
at Netflix
Justin Basilico
RecSys 2020 Expo
2020-09-24
@JustinBasilico
Why do we personalize?
Help members find content
to watch and enjoy to maximize
member satisfaction and retention
Spark joy
What do we personalize?
Ordering of videos is personalized
From how we rank
Ranking
Selection and placement of rows is personalized
... to how we construct a pageRows
... to how we respond to queries
Search query & result recommendation
... to what images we suggest
Frame recommendation for artists
Personalized artwork selection
... and then select
... to how we reach out
Message personalization
Everything is a recommendation!
Isn’t this solved yet?
○ Every person is unique with a variety of interests
… and sometimes they share profiles
○ Help people find what they want when they’re not sure what they want
○ Large datasets but small data per user
… and potentially biased by the output of your system
○ Cold-start problems on all sides
○ Non-stationary, context-dependent, mood-dependent, ...
○ More than just accuracy: Diversity, novelty, freshness, fairness, ...
○ ...
No, personalization is hard!
So what are you doing about it?
Some recent avenues in approaching these challenges:
1. Causality
2. Bandits
3. Reinforcement Learning
4. Objectives
5. Fairness
6. Experience Personalization
Trending Now
Trend 1: Causality
From Correlation to Causation
● Most recommendation algorithms
are correlational
○ Some early recommendation
algorithms literally computed
correlations between users and items
● Did you watch a movie because
we recommended it to you? Or
because you liked it? Or both?
● If you had to watch a movie, would
you like it? [Wang et al., 2020] p(Y|X) → p(Y|X, do(R))
(from http://www.tylervigen.com/spurious-correlations)
Feedback loops
Impression bias
inflates plays
Leads to inflated
item popularity
More plays
More
impressions
Oscillations in
distribution of genre
recommendations
Feedback loops can cause biases to be
reinforced by the recommendation system!
[Chaney et al., 2018]: simulations showing that this can reduce the
usefulness of the system
Lots of feedback loops...
Closed Loop
Training
Data
Watches Model
Recs
Search
Training
Data
Watches Model
Recs
Open Loop
Closed Loop
Training
Data
Watches Model
Recs
Search
Training
Data
Watches Model
Recs
Open Loop
Challenges in Causal Recommendations
● Handling unobserved confounders
● Coming up with the right causal graph for the model
● High variance in many causal models
● Computational challenges (e.g. [Wong, 2020])
● Connecting causal recommendations with other aspects like
off-policy reinforcement learning
● When and how to introduce randomization
Trend 2: Bandits in
Recommendations
Why contextual bandits for recommendations?
● Break feedback loops
● Want to explore to learn
● Uncertainty around user interests and new items
● Sparse and indirect feedback
● Changing trends
▶Early news example: [Li et al., 2010]
Example:
Which artwork to show?
Artwork Personalization as
Contextual Bandit
● Environment: Netflix homepage
● Context: Member, device, page, etc.
● Learner: Artwork selector for a show
● Action: Display specific image for show
● Reward: Member has positive engagement
Artwork Selector
▶
Offline Replay Results
● Bandit finds good images
● Personalization is better
● Artwork variety matters
● Personalization wiggles
around best images
Lift in Replay in the various algorithms as
compared to the Random baseline
[More info in our blog post]
● Designing good exploration is an art
○ Especially to support future algorithm innovation
○ Challenging to do user-level A/B tests comparing fully
on-policy bandits at high scale
● Bandits over large action spaces: rankings and slates
● Layers of bandits that influence each other
● Handling delayed rewards
Challenges in with bandits in the real world
Trend 3: Reinforcement
Learning in
Recommendations
Going Long-Term
● Want to maximize long-term member joy
● Involves many user visits, recommendation actions and delayed reward
● … sounds like Reinforcement Learning
Within a page
RL to optimize a
ranking or slate
How long?
Within a session
RL to optimize
multiple interactions
in a session
Across sessions
RL to optimize
interactions across
multiple sessions
● High-dimensional: Action of recommending a single item is O(|C|);
typically want to do ranking or page construction, which is combinatorial.
So are states such as user histories.
● Off-policy: Need to learn and evaluate from existing system actions
● Concurrent: Don’t observe full trajectories, need to learn simultaneously
from many interactions
● Evolving action space: New actions (items) become available and need to
be cold-started. Non-stationary behavior for existing actions.
● Simulator paradox: A great simulator means you already have a great
recommender
● Reward function design: Expressing the objective in a good way
Challenges of Reinforcement Learning for
Recommendations
Interested in more?
REVEAL Workshop 2020:
Bandit and Reinforcement Learning from User Interactions
Trend 4: Objectives
● We want to optimize long-term member joy
● While accounting for:
○ Avoiding “trust busters”
○ Coldstarting
○ Fairness
○ ...
What is your recommender trying to optimize?
Layers of Metrics
Training
Objective
Offline Metric Online Metric Goal
Layers of Metrics
RMSE
NDCG on
historical data
User
Engagement in
A/B test
Joy
Example case: Misaligned Metrics
Training
Objective
Offline Metric Online Metric Goal
Your recommendations can only be as good as the
metrics you measure it on
Many recommenders to optimize
● Same objective? Different ones?
● Can we train (some of) them
together using multi-task learning?
● Is there a way to know a-priori if
combining tasks will be beneficial
or not?
User
history
Ranking
Page
Rating
Explanation
Search
Image
Context ...
[Some MTL examples: Zhao et al., 2015, Bansal et al., 2016, Lu et al., 2018, ...]
● Nuanced metrics:
○ Differences between what you want and what you can
encapsulate in a metric
○ Where does enjoyment come from? How does that vary by
person?
○ How do you measure that at scale?
● Ways of measuring improvements offline before going to A/B test?
● What about effects beyond typical A/B time horizon?
● Avoiding introducing lots of parameters to tune
Challenges in objectives
Trend 5: Fairness
Personalization has a big impact in people’s lives
How do we ensure that it is fair?
Calibrated Recommendations [Steck, 2018]
● Fairness as matching distribution of user interests
● Accuracy as an objective can lead to unbalanced predictions
● Simple example:
● Many recommendation algorithms exhibit this behavior of exaggerating the
dominant interests and crowd out less frequent ones
30 action70 romance
30% action70% romance
User:
Expectation:
100% romanceReality: Maximizes accuracy
Calibration Results (MovieLens 20M)
Baseline model (wMF):
Many users receive
uncalibrated rec’s
After reranking:
Rec’s are much more
calibrated (smaller )
Userdensity
More calibrated (KL divergence)
Submodular
Reranker:
● Which definition of fairness to use in different recommendation
scenarios? [Mehrabi et. al, 2019 catalogues many types]
● Handling fairness without demographic information: both
methods [Beutel et al., 2020] and metrics
● Relationship of fairness with explainability and trust
● Connecting Fairness with all the prior areas
○ Bandits, RL, causality, …
● Beyond fairness of the algorithm: ensuring a positive impact on
society
Challenges in fairness for recommenders
Trend 6:
Experience Personalization
Rating Ranking Pages
4.7
Experience
Evolution of our Personalization Approach
Personalizing how we recommend
(not just what we recommend…)
● Algorithm level: Ideal balance of diversity, popularity,
novelty, freshness, etc. may depend on the person
● Display level: How you present items or explain
recommendations can also be personalized
● Interaction level: Balancing the needs of lean-back
users and power users
So many dimensions to personalize
Rows
Trailer
Evidence
Synopsis
Image
Row Title
Metadata
Ranking
More Adaptive UI
Experience beyond the app
Recommendations New Arrival New Season AlertComing Soon
[Slides about messaging]
● Novelty and learning effects for new experiences
● Cohesion across pages, devices, and time
● Dealing with indirect feedback
● Handling structures of components
○ See [Elahi & Chandrashekar, 2020] poster today
● Coldstarting new experiences
Challenges in Experience Personalization
1. Causality
2. Bandits
3. Reinforcement Learning
4. Objectives
5. Fairness
6. Experience Personalization
Lots of opportunities to improve our
Personalization
Sound interesting?Join us
research.netflix.com/jobs
Interested in internship opportunities?
Follow @NetflixResearch
Thank you
Questions?
@JustinBasilico
Justin Basilico

More Related Content

What's hot

Personalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningPersonalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningAnoop Deoras
 
Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsYves Raimond
 
Contextualization at Netflix
Contextualization at NetflixContextualization at Netflix
Contextualization at NetflixLinas Baltrunas
 
Sequential Decision Making in Recommendations
Sequential Decision Making in RecommendationsSequential Decision Making in Recommendations
Sequential Decision Making in RecommendationsJaya Kawale
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...Sudeep Das, Ph.D.
 
Making Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableMaking Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableJustin Basilico
 
Interactive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyInteractive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyChris Johnson
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixJaya Kawale
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelFaisal Siddiqi
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsJustin Basilico
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at NetflixLinas Baltrunas
 
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Anoop Deoras
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated RecommendationsHarald Steck
 
Netflix Recommendations - Beyond the 5 Stars
Netflix Recommendations - Beyond the 5 StarsNetflix Recommendations - Beyond the 5 Stars
Netflix Recommendations - Beyond the 5 StarsXavier Amatriain
 
Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Sudeep Das, Ph.D.
 
Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"Mounia Lalmas-Roelleke
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareJustin Basilico
 
Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018 Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018 Fernando Amat
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixGrace T. Huang
 

What's hot (20)

Personalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningPersonalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep Learning
 
Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender Systems
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Contextualization at Netflix
Contextualization at NetflixContextualization at Netflix
Contextualization at Netflix
 
Sequential Decision Making in Recommendations
Sequential Decision Making in RecommendationsSequential Decision Making in Recommendations
Sequential Decision Making in Recommendations
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 
Making Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableMaking Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms Reliable
 
Interactive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyInteractive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and Spotify
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at Netflix
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time Travel
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing Recommendations
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at Netflix
 
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated Recommendations
 
Netflix Recommendations - Beyond the 5 Stars
Netflix Recommendations - Beyond the 5 StarsNetflix Recommendations - Beyond the 5 Stars
Netflix Recommendations - Beyond the 5 Stars
 
Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it!
 
Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018 Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at Netflix
 

Similar to Recent Trends in Personalization at Netflix

Recommender Systems In Industry
Recommender Systems In IndustryRecommender Systems In Industry
Recommender Systems In IndustryXavier Amatriain
 
Digital analytics lecture1
Digital analytics lecture1Digital analytics lecture1
Digital analytics lecture1Joni Salminen
 
Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...
Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...
Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...Clark Boyd
 
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure LeskovecThe Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure LeskovecThe Hive
 
Marketing for NonProfit Organizations
Marketing for NonProfit OrganizationsMarketing for NonProfit Organizations
Marketing for NonProfit OrganizationsAnalytive
 
Leverage The Power of Small Data
Leverage The Power of Small DataLeverage The Power of Small Data
Leverage The Power of Small DataKaryn Zuidinga
 
How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...
How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...
How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...jagan477830
 
Requirements Engineering for the Humanities
Requirements Engineering for the HumanitiesRequirements Engineering for the Humanities
Requirements Engineering for the HumanitiesShawn Day
 
Identifying Personas With Agile Research - Dawn of the Data Age Lecture Series
Identifying Personas With Agile Research - Dawn of the Data Age Lecture SeriesIdentifying Personas With Agile Research - Dawn of the Data Age Lecture Series
Identifying Personas With Agile Research - Dawn of the Data Age Lecture SeriesLuciano Pesci, PhD
 
User Experience Research: Deriving Insights for Customer Development
User Experience Research: Deriving Insights for Customer DevelopmentUser Experience Research: Deriving Insights for Customer Development
User Experience Research: Deriving Insights for Customer DevelopmentNoreen Whysel
 
Week 1 Welcome, intro & overview-1.pptx
Week 1 Welcome, intro & overview-1.pptxWeek 1 Welcome, intro & overview-1.pptx
Week 1 Welcome, intro & overview-1.pptxMingLin41
 
Solving Design and Business Problems in 3 Days with Google Design Sprint by B...
Solving Design and Business Problems in 3 Days with Google Design Sprint by B...Solving Design and Business Problems in 3 Days with Google Design Sprint by B...
Solving Design and Business Problems in 3 Days with Google Design Sprint by B...Borrys Hasian
 
2023-04-11-who-ai-win-fbg.pdf
2023-04-11-who-ai-win-fbg.pdf2023-04-11-who-ai-win-fbg.pdf
2023-04-11-who-ai-win-fbg.pdfJonti Bolles
 
Why your analytics land with a thud
Why your analytics land with a thudWhy your analytics land with a thud
Why your analytics land with a thudOne North
 
How to get stakeholder buy in for ux research
How to get stakeholder buy in for ux researchHow to get stakeholder buy in for ux research
How to get stakeholder buy in for ux researchAlicia Zhong
 
[UPDATE] Udacity webinar on Recommendation Systems
[UPDATE] Udacity webinar on Recommendation Systems[UPDATE] Udacity webinar on Recommendation Systems
[UPDATE] Udacity webinar on Recommendation SystemsAxel de Romblay
 
Aiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversionAiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversionDeepak Agarwal
 
Udacity webinar on Recommendation Systems
Udacity webinar on Recommendation SystemsUdacity webinar on Recommendation Systems
Udacity webinar on Recommendation SystemsAxel de Romblay
 

Similar to Recent Trends in Personalization at Netflix (20)

Recommender Systems In Industry
Recommender Systems In IndustryRecommender Systems In Industry
Recommender Systems In Industry
 
Digital analytics lecture1
Digital analytics lecture1Digital analytics lecture1
Digital analytics lecture1
 
Building a Usability Practice
Building a Usability PracticeBuilding a Usability Practice
Building a Usability Practice
 
Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...
Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...
Your smarter data analytics strategy - Social Media Strategies Summit (SMSS) ...
 
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure LeskovecThe Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
 
Marketing for NonProfit Organizations
Marketing for NonProfit OrganizationsMarketing for NonProfit Organizations
Marketing for NonProfit Organizations
 
Leverage The Power of Small Data
Leverage The Power of Small DataLeverage The Power of Small Data
Leverage The Power of Small Data
 
How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...
How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...
How to Choose the Perfect Mtech Project Topic for Your Interests and Career G...
 
Requirements Engineering for the Humanities
Requirements Engineering for the HumanitiesRequirements Engineering for the Humanities
Requirements Engineering for the Humanities
 
Identifying Personas With Agile Research - Dawn of the Data Age Lecture Series
Identifying Personas With Agile Research - Dawn of the Data Age Lecture SeriesIdentifying Personas With Agile Research - Dawn of the Data Age Lecture Series
Identifying Personas With Agile Research - Dawn of the Data Age Lecture Series
 
User Experience Research: Deriving Insights for Customer Development
User Experience Research: Deriving Insights for Customer DevelopmentUser Experience Research: Deriving Insights for Customer Development
User Experience Research: Deriving Insights for Customer Development
 
Week 1 Welcome, intro & overview-1.pptx
Week 1 Welcome, intro & overview-1.pptxWeek 1 Welcome, intro & overview-1.pptx
Week 1 Welcome, intro & overview-1.pptx
 
Solving Design and Business Problems in 3 Days with Google Design Sprint by B...
Solving Design and Business Problems in 3 Days with Google Design Sprint by B...Solving Design and Business Problems in 3 Days with Google Design Sprint by B...
Solving Design and Business Problems in 3 Days with Google Design Sprint by B...
 
2023-04-11-who-ai-win-fbg.pdf
2023-04-11-who-ai-win-fbg.pdf2023-04-11-who-ai-win-fbg.pdf
2023-04-11-who-ai-win-fbg.pdf
 
Why your analytics land with a thud
Why your analytics land with a thudWhy your analytics land with a thud
Why your analytics land with a thud
 
The UX Analyst
The UX AnalystThe UX Analyst
The UX Analyst
 
How to get stakeholder buy in for ux research
How to get stakeholder buy in for ux researchHow to get stakeholder buy in for ux research
How to get stakeholder buy in for ux research
 
[UPDATE] Udacity webinar on Recommendation Systems
[UPDATE] Udacity webinar on Recommendation Systems[UPDATE] Udacity webinar on Recommendation Systems
[UPDATE] Udacity webinar on Recommendation Systems
 
Aiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversionAiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversion
 
Udacity webinar on Recommendation Systems
Udacity webinar on Recommendation SystemsUdacity webinar on Recommendation Systems
Udacity webinar on Recommendation Systems
 

More from Justin Basilico

Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...
Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...
Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...Justin Basilico
 
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...Justin Basilico
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareJustin Basilico
 
Lessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at NetflixLessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at NetflixJustin Basilico
 
Recommendation at Netflix Scale
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix ScaleJustin Basilico
 

More from Justin Basilico (6)

Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...
Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...
Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Band...
 
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Lessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at NetflixLessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at Netflix
 
Learning to Personalize
Learning to PersonalizeLearning to Personalize
Learning to Personalize
 
Recommendation at Netflix Scale
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix Scale
 

Recently uploaded

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 

Recently uploaded (20)

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 

Recent Trends in Personalization at Netflix

  • 1. Recent Trends in Personalization at Netflix Justin Basilico RecSys 2020 Expo 2020-09-24 @JustinBasilico
  • 2. Why do we personalize?
  • 3. Help members find content to watch and enjoy to maximize member satisfaction and retention
  • 5. What do we personalize?
  • 6. Ordering of videos is personalized From how we rank Ranking
  • 7. Selection and placement of rows is personalized ... to how we construct a pageRows
  • 8. ... to how we respond to queries Search query & result recommendation
  • 9. ... to what images we suggest Frame recommendation for artists
  • 11. ... to how we reach out Message personalization
  • 12. Everything is a recommendation!
  • 14. ○ Every person is unique with a variety of interests … and sometimes they share profiles ○ Help people find what they want when they’re not sure what they want ○ Large datasets but small data per user … and potentially biased by the output of your system ○ Cold-start problems on all sides ○ Non-stationary, context-dependent, mood-dependent, ... ○ More than just accuracy: Diversity, novelty, freshness, fairness, ... ○ ... No, personalization is hard!
  • 15. So what are you doing about it?
  • 16. Some recent avenues in approaching these challenges: 1. Causality 2. Bandits 3. Reinforcement Learning 4. Objectives 5. Fairness 6. Experience Personalization Trending Now
  • 18. From Correlation to Causation ● Most recommendation algorithms are correlational ○ Some early recommendation algorithms literally computed correlations between users and items ● Did you watch a movie because we recommended it to you? Or because you liked it? Or both? ● If you had to watch a movie, would you like it? [Wang et al., 2020] p(Y|X) → p(Y|X, do(R)) (from http://www.tylervigen.com/spurious-correlations)
  • 19. Feedback loops Impression bias inflates plays Leads to inflated item popularity More plays More impressions Oscillations in distribution of genre recommendations Feedback loops can cause biases to be reinforced by the recommendation system! [Chaney et al., 2018]: simulations showing that this can reduce the usefulness of the system
  • 20. Lots of feedback loops...
  • 23. Challenges in Causal Recommendations ● Handling unobserved confounders ● Coming up with the right causal graph for the model ● High variance in many causal models ● Computational challenges (e.g. [Wong, 2020]) ● Connecting causal recommendations with other aspects like off-policy reinforcement learning ● When and how to introduce randomization
  • 24. Trend 2: Bandits in Recommendations
  • 25. Why contextual bandits for recommendations? ● Break feedback loops ● Want to explore to learn ● Uncertainty around user interests and new items ● Sparse and indirect feedback ● Changing trends ▶Early news example: [Li et al., 2010]
  • 27. Artwork Personalization as Contextual Bandit ● Environment: Netflix homepage ● Context: Member, device, page, etc. ● Learner: Artwork selector for a show ● Action: Display specific image for show ● Reward: Member has positive engagement Artwork Selector ▶
  • 28. Offline Replay Results ● Bandit finds good images ● Personalization is better ● Artwork variety matters ● Personalization wiggles around best images Lift in Replay in the various algorithms as compared to the Random baseline [More info in our blog post]
  • 29. ● Designing good exploration is an art ○ Especially to support future algorithm innovation ○ Challenging to do user-level A/B tests comparing fully on-policy bandits at high scale ● Bandits over large action spaces: rankings and slates ● Layers of bandits that influence each other ● Handling delayed rewards Challenges in with bandits in the real world
  • 30. Trend 3: Reinforcement Learning in Recommendations
  • 31. Going Long-Term ● Want to maximize long-term member joy ● Involves many user visits, recommendation actions and delayed reward ● … sounds like Reinforcement Learning
  • 32. Within a page RL to optimize a ranking or slate How long? Within a session RL to optimize multiple interactions in a session Across sessions RL to optimize interactions across multiple sessions
  • 33. ● High-dimensional: Action of recommending a single item is O(|C|); typically want to do ranking or page construction, which is combinatorial. So are states such as user histories. ● Off-policy: Need to learn and evaluate from existing system actions ● Concurrent: Don’t observe full trajectories, need to learn simultaneously from many interactions ● Evolving action space: New actions (items) become available and need to be cold-started. Non-stationary behavior for existing actions. ● Simulator paradox: A great simulator means you already have a great recommender ● Reward function design: Expressing the objective in a good way Challenges of Reinforcement Learning for Recommendations
  • 34. Interested in more? REVEAL Workshop 2020: Bandit and Reinforcement Learning from User Interactions
  • 36. ● We want to optimize long-term member joy ● While accounting for: ○ Avoiding “trust busters” ○ Coldstarting ○ Fairness ○ ... What is your recommender trying to optimize?
  • 37. Layers of Metrics Training Objective Offline Metric Online Metric Goal
  • 38. Layers of Metrics RMSE NDCG on historical data User Engagement in A/B test Joy Example case: Misaligned Metrics Training Objective Offline Metric Online Metric Goal
  • 39. Your recommendations can only be as good as the metrics you measure it on
  • 40. Many recommenders to optimize ● Same objective? Different ones? ● Can we train (some of) them together using multi-task learning? ● Is there a way to know a-priori if combining tasks will be beneficial or not? User history Ranking Page Rating Explanation Search Image Context ... [Some MTL examples: Zhao et al., 2015, Bansal et al., 2016, Lu et al., 2018, ...]
  • 41. ● Nuanced metrics: ○ Differences between what you want and what you can encapsulate in a metric ○ Where does enjoyment come from? How does that vary by person? ○ How do you measure that at scale? ● Ways of measuring improvements offline before going to A/B test? ● What about effects beyond typical A/B time horizon? ● Avoiding introducing lots of parameters to tune Challenges in objectives
  • 43. Personalization has a big impact in people’s lives How do we ensure that it is fair?
  • 44. Calibrated Recommendations [Steck, 2018] ● Fairness as matching distribution of user interests ● Accuracy as an objective can lead to unbalanced predictions ● Simple example: ● Many recommendation algorithms exhibit this behavior of exaggerating the dominant interests and crowd out less frequent ones 30 action70 romance 30% action70% romance User: Expectation: 100% romanceReality: Maximizes accuracy
  • 45. Calibration Results (MovieLens 20M) Baseline model (wMF): Many users receive uncalibrated rec’s After reranking: Rec’s are much more calibrated (smaller ) Userdensity More calibrated (KL divergence) Submodular Reranker:
  • 46. ● Which definition of fairness to use in different recommendation scenarios? [Mehrabi et. al, 2019 catalogues many types] ● Handling fairness without demographic information: both methods [Beutel et al., 2020] and metrics ● Relationship of fairness with explainability and trust ● Connecting Fairness with all the prior areas ○ Bandits, RL, causality, … ● Beyond fairness of the algorithm: ensuring a positive impact on society Challenges in fairness for recommenders
  • 48. Rating Ranking Pages 4.7 Experience Evolution of our Personalization Approach
  • 49. Personalizing how we recommend (not just what we recommend…) ● Algorithm level: Ideal balance of diversity, popularity, novelty, freshness, etc. may depend on the person ● Display level: How you present items or explain recommendations can also be personalized ● Interaction level: Balancing the needs of lean-back users and power users
  • 50. So many dimensions to personalize Rows Trailer Evidence Synopsis Image Row Title Metadata Ranking
  • 52. Experience beyond the app Recommendations New Arrival New Season AlertComing Soon [Slides about messaging]
  • 53. ● Novelty and learning effects for new experiences ● Cohesion across pages, devices, and time ● Dealing with indirect feedback ● Handling structures of components ○ See [Elahi & Chandrashekar, 2020] poster today ● Coldstarting new experiences Challenges in Experience Personalization
  • 54.
  • 55. 1. Causality 2. Bandits 3. Reinforcement Learning 4. Objectives 5. Fairness 6. Experience Personalization Lots of opportunities to improve our Personalization
  • 56. Sound interesting?Join us research.netflix.com/jobs Interested in internship opportunities? Follow @NetflixResearch