SlideShare a Scribd company logo
Artwork
Personalization
at Netflix
Fernando Amat
RecSys, Oct 2018
Quickly help members discover content they’ll love
Global Members, Personalized Tastes
130 Million Members
~180 Countries
98% Match
Spot the
Algorithms!
Artwork Optimization
Artwork Optimization
Goal: Recommend a personalized
artwork or imagery for a title to help
members decide if they will enjoy the
title or not.
Intuition for Personalized Assets
● Emphasize themes through different artwork according to some
context (user, viewing history, country, etc.)
Preferences in genre
Intuition for Personalized Assets
● Emphasize themes through different artwork according to some
context (user, viewing history, country, etc)
Preferences in cast members
Bandit Algorithms Setting
For each (user, show) request:
● Actions: set of candidate images available
● Reward: how many minutes did the user play from that impression
● Environment: Netflix homepage in user’s device
● Learner: its goal is to maximize the cumulative reward after N requests
Learner Environment
Action
Reward
Context
Numerous Variants
● Different Strategies: ε-Greedy, Thompson Sampling (TS), Upper Confidence
Bound (UCB), etc.
● Different Environments:
○ Stochastic and stationary: Reward is generated i.i.d. from a distribution
specific to the action. No payoff drift.
○ Adversarial: No assumptions on how rewards are generated.
● Different objectives: Cumulative regret, tracking the best expert
● Continuous or discrete set of actions, finite vs infinite
● Extensions: Varying set of arms, Contextual Bandits, etc.
Specific challenges
● Play attribution and reward assignment
○ Incremental effect of the image on top of recommender system
● Only one image per title can be presented
○ Although inherently it is a ranking problem
Would you play because the movie is recommended or because of the artwork? Or both?
Specific challenges
● Change effect
○ Can changing images too often make users confused?
Session 1 Session 2 Session 3 ... Session N
Sequence A
Sequence B
● We have control over the set of actions
○ How many images per show
○ Image design
● What makes a good asset?
○ Representative (no clickbait)
○ Differential
○ Informative
○ Engaging
Actions
Personal (i.e. contextual)
Explore
show?
Choose
Epsilon Greedy Example
εprofile
1-εprofile
εshow
1-εshow Personalized Image
Image
At Random
● Learn a binary classifier per image to predict probability of play
● Pick the winner (arg max)
Member
(context)
Features
Image Pool
Model 1
Winner
arg
max
Model 2
Model 3
Model 4
Greedy Policy Example
Take Fraction Example: Luke Cage
Take Fraction = 1 / 3
Play
No play
User A
User B
User C
● Unbiased offline evaluation from explore data
Offline metric: Replay [Li et al, 2010]
Offline Take Fraction = 2 / 3
User 1 User 2 User 3 User 4 User 5 User 6
Random Assignment
Play?
Model Assignment
Offline Replay
● Context matters
● Artwork diversity matters
● Personalization wiggles
around most popular images
Lift in Replay in the various algorithms as
compared to the Random baseline
Online results
● Rollout to our >130M member base
● Most beneficial for lesser known titles
● Compression from title -level offline metrics due to cannibalization
between titles
Research
Directions
Action selection orchestration
● Neighboring image selection influences result
● Title-level optimization is not enough
Row A
(diverse
images)
Row B
(the
microphone
row)
Stand-up comedy
Automatic image selection
● Generating new artwork is costly and time consuming
● Develop algorithm to predict asset quality from raw image
Long-term Reward: Road to RL
● Maximize long term reward: reinforcement learning
○ User long term joy rather than plays
Thank you.
Fernando Amat (famat@netflix.com)
Blogpost
We are hiring!

More Related Content

What's hot

Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender Systems
Yves Raimond
 
Recent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveRecent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix Perspective
Justin Basilico
 
Sequential Decision Making in Recommendations
Sequential Decision Making in RecommendationsSequential Decision Making in Recommendations
Sequential Decision Making in Recommendations
Jaya Kawale
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
Yves Raimond
 
Shallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemShallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender System
Anoop Deoras
 
Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it!
Sudeep Das, Ph.D.
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at Netflix
Linas Baltrunas
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019
Faisal Siddiqi
 
Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender models
Parmeshwar Khurd
 
Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018 Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018
Fernando Amat
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing Recommendations
Justin Basilico
 
Making Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableMaking Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms Reliable
Justin Basilico
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Sudeep Das, Ph.D.
 
Learning a Personalized Homepage
Learning a Personalized HomepageLearning a Personalized Homepage
Learning a Personalized Homepage
Justin Basilico
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time Travel
Faisal Siddiqi
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated Recommendations
Harald Steck
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
Justin Basilico
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at Netflix
Grace T. Huang
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
MLconf
 
Past, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry PerspectivePast, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry Perspective
Justin Basilico
 

What's hot (20)

Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender Systems
 
Recent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveRecent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix Perspective
 
Sequential Decision Making in Recommendations
Sequential Decision Making in RecommendationsSequential Decision Making in Recommendations
Sequential Decision Making in Recommendations
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
 
Shallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemShallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender System
 
Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it!
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at Netflix
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019
 
Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender models
 
Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018 Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing Recommendations
 
Making Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableMaking Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms Reliable
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 
Learning a Personalized Homepage
Learning a Personalized HomepageLearning a Personalized Homepage
Learning a Personalized Homepage
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time Travel
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated Recommendations
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at Netflix
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
 
Past, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry PerspectivePast, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry Perspective
 

Similar to Artworks personalization on Netflix

A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at Netflix
Jaya Kawale
 
Building a deep learning ai.pptx
Building a deep learning ai.pptxBuilding a deep learning ai.pptx
Building a deep learning ai.pptx
Daniel Slater
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Xavier Amatriain
 
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Silicon Studio Corporation
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Xavier Amatriain
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
Xavier Amatriain
 
Andrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and PreservationAndrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and Preservation
PhotoPrism.org
 
Game Design for Modern Times
Game Design for Modern TimesGame Design for Modern Times
Game Design for Modern Times
Deborah Mensah-Bonsu
 
[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用
台灣資料科學年會
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
Francesco Casalegno
 
Analytics toolbox
Analytics toolboxAnalytics toolbox
Analytics toolbox
Douglas Cohen
 
Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the World
Yves Raimond
 
Learning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive DisabilitiesLearning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive Disabilities
Baltasar Fernández-Manjón
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
Förderverein Technische Fakultät
 
Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)
Rajkumar Pawar
 
Deep Learning Jump Start
Deep Learning Jump StartDeep Learning Jump Start
Deep Learning Jump Start
Michele Toni
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
MLconf
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
Xavier Amatriain
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
Xavier Amatriain
 

Similar to Artworks personalization on Netflix (20)

A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at Netflix
 
Building a deep learning ai.pptx
Building a deep learning ai.pptxBuilding a deep learning ai.pptx
Building a deep learning ai.pptx
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
 
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
 
Andrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and PreservationAndrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and Preservation
 
Real Talk
Real TalkReal Talk
Real Talk
 
Game Design for Modern Times
Game Design for Modern TimesGame Design for Modern Times
Game Design for Modern Times
 
[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Analytics toolbox
Analytics toolboxAnalytics toolbox
Analytics toolbox
 
Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the World
 
Learning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive DisabilitiesLearning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive Disabilities
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)
 
Deep Learning Jump Start
Deep Learning Jump StartDeep Learning Jump Start
Deep Learning Jump Start
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
 

More from IntoTheMinds

Voilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la repriseVoilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la reprise
IntoTheMinds
 
The advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisisThe advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisis
IntoTheMinds
 
Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"
IntoTheMinds
 
Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"
IntoTheMinds
 
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
IntoTheMinds
 
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
IntoTheMinds
 
Purchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sectorPurchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sector
IntoTheMinds
 
A robot called Voitto
A robot called VoittoA robot called Voitto
A robot called Voitto
IntoTheMinds
 
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
IntoTheMinds
 
Privacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing EconomyPrivacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing Economy
IntoTheMinds
 
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
IntoTheMinds
 
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
IntoTheMinds
 
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
IntoTheMinds
 
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
IntoTheMinds
 
Big Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrandBig Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrand
IntoTheMinds
 
Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora) Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora)
IntoTheMinds
 
Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017
IntoTheMinds
 
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
IntoTheMinds
 
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic..."Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
IntoTheMinds
 
Presentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshopPresentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshop
IntoTheMinds
 

More from IntoTheMinds (20)

Voilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la repriseVoilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la reprise
 
The advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisisThe advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisis
 
Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"
 
Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"
 
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
 
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
 
Purchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sectorPurchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sector
 
A robot called Voitto
A robot called VoittoA robot called Voitto
A robot called Voitto
 
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
 
Privacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing EconomyPrivacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing Economy
 
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
 
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
 
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
 
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
 
Big Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrandBig Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrand
 
Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora) Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora)
 
Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017
 
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
 
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic..."Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
 
Presentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshopPresentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshop
 

Recently uploaded

My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
rwarrenll
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
manishkhaire30
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
javier ramirez
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
74nqk8xf
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
Nanandann Nilekani's ppt On India's .pdf
Nanandann Nilekani's ppt On India's .pdfNanandann Nilekani's ppt On India's .pdf
Nanandann Nilekani's ppt On India's .pdf
eddie19851
 
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdfCh03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
haila53
 
做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样
做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样
做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样
axoqas
 
一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理
一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理
一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理
dwreak4tg
 
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
mbawufebxi
 
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
John Andrews
 
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Subhajit Sahu
 
Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)
TravisMalana
 
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
g4dpvqap0
 
一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理
v3tuleee
 
一比一原版(CBU毕业证)卡普顿大学毕业证如何办理
一比一原版(CBU毕业证)卡普顿大学毕业证如何办理一比一原版(CBU毕业证)卡普顿大学毕业证如何办理
一比一原版(CBU毕业证)卡普顿大学毕业证如何办理
ahzuo
 
一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理
一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理
一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理
oz8q3jxlp
 
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
slg6lamcq
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
ahzuo
 

Recently uploaded (20)

My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
Nanandann Nilekani's ppt On India's .pdf
Nanandann Nilekani's ppt On India's .pdfNanandann Nilekani's ppt On India's .pdf
Nanandann Nilekani's ppt On India's .pdf
 
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdfCh03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
 
做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样
做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样
做(mqu毕业证书)麦考瑞大学毕业证硕士文凭证书学费发票原版一模一样
 
一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理
一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理
一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理
 
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
 
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
 
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
 
Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)
 
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
 
一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理
 
一比一原版(CBU毕业证)卡普顿大学毕业证如何办理
一比一原版(CBU毕业证)卡普顿大学毕业证如何办理一比一原版(CBU毕业证)卡普顿大学毕业证如何办理
一比一原版(CBU毕业证)卡普顿大学毕业证如何办理
 
一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理
一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理
一比一原版(Deakin毕业证书)迪肯大学毕业证如何办理
 
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
 

Artworks personalization on Netflix

  • 2. Quickly help members discover content they’ll love
  • 3. Global Members, Personalized Tastes 130 Million Members ~180 Countries
  • 6. Artwork Optimization Goal: Recommend a personalized artwork or imagery for a title to help members decide if they will enjoy the title or not.
  • 7. Intuition for Personalized Assets ● Emphasize themes through different artwork according to some context (user, viewing history, country, etc.) Preferences in genre
  • 8. Intuition for Personalized Assets ● Emphasize themes through different artwork according to some context (user, viewing history, country, etc) Preferences in cast members
  • 9. Bandit Algorithms Setting For each (user, show) request: ● Actions: set of candidate images available ● Reward: how many minutes did the user play from that impression ● Environment: Netflix homepage in user’s device ● Learner: its goal is to maximize the cumulative reward after N requests Learner Environment Action Reward Context
  • 10. Numerous Variants ● Different Strategies: ε-Greedy, Thompson Sampling (TS), Upper Confidence Bound (UCB), etc. ● Different Environments: ○ Stochastic and stationary: Reward is generated i.i.d. from a distribution specific to the action. No payoff drift. ○ Adversarial: No assumptions on how rewards are generated. ● Different objectives: Cumulative regret, tracking the best expert ● Continuous or discrete set of actions, finite vs infinite ● Extensions: Varying set of arms, Contextual Bandits, etc.
  • 11. Specific challenges ● Play attribution and reward assignment ○ Incremental effect of the image on top of recommender system ● Only one image per title can be presented ○ Although inherently it is a ranking problem Would you play because the movie is recommended or because of the artwork? Or both?
  • 12. Specific challenges ● Change effect ○ Can changing images too often make users confused? Session 1 Session 2 Session 3 ... Session N Sequence A Sequence B
  • 13. ● We have control over the set of actions ○ How many images per show ○ Image design ● What makes a good asset? ○ Representative (no clickbait) ○ Differential ○ Informative ○ Engaging Actions Personal (i.e. contextual)
  • 15. ● Learn a binary classifier per image to predict probability of play ● Pick the winner (arg max) Member (context) Features Image Pool Model 1 Winner arg max Model 2 Model 3 Model 4 Greedy Policy Example
  • 16. Take Fraction Example: Luke Cage Take Fraction = 1 / 3 Play No play User A User B User C
  • 17. ● Unbiased offline evaluation from explore data Offline metric: Replay [Li et al, 2010] Offline Take Fraction = 2 / 3 User 1 User 2 User 3 User 4 User 5 User 6 Random Assignment Play? Model Assignment
  • 18. Offline Replay ● Context matters ● Artwork diversity matters ● Personalization wiggles around most popular images Lift in Replay in the various algorithms as compared to the Random baseline
  • 19. Online results ● Rollout to our >130M member base ● Most beneficial for lesser known titles ● Compression from title -level offline metrics due to cannibalization between titles
  • 21. Action selection orchestration ● Neighboring image selection influences result ● Title-level optimization is not enough Row A (diverse images) Row B (the microphone row) Stand-up comedy
  • 22. Automatic image selection ● Generating new artwork is costly and time consuming ● Develop algorithm to predict asset quality from raw image
  • 23. Long-term Reward: Road to RL ● Maximize long term reward: reinforcement learning ○ User long term joy rather than plays
  • 24. Thank you. Fernando Amat (famat@netflix.com) Blogpost We are hiring!