SlideShare a Scribd company logo
1 of 24
Download to read offline
Artwork
Personalization
at Netflix
Fernando Amat
RecSys, Oct 2018
Quickly help members discover content they’ll love
Global Members, Personalized Tastes
130 Million Members
~180 Countries
98% Match
Spot the
Algorithms!
Artwork Optimization
Artwork Optimization
Goal: Recommend a personalized
artwork or imagery for a title to help
members decide if they will enjoy the
title or not.
Intuition for Personalized Assets
● Emphasize themes through different artwork according to some
context (user, viewing history, country, etc.)
Preferences in genre
Intuition for Personalized Assets
● Emphasize themes through different artwork according to some
context (user, viewing history, country, etc)
Preferences in cast members
Bandit Algorithms Setting
For each (user, show) request:
● Actions: set of candidate images available
● Reward: how many minutes did the user play from that impression
● Environment: Netflix homepage in user’s device
● Learner: its goal is to maximize the cumulative reward after N requests
Learner Environment
Action
Reward
Context
Numerous Variants
● Different Strategies: ε-Greedy, Thompson Sampling (TS), Upper Confidence
Bound (UCB), etc.
● Different Environments:
○ Stochastic and stationary: Reward is generated i.i.d. from a distribution
specific to the action. No payoff drift.
○ Adversarial: No assumptions on how rewards are generated.
● Different objectives: Cumulative regret, tracking the best expert
● Continuous or discrete set of actions, finite vs infinite
● Extensions: Varying set of arms, Contextual Bandits, etc.
Specific challenges
● Play attribution and reward assignment
○ Incremental effect of the image on top of recommender system
● Only one image per title can be presented
○ Although inherently it is a ranking problem
Would you play because the movie is recommended or because of the artwork? Or both?
Specific challenges
● Change effect
○ Can changing images too often make users confused?
Session 1 Session 2 Session 3 ... Session N
Sequence A
Sequence B
● We have control over the set of actions
○ How many images per show
○ Image design
● What makes a good asset?
○ Representative (no clickbait)
○ Differential
○ Informative
○ Engaging
Actions
Personal (i.e. contextual)
Explore
show?
Choose
Epsilon Greedy Example
εprofile
1-εprofile
εshow
1-εshow Personalized Image
Image
At Random
● Learn a binary classifier per image to predict probability of play
● Pick the winner (arg max)
Member
(context)
Features
Image Pool
Model 1
Winner
arg
max
Model 2
Model 3
Model 4
Greedy Policy Example
Take Fraction Example: Luke Cage
Take Fraction = 1 / 3
Play
No play
User A
User B
User C
● Unbiased offline evaluation from explore data
Offline metric: Replay [Li et al, 2010]
Offline Take Fraction = 2 / 3
User 1 User 2 User 3 User 4 User 5 User 6
Random Assignment
Play?
Model Assignment
Offline Replay
● Context matters
● Artwork diversity matters
● Personalization wiggles
around most popular images
Lift in Replay in the various algorithms as
compared to the Random baseline
Online results
● Rollout to our >130M member base
● Most beneficial for lesser known titles
● Compression from title -level offline metrics due to cannibalization
between titles
Research
Directions
Action selection orchestration
● Neighboring image selection influences result
● Title-level optimization is not enough
Row A
(diverse
images)
Row B
(the
microphone
row)
Stand-up comedy
Automatic image selection
● Generating new artwork is costly and time consuming
● Develop algorithm to predict asset quality from raw image
Long-term Reward: Road to RL
● Maximize long term reward: reinforcement learning
○ User long term joy rather than plays
Thank you.
Fernando Amat (famat@netflix.com)
Blogpost
We are hiring!

More Related Content

What's hot

Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender SystemsYves Raimond
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at NetflixLinas Baltrunas
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixJustin Basilico
 
Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsYves Raimond
 
Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender modelsParmeshwar Khurd
 
Contextualization at Netflix
Contextualization at NetflixContextualization at Netflix
Contextualization at NetflixLinas Baltrunas
 
Making Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableMaking Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableJustin Basilico
 
Recommendation at Netflix Scale
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix ScaleJustin Basilico
 
Tableau Conference 2018: Binging on Data - Enabling Analytics at Netflix
Tableau Conference 2018: Binging on Data - Enabling Analytics at NetflixTableau Conference 2018: Binging on Data - Enabling Analytics at Netflix
Tableau Conference 2018: Binging on Data - Enabling Analytics at NetflixBlake Irvine
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated RecommendationsHarald Steck
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixJaya Kawale
 
Shallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemShallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemAnoop Deoras
 
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Anoop Deoras
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsJustin Basilico
 
Déjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender SystemsDéjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender SystemsJustin Basilico
 
ML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talkML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talkFaisal Siddiqi
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...MLconf
 
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020Zachary Schendel
 
Deep Learning for Personalized Search and Recommender Systems
Deep Learning for Personalized Search and Recommender SystemsDeep Learning for Personalized Search and Recommender Systems
Deep Learning for Personalized Search and Recommender SystemsBenjamin Le
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Faisal Siddiqi
 

What's hot (20)

Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at Netflix
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender Systems
 
Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender models
 
Contextualization at Netflix
Contextualization at NetflixContextualization at Netflix
Contextualization at Netflix
 
Making Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms ReliableMaking Netflix Machine Learning Algorithms Reliable
Making Netflix Machine Learning Algorithms Reliable
 
Recommendation at Netflix Scale
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix Scale
 
Tableau Conference 2018: Binging on Data - Enabling Analytics at Netflix
Tableau Conference 2018: Binging on Data - Enabling Analytics at NetflixTableau Conference 2018: Binging on Data - Enabling Analytics at Netflix
Tableau Conference 2018: Binging on Data - Enabling Analytics at Netflix
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated Recommendations
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at Netflix
 
Shallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemShallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender System
 
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing Recommendations
 
Déjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender SystemsDéjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender Systems
 
ML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talkML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talk
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
 
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
 
Deep Learning for Personalized Search and Recommender Systems
Deep Learning for Personalized Search and Recommender SystemsDeep Learning for Personalized Search and Recommender Systems
Deep Learning for Personalized Search and Recommender Systems
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019
 

Similar to Artworks personalization on Netflix

Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at NetflixJustin Basilico
 
Building a deep learning ai.pptx
Building a deep learning ai.pptxBuilding a deep learning ai.pptx
Building a deep learning ai.pptxDaniel Slater
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning SystemsXavier Amatriain
 
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...Silicon Studio Corporation
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Xavier Amatriain
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsXavier Amatriain
 
Andrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and PreservationAndrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and PreservationPhotoPrism.org
 
[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用台灣資料科學年會
 
Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the WorldYves Raimond
 
Learning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive DisabilitiesLearning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive DisabilitiesBaltasar Fernández-Manjón
 
Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)Rajkumar Pawar
 
Deep Learning Jump Start
Deep Learning Jump StartDeep Learning Jump Start
Deep Learning Jump StartMichele Toni
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15MLconf
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConfXavier Amatriain
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systemsXavier Amatriain
 

Similar to Artworks personalization on Netflix (20)

Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at Netflix
 
Building a deep learning ai.pptx
Building a deep learning ai.pptxBuilding a deep learning ai.pptx
Building a deep learning ai.pptx
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
 
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using ...
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
 
Andrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and PreservationAndrea Ceroni: Personal Photo Management and Preservation
Andrea Ceroni: Personal Photo Management and Preservation
 
Real Talk
Real TalkReal Talk
Real Talk
 
Game Design for Modern Times
Game Design for Modern TimesGame Design for Modern Times
Game Design for Modern Times
 
[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用[系列活動] 人工智慧與機器學習在推薦系統上的應用
[系列活動] 人工智慧與機器學習在推薦系統上的應用
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Analytics toolbox
Analytics toolboxAnalytics toolbox
Analytics toolbox
 
Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the World
 
Learning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive DisabilitiesLearning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive Disabilities
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)Technical aspectof game design (Game Architecture)
Technical aspectof game design (Game Architecture)
 
Deep Learning Jump Start
Deep Learning Jump StartDeep Learning Jump Start
Deep Learning Jump Start
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
 

More from IntoTheMinds

Voilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la repriseVoilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la repriseIntoTheMinds
 
The advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisisThe advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisisIntoTheMinds
 
Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"IntoTheMinds
 
Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"IntoTheMinds
 
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"IntoTheMinds
 
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"IntoTheMinds
 
Purchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sectorPurchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sectorIntoTheMinds
 
A robot called Voitto
A robot called VoittoA robot called Voitto
A robot called VoittoIntoTheMinds
 
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018IntoTheMinds
 
Privacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing EconomyPrivacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing EconomyIntoTheMinds
 
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...IntoTheMinds
 
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...IntoTheMinds
 
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...IntoTheMinds
 
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...IntoTheMinds
 
Big Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrandBig Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrandIntoTheMinds
 
Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora) Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora) IntoTheMinds
 
Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017IntoTheMinds
 
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)IntoTheMinds
 
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic..."Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...IntoTheMinds
 
Presentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshopPresentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshopIntoTheMinds
 

More from IntoTheMinds (20)

Voilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la repriseVoilà à quoi ressemblera la reprise
Voilà à quoi ressemblera la reprise
 
The advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisisThe advertising campaigns run in Belgium during the Covid-19 crisis
The advertising campaigns run in Belgium during the Covid-19 crisis
 
Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"Presentation Christian Radler at EBU Conference "data in the newsroom"
Presentation Christian Radler at EBU Conference "data in the newsroom"
 
Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"Presentation Sabino Metta at EBU Conference "data in the newsroom"
Presentation Sabino Metta at EBU Conference "data in the newsroom"
 
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
Presentation Stéphane Saulnier at EBU Conference "data in the newsroom"
 
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
Presentation Kristofer Sjoholm at EBU Conference "data in the newsroom"
 
Purchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sectorPurchase drivers for iconic products in the luxury sector
Purchase drivers for iconic products in the luxury sector
 
A robot called Voitto
A robot called VoittoA robot called Voitto
A robot called Voitto
 
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
presentation Newsbridge by Philippe Petitpont at Media Fast Forward 2018
 
Privacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing EconomyPrivacy Calculus in the Sharing Economy
Privacy Calculus in the Sharing Economy
 
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
Toon borré presentation at Meetup Big Data and Ethics at DigitYser Brussels 1...
 
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
Leenke De Donder presentation at Meetup Big Data and Ethics at DigitYser Brus...
 
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
Jochanen eynikel presentation at Meetup Big Data and Ethics at DigitYser Brus...
 
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
Thomas carette presentation at Meetup Big Data and Ethics at DigitYser Brusse...
 
Big Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrandBig Data and ethics meetup : slides presentation michael ekstrand
Big Data and ethics meetup : slides presentation michael ekstrand
 
Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora) Presentatie big data (Dag van de verkoper, Cevora)
Presentatie big data (Dag van de verkoper, Cevora)
 
Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017Presentatie big data in verkoop (cevora) gent 16 Mei 2017
Presentatie big data in verkoop (cevora) gent 16 Mei 2017
 
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
Slides pierre nicolas schwab DISummit 2017 (Big Data, Brussels)
 
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic..."Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
"Building Trust" discussion panel at EBU Big Data conference 2017 (Pierre-Nic...
 
Presentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshopPresentation by Steven Bourke at the EBU Big Data and Society workshop
Presentation by Steven Bourke at the EBU Big Data and Society workshop
 

Recently uploaded

testingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdftestingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdfDSP Mutual Fund
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfnikeshsingh56
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformationAnnie Melnic
 
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...ThinkInnovation
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)
 
prediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachprediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachAdekunleJoseph4
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelBoston Institute of Analytics
 
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfRabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j
 
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbaAdobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbas73678sri
 
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvwAdobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvws73678sri
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 
Data Discovery With Power Query in excel
Data Discovery With Power Query in excelData Discovery With Power Query in excel
Data Discovery With Power Query in excelKapilSidhpuria3
 
Inference rules in artificial intelligence
Inference rules in artificial intelligenceInference rules in artificial intelligence
Inference rules in artificial intelligencePriyadharshiniG41
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 

Recently uploaded (20)

testingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdftestingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdf
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdf
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformation
 
Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
 
prediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachprediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approach
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
 
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfRabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
 
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbaAdobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
 
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvwAdobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
Data Discovery With Power Query in excel
Data Discovery With Power Query in excelData Discovery With Power Query in excel
Data Discovery With Power Query in excel
 
2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 
Inference rules in artificial intelligence
Inference rules in artificial intelligenceInference rules in artificial intelligence
Inference rules in artificial intelligence
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 

Artworks personalization on Netflix

  • 2. Quickly help members discover content they’ll love
  • 3. Global Members, Personalized Tastes 130 Million Members ~180 Countries
  • 6. Artwork Optimization Goal: Recommend a personalized artwork or imagery for a title to help members decide if they will enjoy the title or not.
  • 7. Intuition for Personalized Assets ● Emphasize themes through different artwork according to some context (user, viewing history, country, etc.) Preferences in genre
  • 8. Intuition for Personalized Assets ● Emphasize themes through different artwork according to some context (user, viewing history, country, etc) Preferences in cast members
  • 9. Bandit Algorithms Setting For each (user, show) request: ● Actions: set of candidate images available ● Reward: how many minutes did the user play from that impression ● Environment: Netflix homepage in user’s device ● Learner: its goal is to maximize the cumulative reward after N requests Learner Environment Action Reward Context
  • 10. Numerous Variants ● Different Strategies: ε-Greedy, Thompson Sampling (TS), Upper Confidence Bound (UCB), etc. ● Different Environments: ○ Stochastic and stationary: Reward is generated i.i.d. from a distribution specific to the action. No payoff drift. ○ Adversarial: No assumptions on how rewards are generated. ● Different objectives: Cumulative regret, tracking the best expert ● Continuous or discrete set of actions, finite vs infinite ● Extensions: Varying set of arms, Contextual Bandits, etc.
  • 11. Specific challenges ● Play attribution and reward assignment ○ Incremental effect of the image on top of recommender system ● Only one image per title can be presented ○ Although inherently it is a ranking problem Would you play because the movie is recommended or because of the artwork? Or both?
  • 12. Specific challenges ● Change effect ○ Can changing images too often make users confused? Session 1 Session 2 Session 3 ... Session N Sequence A Sequence B
  • 13. ● We have control over the set of actions ○ How many images per show ○ Image design ● What makes a good asset? ○ Representative (no clickbait) ○ Differential ○ Informative ○ Engaging Actions Personal (i.e. contextual)
  • 15. ● Learn a binary classifier per image to predict probability of play ● Pick the winner (arg max) Member (context) Features Image Pool Model 1 Winner arg max Model 2 Model 3 Model 4 Greedy Policy Example
  • 16. Take Fraction Example: Luke Cage Take Fraction = 1 / 3 Play No play User A User B User C
  • 17. ● Unbiased offline evaluation from explore data Offline metric: Replay [Li et al, 2010] Offline Take Fraction = 2 / 3 User 1 User 2 User 3 User 4 User 5 User 6 Random Assignment Play? Model Assignment
  • 18. Offline Replay ● Context matters ● Artwork diversity matters ● Personalization wiggles around most popular images Lift in Replay in the various algorithms as compared to the Random baseline
  • 19. Online results ● Rollout to our >130M member base ● Most beneficial for lesser known titles ● Compression from title -level offline metrics due to cannibalization between titles
  • 21. Action selection orchestration ● Neighboring image selection influences result ● Title-level optimization is not enough Row A (diverse images) Row B (the microphone row) Stand-up comedy
  • 22. Automatic image selection ● Generating new artwork is costly and time consuming ● Develop algorithm to predict asset quality from raw image
  • 23. Long-term Reward: Road to RL ● Maximize long term reward: reinforcement learning ○ User long term joy rather than plays
  • 24. Thank you. Fernando Amat (famat@netflix.com) Blogpost We are hiring!