SlideShare a Scribd company logo
1 of 43
Download to read offline
Bandit Infrastructure
for Personalized
Recommendations
Elliot Chow & Fernando Amat Gil
ML Platform Meetup
September 12th 2019
Recommendations at Netflix
User is going to abandon
session in...
Recommendations at Netflix
● Personalized Homepage for each member
● Goal: quickly help members find content they love
● Challenge:
○ 150M+ members in 190 countries
○ New content added daily
● Recommendations Valued at: $1B*
*Carlos A. Gomez-Uribe, Neil Hunt: The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Trans. Management Inf. Syst. 6(4): 13:1-13:19 (2016)
Meaning
● What is this title?
● Why should I watch it?
Evidence to convey meaning
Evidence example I
Evidence example I: web homepage
Evidence example II
Billboard
iOS
Evidence personalization
● Multiple choices for each (show, evidence) pair
● For each title, hard to predict what resonates before launch
● Taste can change over time
Evidence personalization (cont’d)
● Asset choreography (slate): image, video, text, cast, etc
Bandit
Bandit setup
● Generic framework to obtain unbiased training data
● Very vast literature (beyond scope)
● Every session is a mini A/B test
Pred 1-Pred
reward?
Bandit setup for evidence
● Goal: for each mini A/B test collect a sample with
○ Context x
○ Selected action (or treatment) a
○ Propensity p
○ Set of possible actions s
○ Reward r
● Once you have this, very similar to other ML approaches
● Obtaining this data is the hard part infrastructure-wise
(Elliot’s section)
System requirements
● Scalable: at Netflix’s global size
● Generic: common policy (or model) and offline metrics API
● Flexible:
○ Each recommendation problems has different
attribution, reward and context definition
○ Easy to add new canvases to the bandit system (image,
video, text, cards, etc)
Policy API
● Class Slate (List[Items] items)
○ Vector of ids to recreate a composition
(“row_10”, “img_124”, “img_1037”,...,
“synopsis_32”, “card_101”)
1
2 3 4 5
7 6
8
9
Policy API
● Slate select(List[List[Item]] items)
○ Given a list of possible slates, return one of them
(explore-exploit trade-off)
● Map[Slate, Double] propensity(List[List[Item]
items)
○ Return propensity of each possible slate to debias data
Offline policy evaluation
● Unbiased offline evaluation from logged samples
● Inverse Propensity Scoring (IPS), Doubly Robust, etc
E[Reward] = 2 / 3
User 1 User 2 User 3 User 4 User 5 User 6
Logged stochastic
treatment
Play?
(binary
reward)
New policy
assignment
Offline policy evaluation system
● Input:
○ Logged sample (x, a, s, r, p)
○ Trained policy
● Output:
○ Metric (IPS, SNIPS, DM, DR, RECAP, etc)
● Observations:
○ Trivially parallelizable (decomposes per sample)
○ Each sample needs to consider all possible candidates
Data
Actions taken by a Netflix member
- Log-in
- Search for “stra”
- Play Stranger Things
...
Data:
Member Activity
Describes Netflix’s desire to take a specific action
- Recommend Stranger Things
- Display Image
...
Data:
Intent-to-Treat (ITT)
The actual experience as seen by the Netflix member
- Showed Stranger Things on Home Page
- Displayed Image
...
Data:
Treatment
Log In Play Stranger
Things
Post-PlayIntent-to-treatHome
Page
13 Reasons Why
on Popular on
Netflix
Stranger
Things
on My List
Treatment
Member Actions
“Closing the
Loop”
“Closing the
Loop”:
Join Intent-to-
Treat with
Treatment
- Did the intent-to-treat
take effect as desired?
- Which policy was
used?
- What were the
propensities?
- What features were
used?
“Closing the
Loop”:
Attribution /
Reward
Computation
- Define reward based
on short-/long-term
member activity
- Compute in “near”
real-time
“Closing the
Loop”:
Update and
Publish Policy
- Incorporate incoming
observations
- Evaluate offline replay
metrics
- Push updated policy
Infrastructure
Components
Standardized
Logging
Standardize logging to simplify on-
boarding
- Common logging format
- Centralized landing
topics/tables
Real-time
Processing with
Flink
Process member activity,
treatment, and intent-to-treat
events in real-time
- Join intent-to-treat and
treatment events
- Prepare data in format
amenable to reward
computation
- Kafka and Hive outputs
Spark Client For
Attribution and
Reward
Computation
Provide flexible processing of the
joined member activity, treatment,
and intent-to-treat data
- Events sorted by time
- Unbounded windowing
support
- Optimizations for common
access patterns
Configurable,
Common
Attribution
Templatize common attribution
methods for simpler access to data
- Select treatment T events
- Attribute member actions A to
treatments
- Within time window W
- Match treatment and member
action based on key K
- Materialize as Hive tables
Challenges
Microservice Architecture
Edge
Home Page
Service
Image
Selection
Service
...
...
...
... ...
Microservice Architecture
Edge
Home Page
Service
...
Image
Selection
Service
...
...
...
ID1=ITT1:
Image-123,
Features...
ID1
ID1
...
Microservice Architecture
Edge
Home Page
Service
...
Image
Selection
Service
...
...
...
ID1,
Image-123
ID1→ITT1:
Image-123,
Features...
ID1
ID1
...
ID1, Image-
123
ID1, Image-
123
ID1, Image-
123
Microservice Architecture
Edge
Home Page
Service
...
Image
Selection
Service
...
...
...
ID1,
Image-123
ID1→ITT1:
Image-123,
Features...
ID1
ID1
...
ID1, Image-
123
ID1, Image-
123
ID1, Image-
123
Microservice Architecture
Edge
Home Page
Service
...
Image
Selection
Service
...
...
...
Cache
ITT1:
Image-123,
Features..
...
Microservice Architecture
Edge
Home Page
Service
...
Image
Selection
Service
...
...
...
Cache
ITT1:
Image-123,
Features..
ID2
...
ID2,
Image-123
ID2, Image-
123
ID2, Image-
123
Microservice Architecture
Edge
Home Page
Service
...
Image
Selection
Service
...
...
...
Cache
ITT1:
Image-123,
Features...
ID2,
Image-123
ID2
ID2 →
ITT1
...
ID2, Image-
123
ID2, Image-
123
Stream Processing
- Out-of-order data
- Maintaining state for joins
- Widely-varying traffic patterns
Scale
- 150M+ members
- 190 countries
- 450B+ unique events per day
- Many microservices
- Many personalization
algorithms
Used in Production as well
as multiple A/B tests
Thank you

More Related Content

What's hot

Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender modelsParmeshwar Khurd
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsJustin Basilico
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender SystemsYves Raimond
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...MLconf
 
Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at NetflixJustin Basilico
 
An introduction to Recommender Systems
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender SystemsDavid Zibriczky
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixJustin Basilico
 
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Anoop Deoras
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...Sudeep Das, Ph.D.
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixGrace T. Huang
 
Personalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningPersonalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningAnoop Deoras
 
Recent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveRecent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveJustin Basilico
 
Contextualization at Netflix
Contextualization at NetflixContextualization at Netflix
Contextualization at NetflixLinas Baltrunas
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at NetflixLinas Baltrunas
 
ML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talkML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talkFaisal Siddiqi
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Xavier Amatriain
 
Learning a Personalized Homepage
Learning a Personalized HomepageLearning a Personalized Homepage
Learning a Personalized HomepageJustin Basilico
 
Déjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender SystemsDéjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender SystemsJustin Basilico
 

What's hot (20)

Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender models
 
Personalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing RecommendationsPersonalized Page Generation for Browsing Recommendations
Personalized Page Generation for Browsing Recommendations
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
 
Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at Netflix
 
An introduction to Recommender Systems
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender Systems
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019Tutorial on Deep Learning in Recommender System, Lars summer school 2019
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at Netflix
 
Entity2rec recsys
Entity2rec recsysEntity2rec recsys
Entity2rec recsys
 
Personalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningPersonalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep Learning
 
Recent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveRecent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix Perspective
 
Contextualization at Netflix
Contextualization at NetflixContextualization at Netflix
Contextualization at Netflix
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at Netflix
 
ML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talkML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talk
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
 
Learning a Personalized Homepage
Learning a Personalized HomepageLearning a Personalized Homepage
Learning a Personalized Homepage
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Déjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender SystemsDéjà Vu: The Importance of Time and Causality in Recommender Systems
Déjà Vu: The Importance of Time and Causality in Recommender Systems
 

Similar to Netflix talk at ML Platform meetup Sep 2019

A fresh new look into Information Gathering - OWASP Spain
A fresh new look into Information Gathering - OWASP SpainA fresh new look into Information Gathering - OWASP Spain
A fresh new look into Information Gathering - OWASP SpainChristian Martorella
 
Fairness, Transparency, and Privacy in AI @ LinkedIn
Fairness, Transparency, and Privacy in AI @ LinkedInFairness, Transparency, and Privacy in AI @ LinkedIn
Fairness, Transparency, and Privacy in AI @ LinkedInKrishnaram Kenthapadi
 
Real-time big data analytics based on product recommendations case study
Real-time big data analytics based on product recommendations case studyReal-time big data analytics based on product recommendations case study
Real-time big data analytics based on product recommendations case studydeep.bi
 
Pinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at PinterestPinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at PinterestAlluxio, Inc.
 
IRIS.TV Talks Future of Video Personalization at Cross Campus LA
IRIS.TV Talks Future of Video Personalization at Cross Campus LAIRIS.TV Talks Future of Video Personalization at Cross Campus LA
IRIS.TV Talks Future of Video Personalization at Cross Campus LAIRIS.TV
 
Snowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWSSnowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWSGiuseppe Gaviani
 
Supersize your production pipe enjmin 2013 v1.1 hd
Supersize your production pipe    enjmin 2013 v1.1 hdSupersize your production pipe    enjmin 2013 v1.1 hd
Supersize your production pipe enjmin 2013 v1.1 hdslantsixgames
 
Big Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTS
Big Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTSBig Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTS
Big Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTSMatt Stubbs
 
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...Dataiku
 
Making Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, Reddit
Making Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, RedditMaking Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, Reddit
Making Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, RedditLucidworks
 
Using Apache Kafka to Analyze Session Windows
Using Apache Kafka to Analyze Session WindowsUsing Apache Kafka to Analyze Session Windows
Using Apache Kafka to Analyze Session Windowsconfluent
 
SplunkLive! Presentation - Data Onboarding with Splunk
SplunkLive! Presentation - Data Onboarding with SplunkSplunkLive! Presentation - Data Onboarding with Splunk
SplunkLive! Presentation - Data Onboarding with SplunkSplunk
 
Apache Unomi presentation and update. By Serge Huber, CTO Jahia
Apache Unomi presentation and update. By Serge Huber, CTO JahiaApache Unomi presentation and update. By Serge Huber, CTO Jahia
Apache Unomi presentation and update. By Serge Huber, CTO JahiaJahia Solutions Group
 
ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...
ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...
ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...Serge Huber
 
How Celtra Optimizes its Advertising Platform with Databricks
How Celtra Optimizes its Advertising Platformwith DatabricksHow Celtra Optimizes its Advertising Platformwith Databricks
How Celtra Optimizes its Advertising Platform with DatabricksGrega Kespret
 
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
user Behavior Analysis with Session Windows and Apache Kafka's Streams APIuser Behavior Analysis with Session Windows and Apache Kafka's Streams API
user Behavior Analysis with Session Windows and Apache Kafka's Streams APIconfluent
 
Open source for customer analytics
Open source for customer analyticsOpen source for customer analytics
Open source for customer analyticsMatthias Funke
 
PSUG 1 - 2024-01-22 - Onboarding Best Practices .pdf
PSUG 1 - 2024-01-22 - Onboarding Best Practices .pdfPSUG 1 - 2024-01-22 - Onboarding Best Practices .pdf
PSUG 1 - 2024-01-22 - Onboarding Best Practices .pdfTomas Moser
 

Similar to Netflix talk at ML Platform meetup Sep 2019 (20)

A fresh new look into Information Gathering - OWASP Spain
A fresh new look into Information Gathering - OWASP SpainA fresh new look into Information Gathering - OWASP Spain
A fresh new look into Information Gathering - OWASP Spain
 
Fairness, Transparency, and Privacy in AI @ LinkedIn
Fairness, Transparency, and Privacy in AI @ LinkedInFairness, Transparency, and Privacy in AI @ LinkedIn
Fairness, Transparency, and Privacy in AI @ LinkedIn
 
Real-time big data analytics based on product recommendations case study
Real-time big data analytics based on product recommendations case studyReal-time big data analytics based on product recommendations case study
Real-time big data analytics based on product recommendations case study
 
Pinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at PinterestPinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at Pinterest
 
IRIS.TV Talks Future of Video Personalization at Cross Campus LA
IRIS.TV Talks Future of Video Personalization at Cross Campus LAIRIS.TV Talks Future of Video Personalization at Cross Campus LA
IRIS.TV Talks Future of Video Personalization at Cross Campus LA
 
Snowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWSSnowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWS
 
Supersize your production pipe enjmin 2013 v1.1 hd
Supersize your production pipe    enjmin 2013 v1.1 hdSupersize your production pipe    enjmin 2013 v1.1 hd
Supersize your production pipe enjmin 2013 v1.1 hd
 
Big Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTS
Big Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTSBig Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTS
Big Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTS
 
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...
 
kdd2015
kdd2015kdd2015
kdd2015
 
Making Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, Reddit
Making Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, RedditMaking Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, Reddit
Making Reddit Search Relevant and Scalable - Anupama Joshi & Jerry Bao, Reddit
 
Using Apache Kafka to Analyze Session Windows
Using Apache Kafka to Analyze Session WindowsUsing Apache Kafka to Analyze Session Windows
Using Apache Kafka to Analyze Session Windows
 
SplunkLive! Presentation - Data Onboarding with Splunk
SplunkLive! Presentation - Data Onboarding with SplunkSplunkLive! Presentation - Data Onboarding with Splunk
SplunkLive! Presentation - Data Onboarding with Splunk
 
Apache Unomi presentation and update. By Serge Huber, CTO Jahia
Apache Unomi presentation and update. By Serge Huber, CTO JahiaApache Unomi presentation and update. By Serge Huber, CTO Jahia
Apache Unomi presentation and update. By Serge Huber, CTO Jahia
 
ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...
ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...
ApacheCon NA 2018 : Apache Unomi, an Open Source Customer Data Platformapache...
 
F8 tech talk_pinterest_v4
F8 tech talk_pinterest_v4F8 tech talk_pinterest_v4
F8 tech talk_pinterest_v4
 
How Celtra Optimizes its Advertising Platform with Databricks
How Celtra Optimizes its Advertising Platformwith DatabricksHow Celtra Optimizes its Advertising Platformwith Databricks
How Celtra Optimizes its Advertising Platform with Databricks
 
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
user Behavior Analysis with Session Windows and Apache Kafka's Streams APIuser Behavior Analysis with Session Windows and Apache Kafka's Streams API
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
 
Open source for customer analytics
Open source for customer analyticsOpen source for customer analytics
Open source for customer analytics
 
PSUG 1 - 2024-01-22 - Onboarding Best Practices .pdf
PSUG 1 - 2024-01-22 - Onboarding Best Practices .pdfPSUG 1 - 2024-01-22 - Onboarding Best Practices .pdf
PSUG 1 - 2024-01-22 - Onboarding Best Practices .pdf
 

Recently uploaded

UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 

Recently uploaded (20)

UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 

Netflix talk at ML Platform meetup Sep 2019

Editor's Notes

  1. Mention study of 1B is from 2016. It is just an approximation.
  2. One aspect of recommendations is once you have selected a title for the user, trying to explain to them why the system things is a good title for them. This especially important for Netflix Originals, where there is almost no previous knowledge. For example, I go to my home page and I see this title recommended in the big billboard. OK, seems definitely romance. But is it also commedy? Or more drama? Are there any of my favourite actors/actresses? Do I have time to watch it?
  3. Imagine a recommendation homepage without any evidence… this is how it would look like. You would probably only click on things that you know really well (for example, play The Office S2:E3 for the hundredth time).
  4. TODO: run animation
  5. More detail about Data and Infra side
  6. Can be different from ITT - fall backs, business rules overriding
  7. Timeline view of when this data is produced/by whom
  8. Connect these different discrete pieces of data back together to update our policies
  9. Allows us to answer important questions
  10. Ctr, abandonment…. As soon as possible to allow policies up-to-date/reactive to changes
  11. Cool stuff Fernando mentioned before.
  12. As interest in bandits grew organically, learn from every individual use case and build a number of components to make things easier and more reusable.
  13. Instead of each application logging to different places in different formats… Ingesting new data becomes easier because uniform.
  14. Make data available ASAP
  15. Library to access data in batch. Provide events in sorted order, unbounded….
  16. In addition to library access, found common patterns in the way rewards are computed. Templatize this so we can materialize this data for our consumers regularly, enable simple SQL access, with no job scheduling/operations work involved.
  17. I’ll highlight a few high-level challenges we faced
  18. Netflix powered by microservices. All requests from the devices go through a single layer called Edge and it fans out to call many microservices. Makes joining ITT and Treatment a bit challenging.
  19. Let’s say we are collecting data for an algorithm used for selecting the best image for a video. Simple case - ID1 is minted and threaded through subsequent requests for tracing - logged and then passed back to device and then logged with treatment. Join directly.
  20. Let’s say we are collecting data for an algorithm used for selecting the best image for a video. Simple case - ID1 is minted and threaded through subsequent requests for tracing - logged and then passed back to device and then logged with treatment. Join directly.
  21. Let’s say we are collecting data for an algorithm used for selecting the best image for a video. Simple case - ID1 is minted and threaded through subsequent requests for tracing - logged and then passed back to device and then logged with treatment. Join directly.
  22. Many cases - caching/precompute. Computation service is unaware of the use of its output when logging
  23. Many cases - caching/precompute. Computation service is unaware of the use of its output when logging. Cannot connect T2 and ITT1. Could approximate with time but not accurate.
  24. Introduced new thing to be logged as part of standardization - id pairs. Each service that uses data logs id pairs when using data (not computing data) so that we can resolve the ids logged with treatment and ITT data.
  25. Implemented as stream processing - typical challenges faced
  26. A lot of IDs and a lot of (Terabytes) state to be dealt with
  27. The question you probably want to ask… Only scratched the surface - many more parts of the experience to personalize using bandits. Infra will make it much simplet...