Machine Learning At Tubi

© Tubi, proprietary and confidential
Machine Learning at Tubi:
Powering Free Movies, TV
and News for All
Jaya Kawale
Vice President of Engineering (Machine Learning), Tubi

01 Introduction
2

Tubi
● Advertiser Based Video On Demand
service (AVOD)
● Watch free movie, tv, news & sports
● More than 51 Million monthly active
users
● Available across several countries
including US, Canada, Mexico &
LatAm.
3

02 Content Understanding
5

ML for Content
● Content understanding helps
understand the rich metadata
Helps us improve
● Recommendations
● Content acquisition decisions
● Cold starting of titles
● Image ranking
● Content categorization
● …
6
Plot Synopsis
Cast
Genre
Box office
Ratings
Posters/ images
Language
Video trailers

Content Understanding
Easy
Hard
Keyword Search
Review/ Sentiment Classification
Topic Extraction
Embedding Generation
Natural Language Understanding
Video Understanding
Multi-modal data Understanding
(e.g. Text + Images)

Lessons Learned
● Pre-processing and cleaning up is very critical.
● Different tasks require different texts. E.g. sentiment analysis vs
summarization.
● Not all the text is the same. E.g. Reviews vs Subtitles vs Synopsis.

Lessons Learned
● Be careful with the choice of algorithms. E.g. BERT more suitable for next
sentence prediction. “No free lunch” in terms of algorithm & representation.
● Averaging widely used but can lose information. E.g. multiple reviews for a title
averaged together to generate a title embedding. Transformer based fusion.
● Evaluation is hard but critical. E.g. embedding quality assessment on
surrogate tasks.

Spock Platform
● Platform for data ingestion, preprocessing and cleaning.
● Generates a variety of embeddings powering the different use cases across
the product.
● Helps assess embeddings quality via surrogate tasks.

1st & 3rd Party
Data
Audience
Assessment
Viewer-oriented data
Title-oriented
data
Products
Models
Embeddings (CTXT, MD, MMD,
Genre, Demos, Actor, et al)
Universe of Content + Metadata
Use Cases
Beam from
Universe to
Tubiverse
Cold➔
Warm➔Hot
Starting
Content Value
Assessment
Tiering
Inventory in
Tubiverse
Augmented
Search
Seeding
Growth
Coordinated
Pursuit of New
Audience
Portfolio
Analysis /
Simulation
Spock Platform

03 Recommendation
12

Personalized Recommendations
Content Ranking
Container
Ranking
Image Ranking
Search
Notifications
Container
Generation
Cold starting
titles

Personalized Recommendations
Content Ranking
Container
Ranking
Image Ranking
Search
Notifications
Container
Generation
Cold starting
titles
● 70+ models helping organize the
homepage!
● Rank content and containers based on
users’ features and past interactions
● Ranking based on GBDT, Deep Neural
Network
● Retrieval based upon a lot of
Embeddings (e.g. two tower, BERT, etc)
● Distilled models for new users
● Exploration strategies for new titles

01
02
03
04
Offline vs Online
Feedback loops
Changing tastes and catalog
Algorithmic fairness and biases
Why is it challenging ?

01
02
03
04
Offline vs Online
Feedback loops
Changing tastes and catalog
Algorithmic fairness and biases
Why is it challenging ?
Beyond Accuracy
at the Top!

01 Offline vs Online
17

Typical Metrics
18
Typical Offline
Metrics
Typical Online
Metrics
Ranking metrics:
NDCG, NMRR,
Precision @K
Streaming,
Retention

Correlation vs Causation
Offline evaluation
● Use historical data
● Cheap, fast, risk free
● Correlation based
● Counterfactuality of rewards: Do not capture what would have happened if ?
Online evaluation
● Randomized experiment (A/B tests)
● Wait for days to compute the reward
● Reliable but expensive
19

Dynamic Environment
● Recommender dynamics can affect the performance in ways not captured by
the offline metrics. E.g. impression caps.
● Recommendations can influence user preferences in ways not captured by
offline metrics. E.g. Did you watch a title because it was recommended ?
● User dynamics and confounding factors can influence the watch behavior in
ways not captured by offline metrics. E.g. Watching a title because it was
recommended by a friend.
20

Counterfactual evaluation
● Estimate the potential outcome of a policy offline using logged data.
● Inverse Propensity Scoring (IPS): Importance weighting to account for the
mismatch in the distribution of logged data and the policy to evaluate.
● Several variants - CIPS, SNIPS, etc.
21

02 Feedback loops
22
03 Changing User tastes & Catalog

Feedback loops
● Different algorithms on the
homepage influencing one another
● Underlying data influencing the
algorithms
● Recommendations influencing the
watch behavior
● Watch behavior influences the
data.
23
Observational Data

Feedback loops
● Typical offline training - clicks, watched, plays
● Implicit feedback has inherent biases
● Position/ recommendations influence the data collection
24

Changing User tastes & Catalog
● Users adapt and change their preferences over time.
● Also, new titles come into the system whereas some others leave the service.
● Uncertainty around new users and titles
● Trends outside influence the watch behavior.
25

● Tradeoff: Explore unknown choices to gather
information vs exploit known preferences.
● Exploration helps break feedback loops and
helps with uncertainty around new items/
users.
● Caveat: Designing good exploration that works
in practice is hard due to non-stationarity of
the data and large dynamic action spaces.
Reward is myopic
● RL: Optimize long term
Exploration and bandits
26

04 Algorithmic fairness and biases
27

Ranking systems
● Ranking function: Rank items
for a given context →
Learning to Rank
● Slate recommendations: slate
of recommended titles
● Are traditional methods fair ?
28

Fairness
● Many aspects of discrimination relate to correlation vs causation.
● Gender and race may be correlated with factors that shift the distributions.
● Reasons of bias - skewed examples, tainted examples, limited features,
sample size disparity.
29

Fairness
● Pre-processing: Distributions of specific
sensitive or protected variables are biased.
Train a model on a “repaired” data.
● In-processing: Modeling techniques often
become biased by dominant features, other
distributional effects. Incorporate fairness
metrics into the model optimization
● Post-processing: Apply transformations to
improve predictions fairness.
30
Caton & Hass: Fairness in ML survey, 2020

04 Conclusion
31

Conclusion & Acknowledgements
● Lots of challenging problems in content understanding users and
recommendations.
● The road ahead is very exciting and we are hiring!
● All the work is done by my awesome Machine Learning team @Tubi.
32

Thank you.
Jaya Kawale
Email: jkawale@tubi.tv
Twitter: jayakawale

Machine Learning At Tubi

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Machine Learning At Tubi

Similar to Machine Learning At Tubi (20)

Recently uploaded

Recently uploaded (20)

Machine Learning At Tubi