Data Con LA 2020 Description Watchworthy is a personalized TV recommendation app that leverages over 1B votes cast by TV fans on Ranker lists. Visitors to Ranker.com can vote the newest and most talked about TV Shows up and down on various lists, from "Best New Horror Shows" to "Funniest Sitcoms Ever Made." These votes constitute a trove of anonymous crowdsourced data that gives us valuable insight into taste correlations. But despite this massive volume of users we have to train on, like most developers using a user-to-item dataset, we face the classic "cold start" issue: how do we recommend brand-new shows that relatively few people have voted on? This problem is further complicated by a request to use voting behavior (rather than metadata) as much as possible when building out this recommendation engine. We present an unique approach that parses out existing user voting profiles to create additional users, called “split users". These split users will be grouped into separate training sets to create multiple sub-models. By creating an ensemble based on the submodels and applying a "most pleasure" strategy, we achieve the goal of recommending new shows. Speakers Vincent S, Ranker, VP of Data Science Keryu Ong, Ranker, Senior Data Scientist

1. How Ranker Turned Pop Culture
Lists Into Personalized TV
Recommendations
DataCon LA 2020

2. OVERVIEWIntroduction
Dr. Vincent Seah
VP, Data Science
Ranker since July 2019
Ph.D. Mechanical Engineering
UCLA
Fullscreen Media (acq. by AT&T)
KPMG US
Inkiru (acq. by Walmart Labs)
Hoodiny Entertainment Group (acq. by PRISA)
linkedin.com/in/drnotsoevil
Ker-Yu Ong
Senior Data Scientist
Ranker since April 2020
M.Sc. Data Science
University of San Francisco
Deloitte San Francisco
Deloitte Singapore
linkedin.com/in/keryu-ong

3. OVERVIEW
● CEO, Clark Benson
● Media publisher turning engagement into IP
● Over 100 employees
● Headquartered in Los Angeles, with an office in NYC
● 40M monthly unique visitors worldwide
● More than 1B votes cast over last 10 years
● Fan-powered votable content with 10,000 lists
covering everything from TV, Movies to
Sports, Food and Lifestyle
● Products built in-house:
○ Ranker Insights
○ Watchworthy App
○ Data Science Apps

4. WATCHWORTHY
Cross-Platform, Personalized Show Recommendations Based on 1B Data Points
ONBOARDING IN-APP
● Mobile app with an unparalleled
ability to give users targeted,
personalized TV
recommendations
● Using pure, first party voting
data from Ranker website
● Available on Android and iOS

5. DS APP: RecAlgo

6. CHALLENGE
List Selection

7. CHALLENGE
Voters and Biased Voting

8. CHALLENGE
Voters Sentiment versus Metadata
Actor-Based Recs
Voter sentiment casts a
wider net of recs across
genres and decades
Genre-Based Recs
"Rom-Com"

9. CHALLENGE
“If I like this older TV show, what new TV show should I watch?”
Breaking Bad (2008) Chernobyl (2019)

10. RELATED WORK
In Good Company
● A Fairness-aware Hybrid Recommender System
○ G. Farnadi, P. Kouki, S.K. Thompson, S. Srinivasan, L. Getoor [2018]
○ “...A fair recommender system should provide rankings to the protected group that are the
same as the unprotected group…”
● Group Recommender Systems: A Virtual User Approach Based on Precedence Mining
○ V.R. Kagita, A.K. Pujari, V. Padmanabhan [2015]
○ “... introducing a virtual user that can more effectively represent a group..”
● Personalized Real-Time Movie Recommendation System: Practical Prototype and Evaluation
○ J. Zhang, Y. Wang, Z. Yuan, Q. Jin [2019]
○ “...virtual opinion leader is conceived to represent the whole cluster…”
● Innovations in Graph Representation Learning
○ A. Epasto, and B. Perozzi [2019]
○ “...we developed Splitter, an unsupervised embedding method that allows the nodes in a graph
to have multiple embeddings to better encode their participation in multiple communities…”

11. APPROACH
Recap
Recap: How do we recommend new shows when:
● A user’s input taste profile is dominated by older shows
● The rec algo training data is dominated by older shows
Reframed this as a class imbalance problem
Solvable via classification techniques
● Minority-class Upsampling (SMOTE)
● Majority-class Downsampling
● Data Augmentation

12. APPROACH
● Classification: Balance/imbalance of the classes themselves
What is class balance and imbalance in the context of a rec algo?

13. APPROACH
● Recommendation: Balance/imbalance of the class relationships
What is class balance and imbalance in the context of a rec algo?

14. EXPERIMENTS
● Upsampling votes from bridge voters
● Downsampling votes from non-bridge voters
● Applying different thresholds for
○ Vote count
○ Vote type
○ Vote spread
Things We Tried
Challenge:
Because we were preserving each user’s voting
pattern, upsampling did not change the distribution
of bridge voters’ votes

15. EXPERIMENTS
Challenge, Illustrated
User Show Year
Keryu 2019
Keryu 2010
Keryu 2005
Original
Upsampled
User Show Year
Keryu 2019
Keryu 2010
Keryu 2005
Keryu_2 2019
Keryu_2 2010
Keryu_2 2005

16. EXPERIMENTS
Foray into “Splitting”: Upsampling Bridge Votes
User Show Year
Keryu 2019
Keryu 2010
Keryu 2005
Keryu_21 2019
Keryu_21 2010
Keryu_22 2019
Keryu_22 2005
Upsampled
User Show Year
Keryu 2019
Keryu 2010
Keryu 2005
Original

17. ● What about individual models
for each bridge vote?
SPLIT SAMPLING
Foray into “Splitting” - Multiple Models
User Show Year
Keryu 2019
Keryu 2010
Keryu 2005
User Show Year
Keryu 2019
Keryu 2010
User Show Year
Keryu 2019
Keryu 2005
User Show Year
Keryu 2019
Keryu 2010
Keryu 2005
Original
m_0
m_1
m_2

18. SPLIT SAMPLING
Methodology
1. Bin shows into release year decades
2. Split bridge voters’ votes by bridge decade:
a. 1990s to new
b. 2000s to new
c. 2010s to new etc.
3. Build an overall model and individual decade-specific models
4. Ensemble to get maximum number of
new shows per recommendation stream

19. SPLIT SAMPLING
Input List
1990 -
2010 -
2019 -
1990 -
2010 -
2019 -
2010 -
2019 -
show, year, worthy
Q, 1995, 95
T, 1998, 89
X, 2019, 87
D, 2001, 75
..
show, year, worthy
M, 2009, 90
P, 2000, 89
C, 2019, 85
S, 2020, 78
..
show, year, worthy
A, 1995, 90
B, 2000, 89
E, 2011, 84
F, 2013, 81
C, 2019, 79
D, 2019, 78
G, 2012, 77
X, 2019, 77
J, 2003, 60
S, 2020, 57
..
show, year, worthy
A, 1995, 90
B, 2000, 89
X, 2019, 87
E, 2011, 84
C, 2019, 85
F, 2013, 81
C, 2019, 79
D, 2019, 78
S, 2020, 78
G, 2012, 77
X, 2019, 77
J, 2003, 60
S, 2020, 57
..
show, year, worthy
A, 1995, 90
B, 2000, 89
X, 2019, 87
C, 2019, 85
E, 2011, 84
F, 2013, 81
D, 2019, 79
S, 2020, 78
G, 2012, 77
J, 2003, 60
..
1990 -
2019 -
Overall
model
1990s
model
2000s
model
Scoring Pipeline
Split Score Merge

20. EXAMPLE
Grey's Anatomy
Law & Order: Special Victims Unit
Stranger Things
The Big Bang Theory
The Closer
The Crown
This Is Us
black-ish
Bob's Burgers
Breaking Bad
Family Guy
Fresh Off the Boat
Rick and Morty
Riverdale
The Vampire Diaries

21. EXAMPLE
Grey's Anatomy
Law & Order: Special Victims Unit
Stranger Things
The Big Bang Theory
The Closer
The Crown
This Is Us
black-ish
Bob's Burgers
Breaking Bad
Family Guy
Fresh Off the Boat
Rick and Morty
Riverdale
The Vampire Diaries

22. EXAMPLE
Grey's Anatomy
Law & Order: Special Victims Unit
Stranger Things
The Big Bang Theory
The Closer
The Crown
This Is Us
black-ish
Bob's Burgers
Breaking Bad
Family Guy
Fresh Off the Boat
Rick and Morty
Riverdale
The Vampire Diaries
with split sampling
original

23. WHAT’S NEXT
01 Extending to cross category, niche genres
Group watching and recommendations02

24. THANK YOU