The document discusses sequential decision-making in recommendation systems, particularly in the context of Netflix's personalized content recommendations for its 158 million subscribers across ~200 countries. It highlights challenges such as unknown state environments, rewards, and transitions, and explores multi-armed bandit strategies and reinforcement learning techniques for maximizing user satisfaction over time. Various methods and considerations for optimizing recommendations, including exploration versus exploitation and dealing with high-dimensional action spaces, are also presented.

1. Sequential Decision Making in
Recommendations
Jaya Kawale
Toronto Machine Learning Conference,
Nov 2019

3. Quickly help members discover content they’ll love

4. Global Members, Personalized Tastes
158 Million Subscribers
~200 Countries

5. From what shows to recommend

6. Personalized images.
... to what images to select

7. ... to reaching out to our members

8. Everything is a Recommendation!*
* Xavier Amatrian and Justin Basilico - https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429

9. Sequence of decisions
visit
click
play
visit
visit
play
thum
bs
up
time
m
ylistadd

10. Optimize satisfaction
Satisfaction
visit
click
play
visit
visit
play
time

11. Optimize satisfaction
Satisfaction
visit
click
play
visit
visit
play
time
How should
recommendation
algorithms take this
into account ?

12. Sequential Decision
Making
Recommendation
Observations
Rewards
Clinical
Trials
Network
Routing
Online
Advertising
AI for
Games
Hyperparameter
Optimization

13. Recommendation as a Sequential Decision Making
Problem
Learner Environment
Action
Reward
Context

14. Why is it challenging ?
… Because we don’t know!
● The current environment or state: We may not have full knowledge of the
state we are in.
● Reward: Taking an action at some state results in some reward not known
beforehand.
● The transition dynamics: No knowledge of how the environment or state
changes due to a particular action.

15. Multi-Armed Bandits

16. Multi-Armed Bandits
● A gambler playing multiple slot machines with
unknown reward distribution
● Which machine to play to maximize reward?

17. Multi-Armed Bandit For Recommendation
Exploration-Exploitation tradeoff :
Recommend the optimal title given the evidence i.e. exploit
OR
Recommend other titles to gather feedback i.e. explore.

18. Numerous Variants
● Different Strategies: ε-Greedy, Thompson Sampling (TS), Upper Confidence
Bound (UCB), etc.
● Different Environments:
○ Stochastic and stationary: Reward is generated i.i.d. from a distribution
specific to the action. No payoff drift.
○ Adversarial: No assumptions on how rewards are generated.
● Different objectives: Cumulative regret, tracking the best expert
● Continuous or discrete set of actions, finite vs infinite
● Extensions: Varying set of arms, Contextual Bandits, etc.

19. Epsilon Greedy for MABs
● Unbiased
training data
● Greedy
● Select optimal
action
Explore
ε 1-ε
Exploit

20. Greedy Exploit Policy
Member
Features
Candidate Pool
Model 1
Winner
Probability Of Play
Model 2
Model 3
Model 4

21. Considerations for the greedy policy
● Explore
○ Bandwidth allocation and cost of exploration
○ New vs existing titles
● Exploit
○ Title availability
○ Frequency of model update
○ Incremental updates vs batch training
■ Non-stationarity of title popularities
?
?
?
? ??
?

22. Opportunity Cost
Netflix homepage is an expensive real-estate:
- so many titles to promote
- so few opportunities to win a “moment of truth”
D1 D2 D3 D4 D5
Promote?▶ ▶ ▶ ▶
Probability of
Play
Days

23. Lots of feedback loops...

24. p(Y|X, do(R))
Recommendation
Build policy, e.g. what R leads to max Y?

25. Some approaches
● Bandit approaches (with caveats)
● Counterfactual Risk Minimization [Swaminathan & Joachims, 2015]
● IPS Estimator for MF [Schnabel et al., 2016]
○ Train a debiasing model and reweight the data
● Causal Embeddings [Bonner & Vasile, 2018]
○ Jointly learn debiasing model and task model
○ Regularize the two towards each other
● Doubly-Robust MF [Wang et al., 2019]

26. Reinforcement
Learning

27. Long-term Reward: Road to RL
● Maximize user long term satisfaction rather than play clicks or
duration.

28. Some Preliminaries
Everything
we know
about the
user
Action
Changing
User
preferences
Reward
Some
starting
point / user
state
Discount
Factor

29. Policy Gradients
● Learn a policy that maximizes the cumulative
future reward from time t.
● Maximization solved by gradient w.r.t. some policy
parameter.
● E.g. Reinforce*
*Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, Ed H. Chi: Top-K Off-Policy
Correction for a REINFORCE Recommender System. WSDM 2019: 456-464

30. Deep Q-Learning
● Q-value: Optimal value for a state action pair.
● Off-policy algorithm
● Directly learn the function to approximate the
Q-value.
● Challenges on training and making it work in
practice.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin A.
Riedmiller: Playing Atari with Deep Reinforcement Learning CoRR abs/1312.5602 (2013)

31. Many more challenges...
● High-dimensional action space: Recommending a single item is
O(|C|); typically want to do ranking or page construction, which is
combinatorial, e.g. Marginal Slates [Dimakopoulou et al., 2019] or
SlateQ [le et al., 2019]
● Off-policy correction: Need to learn & evaluate from existing system
actions, e.g. [Chen et al., 2019] or ReCap [More et al., 2019]
● Good simulators: Requires knowing feedback for user on
recommended items, e.g. [Rohde et al., 2018]
● Changing rewards: Every action may change our ‘ground truth’
● Changing action space: New actions (items) become available and
need to be cold-started.

32. Thank you.
Jaya Kawale (jkawale@netflix.com)
Twitter: @ jayakawale