This document summarizes Yan Xu's presentation on practical applications of multi-armed bandits. Bandits can be used for personalized recommendation, such as recommending news articles, by balancing exploration of new articles with exploitation of known good articles. Amazon's bandit algorithm allows for real-time optimization of multiple variables by modeling interactions between variables. The algorithm was able to increase website conversion by 21% after a single week of optimization.

1. PRACTICAL BANDITS
FOR BUSINESS
Yan Xu
Houston Machine Learning Meetup
June 22, 2019

2. OUTLINE
- Recap on Bandit Problem
- A Contextual-Bandit Approach to Personalized News
Article Recommendation
http://rob.schapire.net/papers/www10.pdf
- An efficient bandit algorithm for realtime multivariate
optimization
https://www.kdd.org/kdd2017/papers/view/an-
efficient-bandit-algorithm-for-realtime-multivariate-
optimization

3. MULTI-ARMED BANDITS

4. DILEMMA: EXPLORATION VS.
EXPLOITATION
The exploration/exploitation trade-off is a dilemma we
frequently face in choosing between options.
Stay the same route to drive home, or try a new route?
Choose your favorite restaurant, or the new one?
Listen to your favorite music channel, or try a new artist?
Attend a new meetup?

5. HOW TO RESOLVE THE
DILEMMA
https://pavlov.tech/2019/03/02/animated-multi-
armed-bandit-policies/
Epsilon Greedy
UCB (Upper Confidence Bound)
Thompson Sampling

6. REWARD AND REGRET

7. REWARD AND REGRET

8. REWARD AND REGRET

9. MULTI-ARMED BANDITS
FORMULATION

10. PRACTICAL BANDITS
APPLICATION

11. BANDITS FOR PERSONALIZED
RECOMMENDATION

12. BANDITS FOR NEWS
RECOMMENDATION

13. CONTEXTUAL BANDITS

14. CONTEXTUAL BANDITS

15. CONTEXTUAL BANDITS
[0.1, 0.6]
[0.6, 0.4]
[0.7, 0.1]
[0.4, 0.2]

16. LINUCB ALGORITHM

17. LINUCB ALGORITHM

18. LINEAR DISJOINT MODEL

19. LINEAR DISJOINT MODEL

20. UPPER BOUND ILLUSTRATION

21. FEATURE FREE VS LINEAR
CONTEXTUAL BANDIT

22. BANDITS EVALUATION

23. BANDITS EVALUATION

24. BANDITS EVALUATION

25. BANDITS EVALUATION

26. DEALING WITH HIGH
DIMENSIONALITY
～1000 binary features per user; ~100 binary feature per article

27. DEALING WITH HIGH
DIMENSIONALITY

28. DEALING WITH HIGH
DIMENSIONALITY

29. RESULT: PERSONALIZED
NEWS
Omniscient: always chooses the article with highest empirical

30. CONCLUSION

31. AMAZON: BANDITS FOR
MULTIVARIATE OPTIMIZATION
Published at KDD 2017, KDD 2019 is in Alaska!

32. AMAZON: BANDITS FOR
MULTIVARIATE OPTIMIZATION

33. OPTIMIZING WEB LAYOUT

34. PROBLEM FORMULATION

35. STEP 1: PROBIT REGRESSION

36. STEP 2: THOMPSON
SAMPLING

37. STEP 3: HILLING-CLIMBING TO
DECIDE

38. SIMULATION RESULT

39. SIMULATION RESULT
Control widget interaction in simulation
through alpha_2.

40. EXPERIMENT ON REAL
TRAFFIC
• After only a single week of online
optimization, we saw a 21%
conversion increase compared to the
median layout

41. SUMMARY
Contextual bandits
 Linear payoff
 Add interaction components
 UCB: Variance estimation of expected rewards
 Thompson sampling: Sample weights from posterior distribution
Applications
 Recommendation
 Multi-variate optimization
For more details
 - A Contextual-Bandit Approach to Personalized News Article
Recommendation
http://rob.schapire.net/papers/www10.pdf
- An efficient bandit algorithm for realtime multivariate
optimization
https://www.kdd.org/kdd2017/papers/view/an-efficient-
bandit-algorithm-for-realtime-multivariate-optimization