This document provides an overview of bandit algorithms and their applications. It begins by explaining the multi-armed bandit problem and some basic algorithms like epsilon-greedy to solve it. It then discusses more advanced techniques like Thompson sampling and their benefits over naive approaches. Finally, it outlines several real-world uses of bandit algorithms, including UI optimization, recommendation systems, and multiplayer games. Bandit algorithms provide a powerful way to optimize outcomes in situations where rewards are not immediately revealed.

1. BANDIT ALGORITHMS
Max Pagels, Machine Learning Specialist
max.pagels@sc5.io, @maxpagels
Understanding the world using reinforcement learning
DATA & AI

2. ABOUT ME
MAX PAGELS
MSc, BSc, Computer Science (University of Helsinki),
former CS researcher
Currently a machine learning specialist at Helsinki-
based software consultancy SC5, working on applied AI
& cloud-native solutions
Specific areas of interest include probabilistic
supervised learning, deep neural networks and
(algorithmic) complexity theory
Other hats I sometimes wear: full-stack developer,
technical interviewer, architect

3. TRADITIONAL PROGRAMMING, IN A NUTSHELL
OUTPUTINPUT
ALGORITHM

4. MACHINE LEARNING, IN A NUTSHELL
PROGRAMINPUT & OUTPUT
LEARNING ALGORITHM

5. REINFORCEMENT LEARNING, IN A NUTSHELL
ACTION
WORLD
OBSERVATION
AGENT
REWARD

6. ACTION
WORLD
OBSERVATION
AGENT
REWARD
(Rotate/Up/Down/
Left/Right etc)
Score since
previous action
LEARNING HOW TO PLAY TETRIS

7. ACTION
WORLD
STATE
AGENT
REWARD
(Rotate/Up/Down/
Left/Right etc)
Score since
previous action
If fully observable
LEARNING HOW TO PLAY TETRIS

8. IF REWARDS AREN’T DELAYED, IT’S A BANDIT PROBLEM

9. THE (STOCHASTIC)
MULTI-ARMED
BANDIT PROBLEM
Let’s say you are in a Las Vegas casino, and
there are three vacant slot machines (one-
armed bandits). Pulling the arm of a machine
yields either $0 or $1.
Given these three bandit machines, each with
an unknown payoff strategy, how should we
choose which arm to pull to maximise* the
expected reward over time?
* Equivalently, we want to minimise the expected
regret over time (minimise how much money we
lose).

10. FORMALLY(ISH)
Problem
We have three one-armed bandits. Each arm
yields a reward of $1 according to some fixed
unknown probability Pa , or a reward of $0 with
probability 1-Pa.
Objective
Find Pa, ∀a ∈ {1,2,3}, play argmax(Pa, ∀a ∈ {1,2,3})

11. OBSERVATION #1
This is a partial information problem: when we
pull an arm, we don’t get to see the rewards of
the arms we didn’t pull.
(For math aficionados: a bandit problem is an
MDP with a single, terminal, state).

12. OBSERVATION #2
We need to create some approximation, or best
guess, of how the world works in order to
maximise our reward over time.
We need to create a model.
“All models are wrong but some are useful” —
George Box

13. OBSERVATION #3
Clearly, we need to try (explore) the arms of all
the machines, to get a sense of which one is
best.

14. OBSERVATION #4
Though exploration is necessary, we also need
to choose the best arm as much as possible
(exploit) to maximise our reward over time.

15. THE EXPLORE/EXPLOIT TUG OF WAR
< ACQUIRING KNOWLEDGE MAXIMISING REWARD >

16. HOW DO WE SOLVE THE BANDIT PROBLEM?

17. A NAÏVE ALGORITHM
Step 1: For the first N pulls
Evenly pull all the arms, keeping track of the
number of pulls & wins per arm
Step 2: For the remaining M pulls
Choose the arm the the highest expected
reward (ie. mean reward)

18. A NAÏVE ALGORITHM
Step 1: For the first N pulls
Evenly pull all the arms, keeping track of the
number of pulls & wins per arm
Step 2: For the remaining M pulls
Choose the arm the the highest expected
reward (ie. mean reward)
Arm 1: {trials: 100, wins: 2}
Arm 2: {trials: 100, wins: 21}
Arm 3: {trials: 100, wins: 17}
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17

19. A NAÏVE ALGORITHM
Step 1: For the first N pulls
Evenly pull all the arms, keeping track of the
number of pulls & wins per arm
Step 2: For the remaining M pulls
Choose the arm the the highest expected
reward (ie. mean reward)
Arm 1: {trials: 100, wins: 2}
Arm 2: {trials: 100, wins: 21}
Arm 3: {trials: 100, wins: 17}
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17
Explore
Exploit

20. EPOCH-GREEDY [1]
Step 1: For the first N pulls
Evenly pull all the arms, keeping track of the
number of pulls & wins per arm
Step 2: For the remaining M pulls
Choose the arm the the highest expected
reward (ie. mean reward)
Arm 1: {trials: 100, wins: 2}
Arm 2: {trials: 100, wins: 21}
Arm 3: {trials: 100, wins: 17}
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17
Explore
Exploit
[1]: http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf

21. EPSILON-GREEDY/Ε-GREEDY [2]
Choose a value for the hyperparameter ε (e.g. 0.1 = 10%)
Loop forever
With probability ε:
Choose an arm uniformly at random, keep track of trials and wins
With probability 1-ε:
Choose the arm the the highest expected reward
Explore
Exploit
[2]: people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf

22. OBSERVATIONS
For N = 300 and M = 1000, the Epoch-Greedy algorithm will choose
something other than the best arm 200 times, or about 15% of the time.
At each turn, the ε-Greedy algorithm will choose something other than the
best arm with probability P=(ε/n) × (n-1), n=number of arms. It will always
explore with this probability, no matter how many time steps we have
done.

23. Ε-GREEDY: SUB-OPTIMAL LINEAR REGRET*
- O(N)Cumulativeregret
4
Timesteps
*Assuming no annealing

24. THOMPSON SAMPLING ALGORITHM
For each arm, initialise a uniform probability distribution (prior)
Loop forever
Step 1: sample randomly from the probability distribution of each
arm
Step 2: choose the arm with the highest sample value
Step 3: observe reward for the chosen and update the
hyperparameters of its probability distribution (posterior)

25. EXAMPLE WITH TWO ARMS (BLUE & GREEN)
Assumption: rewards follow a Bernoulli process, which allows us to use a 𝛽-distribution as a
conjugate prior.
FIRST, INITIALISE A UNIFORM RANDOM PROB.
DISTRIBUTION FOR BLUE AND GREEN
0 10 1

26. FIRST, INITIALISE A UNIFORM RANDOM PROB.
DISTRIBUTION FOR BLUE AND GREEN
0 1
EXAMPLE WITH TWO ARMS (BLUE & GREEN)

27. RANDOMLY SAMPLE FROM BOTH ARMS (BLUE GETS THE
HIGHER VALUE IN THIS EXAMPLE)
0 1
EXAMPLE WITH TWO ARMS (BLUE & GREEN)

28. PULL BLUE , OBSERVE REWARD (LET’S SAY WE GOT A
REWARD OF 1)
0 1
EXAMPLE WITH TWO ARMS (BLUE & GREEN)

29. UPDATE DISTRIBUTION OF BLUE
0 1
EXAMPLE WITH TWO ARMS (BLUE & GREEN)

30. REPEAT THE PROCESS: SAMPLE, CHOOSE ARM WITH
HIGHEST SAMPLE VALUE, OBSERVE REWARD, UPDATE.
0 1
EXAMPLE WITH TWO ARMS (BLUE & GREEN)

31. 0 1
AFTER 100 TIMESTEPS, THE DISTRIBUTIONS MIGHT LOOK
LIKE THIS
EXAMPLE WITH TWO ARMS (BLUE & GREEN)

32. 0 1
AFTER 1,000 TIMESTEPS, THE DISTRIBUTIONS MIGHT LOOK
LIKE THIS: THE PROCESS HAS CONVERGED.
EXAMPLE WITH TWO ARMS (BLUE & GREEN)

33. THOMPSON SAMPLING: LOGARITHMIC REGRET -
O(LOG(N))Cumulativeregret
100
Timesteps

34. MULTI-ARMED BANDITS ARE
GOOD…
So far, we’ve covered the “vanilla” multi-armed bandit problem. Solving this
problem lets us find the globally best arm to maximise our expected reward
over time.
However, depending on the problem, the globally best arm may not always
be the best arm.

35. …BUT CONTEXTUAL BANDITS
ARE THE HOLY GRAIL
The the contextual bandit setting, once we pull an arm and receive a reward,
we also get to see a context vector associated with the reward. The
objective is now to maximise the expected reward over time given the
context at each time step.
Solving this problem gives us a higher reward over time in situations where
the globally best arm isn’t always the best arm.
Example algorithm: Thompson Sampling with Linear Payoffs [3]
[3]: https://arxiv.org/abs/1209.3352

36. …ACTUALLY, ADVERSARIAL
CONTEXTUAL BANDITS ARE THE
HOLY GRAIL
Until now, we’ve assumed that each one-armed bandit gives a $100 reward
according to some unknown probability P. In real-world scenarios, the world
rarely behaves this well. Instead of a simple dice roll, the bandit may pay out
differently depending on the day/hour/amount of money in the machine. It
works as an adversary of sorts.
A bandit algorithm that is capable of dealing with changes in payoff
structures in a reasonable amount of time tend to work better than
stochastic bandits.
Example algorithm: EXP4 (a.k.a the “Monster” algorithm) [4]
[4]: https://arxiv.org/abs/1002.4058

37. CONTEXTUAL BANDIT
ALGORITHMS ARE 1) STATE-OF-
THE-ART & 2) COMPLEX
• Many algorithms are <5 years old. Some of the most interesting ones
are less than a year old.
• Be prepared to read lots of academic papers whilst implementing the
algorithms.
• Don’t hesitate to contact the authors if there is some detail you don’t
understand (thanks John Langford and Shipra Agrawal!).
• Always remember to simulate and validate bandits using real data.
• Always remember to log actions and rewards of deployed bandits.

38. PRACTICAL TIPS & CONSIDERATIONS
FOR BANDITS
• If you are implementing your own online-learnable MAB and serving
predictions/receiving rewards over an API, a single process may suffice
• If using Thompson Sampling, sampling 100 arms won’t take more than
a few milliseconds -> possible to implement a single-threaded real-
time learning bandit that can scale to hundreds of requests per
second
• Thousands of arms or thousands of requests per second will require
multiple processes + synchronisation
• For contextual bandits, multiple processes + batching is required
• For some applications, you may need some heuristic for what counts as a
“zero reward”
• Remember to leverage caching on the web, otherwise the UX suffers
• In some cases, you might not want convergence -> tweak algorithm to
always explore with some nonzero probability
• In most applications, the number of arms can vary at any given timestep

39. REAL-WORLD APPLICATIONS OF BANDITS

40. BIVARIATE & MULTIVARIATE TESTING
Vanilla MAB

41. UI MENU OPTIMISATION

42. CODEC SELECTION
Image credit: Microsoft/Skype

43. ONLINE MULTIPLAYER
Image credit: Microsoft/Bungie

44. PERSONALISED RECOMMENDATIONS
Contextual
MAB

45. PERSONALISED RECOMMENDATIONS
Multiple-
play MAB

46. SELFIE OPTIMISATION
Vanilla MAB
(one per
user)
Image: Tinder Tech Blog

47. … & COUNTLESS OTHER POSSIBILITIES

48. DIY: CONTEXTUAL BANDIT IMPLEMENTED IN
PYTHON + AZURE FUNCTIONS + API + REDIS
= OPTIMISATION ENGINE

49. AS-A-SERVICE: AZURE CUSTOM
DECISION SERVICE

50. IF YOU CAN MODEL IT AS A
GAME, YOU CAN USE A BANDIT
ALGORITHM TO SOLVE IT*.
*Given rewards that aren’t delayed.

51. THANK YOU
A special thanks to Microsoft for hosting & Practical AI for organising!
QUESTIONS?