1. BEARS: TOWARDS AN EVALUATION FRAMEWORK FOR
BANDIT-BASED INTERACTIVE RECOMMENDER SYSTEMS
Andrea Barraza-Urbina1
1 Data Science Institute, The Insight Centre for Data Analytics
National University of Ireland Galway
Galway, Ireland
October 7th, 2018
Georgia Koutrika 2, Mathieu d’Aquin 1 and Conor Hayes 1
Offline Evaluation for Recommender Systems Workshop (REVEAL’18)
The ACM Recommender Systems 2018 (RecSys’18)
Vancouver, Canada
2 Athena Research Center
Athens, Greece

2. AGENDA
Introduction
BEARS Evaluation Framework
Conclusion and Future Work
VIDEO OF THE PRESENTATION: https://youtu.be/GIH_ArJ-ylk

3. AGENDA
Introduction
BEARS Evaluation Framework
Conclusion and Future Work

4. 4
Exploitation
Exploration
Classical RecSys
approaches
Accuracy-based
metrics
Exploit the known
user model.
Maximize user
immediate
satisfaction
Adapt to user taste drift
Face user/item cold-start
Active Learning
Beyond-accuracy metrics
Face the Filter Bubble
Explore other preferences the
user might have.
Maximize long-term
user satisfaction
EXPLOITATION VS. EXPLORATION IN RECSYS

5. 5
Tries to Walk
Reinforcement
Branch of Machine Learning.
Learn by interacting with the
environment which provides
reward signals.
Trial – error search / learning.
REINFORCEMENT LEARNING (RL)

6. 6
RL can offer a proper framework to represent RecSys that learn from
evaluative feedback when interacting with an uncertain environment.
RS Agent
RS Environment
Single item/List
Recommendation
Explicit/Implicit
User Feedback
Current
Target User
Agent
actionrewardstate
!"#"$"
#"%&
$"%&
Environment
RECSYS AS REINFORCEMENT LEARNING

7. 7
!"#
!"$
!"%
Unknown reward
probabilities with
mean !"&
.
( trials
Maximize total payoff.
Exploit Explore
THE MULTI-ARMED BANDIT PROBLEM (MAB)
Recent RecSys solutions have proposed to address the exploration and
exploitation trade-off by using Multi-Armed Bandits.
or
?
?

8. CTR estimate =
# -./012
# /345622/782
Action Selection Strategy
8
Epsilon Greedy
Action Value
Estimates
Explore
Exploit1 − ;
;
Actions
CTR< ≈ ><
CTR? ≈ >?
CTR@ ≈ >@
MULTI-ARMED BANDITS AND RECSYS

9. 9
• Evaluation based on Supervised Learning is inappropriate to test
how a system learns over time using interactive feedback.
• There is no standard methodology to evaluate offline interactive
bandit-based RecSys.
• Each work proposes their own evaluation approach.
• No concrete benchmarks.
• Hard to compare between related works.
• All evaluation set-up details are not necessarily shared.
• Difficult to reproduce/replicate evaluation set-up.
EVALUATION CHALLENGES OF BANDIT-BASED
INTERACTIVE RECSYS

10. 10
Goals:
• Enable faster and reproducible evaluation in a common open-
source platform.
• Provide simple and extensible building blocks based on the
Reinforcement Learning framework.
• Allow to share benchmark problem settings and reusable
implementations of solution approaches.
• Incorporate flexible tools to easily keep track of metrics along the
interaction.
BEARS: BANDIT-BASED RECOMMENDER SYSTEM
EVALUATION FRAMEWORK

11. AGENDA
Introduction
BEARS Evaluation Framework
Conclusion and Future Work

12. 12
MAIN COMPONENTS
Experiment Evaluator RS Agent RS Environment
Assess the
agent-environment
interaction.
Keep track of
multiple metrics
over time steps.
Solution
Approach.
Representation of
the real-world
problem setting.
BEARS: BANDIT-BASED RECOMMENDER SYSTEM
EVALUATION FRAMEWORK

13. RS AGENT
Agent
Action Selection Policy
Value Function Agent’s knowledge about the expected payoff of actions.
Uses Action Value Estimates to select the next action
considering both exploration and exploitation goals.
Action Selection
Policy
Value
Function
Action Value
Estimates
Example #2:
Use observed
average reward as an
estimate for
arm/action value.
Example #1:
Matrix Factorization.
Example #2:
!-Greedy.
Example #1:
Greedy.
Example #2:
Simple Bandit.
Example #1:
Classic RS.
action
Estimated Expected
Reward only for
available actions at
the current state "#.
Select Next ActionUpdate Knowledge
observation/state
reward
Bridge
13

14. • Models the Agent’s problem setting. Domain or application
scenario where the Agent is deployed.
• The RS Environment interface aims to be compatible with the
OpenAI Gym toolkit1.
14
observation/state, rewardaction
!"#$%"#$&"
RS ENVIRONMENT
Environment
1 https://gym.openai.com/
ℛ(%" , &") = !"#$
Dynamics of users and
items.

15. EXPERIMENT
An Experiment measures the performance of a single RS Agent when
interacting with a specific RS Environment, in terms of pre-defined
metrics.
BEARS considers two common experiments:
BASE EXPERIMENT MONTE-CARLO EXPERIMENT
15

16. BASE EXPERIMENT
Executes a single Episode, i.e. a complete sequence
of ! agent-environment interactions.
observation
reward
actionAgent
Evaluator
Records a snapshot of
the metric values.
def base_experiment (agent, env, evaluator, T):
for _ in range(T):
action = agent.step()
observation, reward = env.step(action)
agent.observation(observation, action, reward)
evaluator.score(observation, action, reward)
return evaluator.results()
Episode
Environment
EXPERIMENT
def base_experiment (agent, env, evaluator, T):
for _ in range(T):
action = agent.step()
observation, reward = env.step(action)
agent.observation(observation, action, reward)
16

17. MONTE CARLO
EXPERIMENT
Executes E Episodes. Uses repeated sampling to consider the
stochastic properties of a RS Agent and/or RS Environment.
Episode
Episode
Episode
Episode
Episode
E
Before each episode run:
• Informs to the Evaluator that a new
episode will be executed.
• Reset the Agent and the Environment.
After an episode run has executed:
• Informs to the Evaluator that the
episode run has finished.
EXPERIMENT
Result: Summary statistics about the overall performance of the RS Agent across
the different episode runs.
17

18. 18
• Supports the collection of metric values throughout an Experiment.
• Array of Scorer components: Each Scorer measures a different Metric.
SCORER
METRIC
TIME
AGGREGATOR
EPISODE
AGGREGATOR
EVALUATOR

19. 19
1 "2 $. . . . . .
Assess the issued reward '( according to the Metric.
Every Checkpoint time steps:
Keep track of the Time Aggregated score: )*+',(-./["].
DEFAULT )*+',(234 " = ,(
CUMULATIVE )*+',(-./["] = )*+',(-./ " − 1 + ,(
AVERAGE )*+',(-./ " = )*+',(-./ " − 1 +
,( − )*+',(-./ " − 1
8 + 1
METRIC TIME AGGREGATOREVALUATOR
,( = 9,"':* +;),'<=":+8, =*":+8, ',?='@
Result: Time series of time aggregated metric values.

20. 20
• Capture performance across different episode runs.
• The Evaluator is informed of the start and end of an episode.
Aggregate collected episode sample ( !"#$%&'() ) to the Episode
Aggregated score.
!"#$%)*'+,-)+ = Incrementally
updated summary
statistics
per checkpoint.
mean
min
max
var
std
Checkpoints
EVALUATOR EPISODE AGGREGATOR
Result: Episode Aggregated time series of summary statistics related to metric values.
At the end of an Episode:

21. 21
BEARS: BANDIT-BASED RECOMMENDER SYSTEM
EVALUATION FRAMEWORK
Elements to build Experiments using BEARS:
Environment
Agent
Action Selection Policy
Value Function
Experiment
Monte Carlo Experiment
Base Experiment
Scorer
Metric
Time Aggregator
Episode Aggregator
Evaluator

22. AGENDA
Introduction
BEARS Evaluation Framework
Conclusion and Future Work

23. 23
CONCLUSION AND FUTURE WORK
MOTIVATION
• Standard RecSys evaluation methods are inappropriate to test
solutions that learn through interaction.
• Reproducing past results can be a challenge due to the lack of tools
and benchmarks.
PROPOSED SOLUTION
• BEARS is an open-source evaluation framework designed to enable the
construction of shareable experiments.
INTERESTING EXTENSIONS
• Expand built-in tools to include more baseline Agents, benchmark
Environments, common Experiments and Scorers.
• Broaden the use of BEARS to include RecSys approaches based on RL
concepts beyond the bandit approach.

24. We would like to hear your feedback.
Any comments, contributions or requests to test the
framework, please write to:
Andrea Barraza
<andrea.barraza@insight-centre.org>
VIDEO OF THE PRESENTATION: https://youtu.be/GIH_ArJ-ylk