This document discusses predicting the performance of recommender systems. It proposes that data commonly available to recommender systems could enable estimating the success of recommendations. Specifically, it aims to 1) define a performance prediction theory for recommender systems, 2) adapt query performance techniques from information retrieval to recommendations, and 3) evaluate appropriate performance metrics. The research also explores applying these models when combining multiple recommendation strategies or hybrid recommender systems.

1. Predicting Performance in Recommender Systems
Alejandro Bellogín
Supervised by Pablo Castells and Iván Cantador
Information Retrieval Group, Universidad Autónoma de Madrid, Spain
alejandro.bellogin@uam.es
IRG
IR Group @ UAM
Motivation Research questions
Is it possible to predict the accuracy of a recommendation? 1. Is it possible to define a performance prediction theory for recommender
E.g., we can decide whether to deliver a recommendation or not, systems in a sound, formal way?
depending on such prediction. Or, even, to combine different
2. Is it possible to adapt query performance techniques (from IR) to the
recommenders according to the expected performance of each one.
recommendation task?
3. What kind of evaluation should be performed? Is IR evaluation still valid in our
Hypothesis problem?
Data that are commonly available to a Recommender System could
contain signals that enable an a priori estimation of the success of
4. What kind of recommendation problems can these models be applied to?
the recommendation
Research question 1 Research question 2
Is it possible to define a performance prediction theory User clarity
Is it possible to adapt query performance
for recommender systems in a sound, formal way? techniques (from IR) to the recommendation task? It captures the uncertainty in user’s data
• In Information Retrieval: “Estimation of the system’s
 p  x | u   user’s model
a) Define a predictor of performance performance in response to a specific query” clarity  u    p  x | u  log 
 p  x  
x X  c  system’s model
 = (u, i, r, …) • Several predictors proposed Distance between the user’s and the
b) Agree on a performance metric • We focus on query clarity  user clarity system’s probability model
 = (u, i, r, …)
0.7 RatItem p_c(x)
0.6 p(x|u1)
p(x|u2)
We propose 3 formulations (for space X):
c) Check predictive power by measuring correlation
0.5
• Based on ratings
0.4
corr([(x1), …, (xn)], [(x1), …, (xn)])
0.3
0.2
• Based on items
• Based on ratings and items
0.1
d) Evaluate final performance: dynamic vs static 0.0
4 3 5 2 1 Ratings 4 3 5 2 1
Correlation coefficients
r ~ 0.57
Research question 4
What kind of recommendation problems can these models be applied to?
Pearson: linear correlation • Whenever a combination of strategies is available
Spearman: rank correlation
• Example 1: dynamic neighbor weighting • Example 2: dynamic ensemble recommendation
• The user’s neighbors are weighted • Weight is the same for every item and user
according to their similarity (learnt from training)
• Can we take into account the uncertainty in • What about boosting those users predicted to
Research question 3 neighbor’s data? perform better for some recommender?
What kind of evaluation should be performed? Is IR • User neighbor weighting: • Hybrid recommendation:
evaluation still valid in our problem? • Static: g  u, i   C  sim u, v   rat  v, i  • Static: g u, i     gR1 u, i   1     gR2 u, i 
vN [ u ]
• In IR: Mean Average Precision + correlation • Dynamic: g  u, i   C  γ  v   sim u, v   rat  v, i  • Dynamic: g u, i   γ u   gR1 u, i   1  γ u   gR2 u, i 
vN [ u ]
50 points (queries) vs 1000+ points (users) • Correlation analysis: • Correlation analysis:
• Performance metric is not clear:
 error-based?
• Performance:
 precision-based? 0,98 Standard CF 0,88 b) Neighbourhood size: 500
Standard CF b) 80% training
0,96 Clarity-enhanced CF 0,87 Clarity-enhanced CF
• What is performance? 0,94 0,86
nDCG@50
0,92
0,90
0,85
• Performance:
MAE
0.2
MAE
 It may depend on the final application
0,84
0,88
0,83 0.15
0,86
0,84 0,82
• Possible bias 0,82
0,80
0,81
0,80
0.1
0.05
 E.g., towards users or items with larger profiles
10 20 30 40 50 60 70 80 90 10 20 100 150 200 250 300 350 400 450 500
30 40 50 60 70 80 90 100 150 200 250 300 350 400 450 500
% of ratings for training Neighbourhood size 0
H1 H2 H3 H4
Adaptive Static
Future Work Publications
Explore other input sources We need a theoretical background • A Performance Prediction Aproach to Enhance
Why do some predictors work better? Collaborative Filtering Performance. A. Bellogín
 Implicit data (with time)  Social links
and P. Castells. In ECIR 2010.
 Item predictors o Use graph-based measures
• Predicting the Performance of Recommender
o Different recommender as indicators of user strength
Systems: An Information Theoretic Approach. A.
behavior depending on item o First results are positive Bellogín, P. Castells, and I. Cantador. In ICTIR
attributes 2011.
o They would allow to capture • Self-adjusting Hybrid Recommenders based on
popularity, diversity, etc. Social Network Analysis. A. Bellogín, P. Castells,
Larger datasets
and I. Cantador. In SIGIR 2011.
5th ACM Conference on Recommender Systems (RecSys2011) – Doctoral Symposium Acknowledgments to the National Science Foundation
Chicago, USA, 23-27 October 2011 for the funding to attend the conference