Predicting performance in Recommender Systems - Poster

382 views
333 views

Published on

Poster from my Doctoral Symposium presented at ACM Recommender Systems 2011

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
382
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Predicting performance in Recommender Systems - Poster

  1. 1. Predicting Performance in Recommender Systems Alejandro Bellogín Supervised by Pablo Castells and Iván Cantador Information Retrieval Group, Universidad Autónoma de Madrid, Spain alejandro.bellogin@uam.es IRG IR Group @ UAM Motivation Research questions Is it possible to predict the accuracy of a recommendation? 1. Is it possible to define a performance prediction theory for recommender E.g., we can decide whether to deliver a recommendation or not, systems in a sound, formal way? depending on such prediction. Or, even, to combine different 2. Is it possible to adapt query performance techniques (from IR) to the recommenders according to the expected performance of each one. recommendation task? 3. What kind of evaluation should be performed? Is IR evaluation still valid in our Hypothesis problem? Data that are commonly available to a Recommender System could contain signals that enable an a priori estimation of the success of 4. What kind of recommendation problems can these models be applied to? the recommendation Research question 1 Research question 2 Is it possible to define a performance prediction theory User clarity Is it possible to adapt query performance for recommender systems in a sound, formal way? techniques (from IR) to the recommendation task? It captures the uncertainty in user’s data • In Information Retrieval: “Estimation of the system’s  p  x | u   user’s model a) Define a predictor of performance performance in response to a specific query” clarity  u    p  x | u  log   p  x   x X  c  system’s model  = (u, i, r, …) • Several predictors proposed Distance between the user’s and the b) Agree on a performance metric • We focus on query clarity  user clarity system’s probability model  = (u, i, r, …) 0.7 RatItem p_c(x) 0.6 p(x|u1) p(x|u2) We propose 3 formulations (for space X): c) Check predictive power by measuring correlation 0.5 • Based on ratings 0.4 corr([(x1), …, (xn)], [(x1), …, (xn)]) 0.3 0.2 • Based on items • Based on ratings and items 0.1 d) Evaluate final performance: dynamic vs static 0.0 4 3 5 2 1 Ratings 4 3 5 2 1Correlation coefficients r ~ 0.57 Research question 4 What kind of recommendation problems can these models be applied to?Pearson: linear correlation • Whenever a combination of strategies is availableSpearman: rank correlation • Example 1: dynamic neighbor weighting • Example 2: dynamic ensemble recommendation • The user’s neighbors are weighted • Weight is the same for every item and user according to their similarity (learnt from training) • Can we take into account the uncertainty in • What about boosting those users predicted to Research question 3 neighbor’s data? perform better for some recommender? What kind of evaluation should be performed? Is IR • User neighbor weighting: • Hybrid recommendation: evaluation still valid in our problem? • Static: g  u, i   C  sim u, v   rat  v, i  • Static: g u, i     gR1 u, i   1     gR2 u, i  vN [ u ] • In IR: Mean Average Precision + correlation • Dynamic: g  u, i   C  γ  v   sim u, v   rat  v, i  • Dynamic: g u, i   γ u   gR1 u, i   1  γ u   gR2 u, i  vN [ u ] 50 points (queries) vs 1000+ points (users) • Correlation analysis: • Correlation analysis: • Performance metric is not clear:  error-based? • Performance:  precision-based? 0,98 Standard CF 0,88 b) Neighbourhood size: 500 Standard CF b) 80% training 0,96 Clarity-enhanced CF 0,87 Clarity-enhanced CF • What is performance? 0,94 0,86 nDCG@50 0,92 0,90 0,85 • Performance: MAE 0.2 MAE  It may depend on the final application 0,84 0,88 0,83 0.15 0,86 0,84 0,82 • Possible bias 0,82 0,80 0,81 0,80 0.1 0.05  E.g., towards users or items with larger profiles 10 20 30 40 50 60 70 80 90 10 20 100 150 200 250 300 350 400 450 500 30 40 50 60 70 80 90 100 150 200 250 300 350 400 450 500 % of ratings for training Neighbourhood size 0 H1 H2 H3 H4 Adaptive Static Future Work Publications Explore other input sources We need a theoretical background • A Performance Prediction Aproach to Enhance Why do some predictors work better? Collaborative Filtering Performance. A. Bellogín  Implicit data (with time)  Social links and P. Castells. In ECIR 2010.  Item predictors o Use graph-based measures • Predicting the Performance of Recommender o Different recommender as indicators of user strength Systems: An Information Theoretic Approach. A. behavior depending on item o First results are positive Bellogín, P. Castells, and I. Cantador. In ICTIR attributes 2011. o They would allow to capture • Self-adjusting Hybrid Recommenders based on popularity, diversity, etc. Social Network Analysis. A. Bellogín, P. Castells, Larger datasets and I. Cantador. In SIGIR 2011. 5th ACM Conference on Recommender Systems (RecSys2011) – Doctoral Symposium Acknowledgments to the National Science Foundation Chicago, USA, 23-27 October 2011 for the funding to attend the conference

×