The document discusses three questions about evaluating recommender systems: 1) How comparable are results reported using precision-oriented metrics? 2) How meaningful are absolute performance metric values? 3) How sensitive are recommenders to different evaluation methodologies? It presents comparative experiments using different metrics and evaluation designs to analyze recommenders' performance under precision-oriented evaluation.

1. Three questions about evaluation in
Recommender Systems
ACM Conference on Recommender Systems 2011 – Poster slam
October 24, Chicago, USA IRG
IR Group @ UAM

2. 1. How comparable are reported results
with precision-oriented metrics ?
ACM Conference on Recommender Systems 2011 – Poster slam
October 24, Chicago, USA IRG
IR Group @ UAM

3. 1. How comparable are reported results
with precision-oriented metrics ?
2. How meaningful are absolute
performance metric values?
ACM Conference on Recommender Systems 2011 – Poster slam
October 24, Chicago, USA IRG
IR Group @ UAM

4. 1. How comparable are reported results
with precision-oriented metrics ?
2. How meaningful are absolute
performance metric values?
3. How sensitive are the recommenders
to different evaluation methodologies?
ACM Conference on Recommender Systems 2011 – Poster slam
October 24, Chicago, USA IRG
IR Group @ UAM

5. Comparative experiments with different metrics
and alternative offline experimental designs
0.40 1.10
P@50 SVD50 RMSE
IB
0.35 UB50 1.05
0.30 1.00
0.95
0.05 0.90
0 0.85
TR 3 TR 4 TeI TrI AI OPR SVD IB UB
ACM Conference on Recommender Systems 2011 – Poster slam
October 24, Chicago, USA IRG
IR Group @ UAM

6. Precision-Oriented Evaluation of
Recommender Systems: An Algorithmic
Comparison
Alejandro Bellogín, Pablo Castells, Iván Cantador
Escuela Politécnica Superior
Universidad Autónoma de Madrid
ACM Conference on Recommender Systems 2011 – Poster slam
October 24, Chicago, USA IRG
IR Group @ UAM