Successfully reported this slideshow.
User-Centric Evaluation of a K-Furthest Neighbor    Collaborative Filtering Recommender Algorithm    Alan Said*1, Ben Fiel...
Abstract• New recommendation algorithm for diverse recommendations• Based on the k-nearest neighbor algorithm• Two types o...
Outline•   Background•   Recommendation•   K-Nearest Neighbors (knn)•   K-Furthest Neighbors (kfn)•   Evaluation & Results...
Background and AcknowledgementsStarted as a (not very serious) discussion at IJCAI & ICWSM 2011•   Ben Fields - @alsothing...
Recommendation• What is it?   o Personalized information filtering• What is the difference to search?   o Implicit   o Pas...
Recommendation - An example (knn)Recommending a movie to Bert:                                                     Cookie ...
Recommendation - An example (knn)  Recommending a movie to Bert:                                               Similar to ...
Recommendation - A counter example       What happens if we flip it?                              Can we recommend movies ...
Recommendation - A counter example (kfn)      Recommending a movie to Bert:1. Who is dissimilar to Bert?                  ...
Recommendation - A counter example (kfn)      Recommending a movie to Bert:1. Who is dissimilar to Bert?                  ...
EvaluationWhat are the effects of this?                        Diversity :                        •     Less popular items...
Evaluation - Recommendation AccuracyTraditional - Offline Evaluation • Movielens 10M, 70k users • Precision@N for users wi...
Evaluation•Are we missing something?     Yestraintest                       February 27th, 2013   12
Evaluation - Online User Study                       February 27th, 2013   13
Evaluation - Online User Study10 recommended     movies                                               7 questions         ...
Evaluation – Recommendation UtilityData                               Questions • 132 users                        • Novel...
Evaluation – Recommendation Utility                  Do you know the movie?                      February 27th, 2013   15
Evaluation – Recommendation Utility                     Have you seen it?                      February 27th, 2013   15
Evaluation – Recommendation Utility                                   Would you watch it?                     February 27t...
Evaluation – Recommendation Utility                                                                                       ...
ConclusionRecommending what your anti-peers do not like creates: • more diverse recommendations, • with comparable overall...
Questions?         Thank you for listening!         For more RecSys stuff, check out:         www.recsyswiki.com          ...
Upcoming SlideShare
Loading in …5
×

User-Centric Evaluation of a K-Furthest Neighbor Collaborative Filtering Recommender Algorithm

3,216 views

Published on

Collaborative filtering recommender systems often use nearest neighbor methods to identify candidate items. In this paper we present an inverted neighborhood model, k-Furthest Neighbors, to identify less ordinary neighborhoods for the purpose of creating more diverse recommendations. The approach is evaluated two-fold, once in a traditional information retrieval evaluation setting where the model is trained and validated on a split train/test set, and once through an online user study (N=132) to identify users’ erceived quality of the recommender. A standard k-nearest neighbor recommender is used as a baseline in both evaluation settings. our evaluation shows that even though the proposed furthest neighbor model is outperformed in the traditional evaluation setting, the perceived usefulness of the algorithm shows no significant difference in the results of the user study.

Published in: Technology
  • Be the first to comment

User-Centric Evaluation of a K-Furthest Neighbor Collaborative Filtering Recommender Algorithm

  1. 1. User-Centric Evaluation of a K-Furthest Neighbor Collaborative Filtering Recommender Algorithm Alan Said*1, Ben Fields#2, Brijnesh J. Jain*, Sahin Albayrak** Technische Universität Berlin# musicmetric 1 @alansaid 2 CSCW 2013, San Antonio, TX, USA @alsothings February 27th, 2013
  2. 2. Abstract• New recommendation algorithm for diverse recommendations• Based on the k-nearest neighbor algorithm• Two types of evaluation o standard offline evaluation o user-centric online evaluation• Proposed algorithm performs worse than baseline in offline evaluation but has higher perceived usefulness from the users in online evaluation February 27th, 2013 2
  3. 3. Outline• Background• Recommendation• K-Nearest Neighbors (knn)• K-Furthest Neighbors (kfn)• Evaluation & Results• Conclusions February 27th, 2013 3
  4. 4. Background and AcknowledgementsStarted as a (not very serious) discussion at IJCAI & ICWSM 2011• Ben Fields - @alsothings• Òscar Celma - @ocelma• Markus Schedl - @m_schedl• Mohamed Sordo - @neomoha February 27th, 2013 4
  5. 5. Recommendation• What is it? o Personalized information filtering• What is the difference to search? o Implicit o Passively finds most interesting items• How? February 27th, 2013 5
  6. 6. Recommendation - An example (knn)Recommending a movie to Bert: Cookie Herry what/who Bert Ernie Big Bird Elmo Monster Monster Toy Story 4 4 5 1 4 E.T. 2 5 2 Beetlejuice 4 4 5 2 3 Shrek 1 3 1 Zoolander 4 1 February 27th, 2013 6
  7. 7. Recommendation - An example (knn) Recommending a movie to Bert: Similar to Bert Cookie Herry what/who Bert Ernie Big Bird Elmo Monster Monster Toy Story 4 4 5 1 K-Nearest Neighbor 4 poor E.T. 2 5 2 rating Beetlejuice 4 4 5 Potential movies 2 3 to recommend Shrek 1 3 1recommendation Zoolander 4 1 February 27th, 2013 6
  8. 8. Recommendation - A counter example What happens if we flip it? Can we recommend movies disliked by those who are dissimilar to Bert? Yes! February 27th, 2013 7
  9. 9. Recommendation - A counter example (kfn) Recommending a movie to Bert:1. Who is dissimilar to Bert? Cookie Herry what/who Bert Ernie Big Bird Elmo Monster Monster Toy Story 4 4 5 1 4 E.T. 2 5 2 Beetlejuice 4 4 5 2 2 3 Shrek 1 3 1 Zoolander 4 February 27th, 2013 8
  10. 10. Recommendation - A counter example (kfn) Recommending a movie to Bert:1. Who is dissimilar to Bert? Cookie what/who Bert Monster2. What do they dislike? Toy Story 4 1 K-Furthest Neighbor E.T. Beetlejuice 4 2 Disliked by Cookie Monster - Shrek 1 Liked by Bert? Zoolander February 27th, 2013 9
  11. 11. EvaluationWhat are the effects of this? Diversity : • Less popular items • Items the users are not familiar with • Non standard items February 27th, 2013 10
  12. 12. Evaluation - Recommendation AccuracyTraditional - Offline Evaluation • Movielens 10M, 70k users • Precision@N for users with >2N ratings • Furthest performs at ~60% of Nearest neighbor (for N=100) <0.001However • lists of recommended items are practically disjoint February 27th, 2013 11
  13. 13. Evaluation•Are we missing something? Yestraintest February 27th, 2013 12
  14. 14. Evaluation - Online User Study February 27th, 2013 13
  15. 15. Evaluation - Online User Study10 recommended movies 7 questions February 27th, 2013 13
  16. 16. Evaluation – Recommendation UtilityData Questions • 132 users • Novel? • 10 recommended • Obvious? movies each • Recognizable? • Serendipitous? • knn: 47 users • Useful? • kfn: 43 users • Best movie? • random: 42 users • Worst movie? • training set: Movielens • Rate each seen movie 10M • State whether movie is familiar • State whether you would see it February 27th, 2013 14
  17. 17. Evaluation – Recommendation Utility Do you know the movie? February 27th, 2013 15
  18. 18. Evaluation – Recommendation Utility Have you seen it? February 27th, 2013 15
  19. 19. Evaluation – Recommendation Utility Would you watch it? February 27th, 2013 15
  20. 20. Evaluation – Recommendation Utility Likert scale 1: least agree; 5: most agree rating novelty obviousness recognizable serendipity usefulness knn 3.64 3.83 2.27 2.69 2.71 2.69 kfn 3.65 3.95 1.79 2.07 2.65 2.63 random 3.07 4.17 1.64 1.81 2.48 2.24 highest rating less obvious/recognizable comparable serendipity and usefulness remember: knn and kfn recommend different items, still the experienced quality is similar (or higher) February 27th, 2013 16
  21. 21. ConclusionRecommending what your anti-peers do not like creates: • more diverse recommendations, • with comparable overall usefulness, • even though standard offline evaluation says otherwise February 27th, 2013 17
  22. 22. Questions? Thank you for listening! For more RecSys stuff, check out: www.recsyswiki.com February 27th, 2013 18

×