• Like
The Wisdom of the Few @SIGIR09
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

The Wisdom of the Few @SIGIR09

  • 1,376 views
Published

Presenting The Wisdom of the Few, a Collaborative Filtering approach based on Expert opinions from the Web. This presentation was done in SIGIR 2009 (July 09, Boston, MA)

Presenting The Wisdom of the Few, a Collaborative Filtering approach based on Expert opinions from the Web. This presentation was done in SIGIR 2009 (July 09, Boston, MA)

Published in Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,376
On SlideShare
0
From Embeds
0
Number of Embeds
10

Actions

Shares
Downloads
65
Comments
0
Likes
8

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. The Wisdom of the Few
      • A Collaborative Filtering Approach Based on Expert Opinions from the Web
      • Xavier Amatriain (@xamat), Josep M. Pujol, Nuria Oliver
      • Telefonica Research (Barcelona)
      • Neal Lathia
      • UCL (London)
  • 2. First, a little quiz
    • Name that Book....
    “ It is really only experts who can reliably account for their reactions”
  • 3. Crowds are not always wise
    • Collaborative filtering is the preferred approach for Recommender Systems
      • Recommendations are drawn from your past behavior and that of similar users in the system
      • Standard CF approach:
        • Find your Neighbors from the set of other users
        • Recommend things that your Neighbors liked and you have not “seen”
    • Problem: predictions are based on a large dataset that is sparse and noisy
  • 4. Overview of the Approach
    • expert = individual that we can trust to have produced thoughtful, consistent and reliable evaluations (ratings) of items in a given domain
    • Expert-based Collaborative Filtering
      • Find neighbors from a reduced set of experts instead of regular users.
        • Identify domain experts with reliable ratings
        • For each user, compute “ expert neighbors ”
        • Compute recommendations similar to standard kNN CF
  • 5. Advantages of the Approach
    • Noise
      • Experts introduce less natural noise
    • Malicious Ratings
      • Dataset can be monitored to avoid shilling
    • Data Sparsity
      • Reduced set of domain experts can be motivated to rate items
    • Cold Start problem
      • Experts rate items as soon as they are available
    • Scalability
      • Dataset is several order of magnitudes smaller
    • Privacy
      • Recommendations can be computed locally
  • 6. Take home message
    • Expert Collaborative Filtering
      • Is a new approach to recommendation but it builds up on standard CF
      • Addresses many of standard CF shortcomings
      • At least in some conditions, users prefer it over standard CF approaches
  • 7. User study
  • 8. User Study
    • 57 participants, only 14.5 ratings/participant
    • 50% of the users consider Expert-based CF to be good or very good
    • Expert-based CF: only algorithm with an average rating over 3 (on a 0-4 scale)
  • 9. User Study
    • Results to the questions: “The recommendation list includes movies I like/dislike” (1-4 Likert)
    • Experts-CF clearly outperforms other methods
  • 10. Expert Collaborative Filtering
  • 11. Expert-based CF
    • Given user u  U and d , find the set of experts E '  E such that: " e  E '  sim ( u , e ) 
    • confidence threshold t = the minimum number of expert neighbors who must have rated the item in order to trust their prediction.
      • Given an item i , find E ''  E ' s.t. " e  E '' r ei  unrated .
        • if n < t ⇒ no prediction, user mean is returned.
        • if n  ⇒ rating can be predicted: similarity-weighted average of the ratings input from each expert e in E ''
  • 12. Experts vs. Users Analysis
  • 13. Mining the Web for Expert Ratings
    • Collections of expert ratings can be obtained almost directly on the web: we crawled the Rotten Tomatoes movie critics mash-up
      • Only those (169) with more than 250 ratings in the Neflix dataset were used
  • 14. Dataset Analysis (# ratings)
    • Sparsity coefficient: 0.01 (users) vs. 0.07 (experts)
    • Average movie has 1K user ratings vs. 100 expert ratings
    • Average expert rated 400 movies, 10% rated > 1K
  • 15. Dataset Analysis ( average)
    • Users: average movie rating ~0.55 (3.2⋆);
      • 10%  0.45(2.8⋆),10%  0.7(3.8⋆)
    • Experts: average movie rating ~0.6 (3.4⋆)
      • 10%  0.4(2.6⋆), 10%  0.8 (4.2⋆)
    • user ratings centered 0.7 (3.8⋆)
    • expert ratings centered 0.6 (3.4⋆): small variability
      • only 10% of the experts have a mean score  0.55 (3.2⋆) and another 10%  0.7 (3.8⋆)
  • 16. Dataset Analysis (std)
    • Users:
      • per movie centered around 0.25 (1⋆), little variation
      • per user centered around 0.25, larger variability
    • Experts:
      • lower std per movie (0.15) and larger variation.
      • average std per expert = 0.2, small variability.
  • 17. Dataset Analysis. Summary
    • Experts...
      • are much less sparse
      • rate movies all over the rating scale instead of being biased towards rating only “good” movies (different incentives).
      • but, they seem to consistently agree on the good movies.
      • have a lower overall standard deviation per movie: they tend to agree more than regular users.
      • tend to deviate less from their personal average rating.
  • 18. Experimental Results
  • 19. Evaluation Procedure
    • Use the 169 experts to predict ratings from 10.000 users sampled from the Netflix dataset
    • Prediction MAE using a 80-20 holdout procedure (5-fold cross-validation)
    • Top-N precision by classifying items as being “recommendable” given a threshold
    • Still, take results with a grain of salt... we have a user study backing up the approach
  • 20. Results. Prediction MAE
    • Setting our parameters to  =10 and  =0.01, we obtain a MAE of 0.781 and a coverage of 97.7%
      • expert-CF yields a significant accuracy improvement with respect to using the experts’ average
      • Accuracy is worse than standard CF (with better coverage)
  • 21. Role of Thresholds
    • MAE is inversely proportional to the similarity threshold (  ) until the 0.06 mark, when it starts to increase as we move to higher  values.
      • below 0.0 it degrades rapidly: too many experts;
    • Coverage decreases as we increase  .
      • For the optimal MAE point of 0.06, coverage is still above 70%.
    • MAE as a function of the confidence threshold (  )  =0.0 and  =0.01(optimal around 9)
  • 22. Comparison to standard CF
    • Standard NN CF has MAE around 10% but coverage is also 10% lower
    • Expert-CF only works worse for the 10% of the users with lower MAE
  • 23. Results2. Top-N Precision
    • Precision of the Top-N Recommendations as a function of the “recommendable” threshold
    • For a threshold of 4, NN-CF outperforms expert-based but if we lower it to 3 they are almost equal
  • 24. Conclusions
    • Different approach to the Recommendation problem
    • At least in some conditions, users prefer recommendations from similar experts than similar users.
    • Expert-based CF has the potential to address many of standard CF shortcomings
  • 25. Future/Curent Work
    • We are currently exploring its performance in other domains and implementing a distributed expert-based CF application (work with Jae-Wook Ahn, Pittsburgh U.)
  • 26. The Wisdom of the Few
      • Thanks!
      • Questions?