The Wisdom of the Few A Collaborative Filtering Approach Based on Expert Opinions from the Web Xavier Amatriain (@xamat), Josep M. Pujol, Nuria Oliver Telefonica Research (Barcelona) Neal Lathia UCL (London)

A Collaborative Filtering Approach Based on Expert Opinions from the Web

Xavier Amatriain (@xamat), Josep M. Pujol, Nuria Oliver

Telefonica Research (Barcelona)

Neal Lathia

UCL (London)

First, a little quiz Name that Book.... “ It is really only experts who can reliably account for their reactions”

Name that Book....

Crowds are not always wise Collaborative filtering is the preferred approach for Recommender Systems Recommendations are drawn from your past behavior and that of similar users in the system Standard CF approach: Find your Neighbors from the set of other users Recommend things that your Neighbors liked and you have not “seen” Problem: predictions are based on a large dataset that is sparse and noisy

Collaborative filtering is the preferred approach for Recommender Systems

Recommendations are drawn from your past behavior and that of similar users in the system

Standard CF approach:

Find your Neighbors from the set of other users

Recommend things that your Neighbors liked and you have not “seen”

Problem: predictions are based on a large dataset that is sparse and noisy

Overview of the Approach expert = individual that we can trust to have produced thoughtful, consistent and reliable evaluations (ratings) of items in a given domain Expert-based Collaborative Filtering Find neighbors from a reduced set of experts instead of regular users. Identify domain experts with reliable ratings For each user, compute “ expert neighbors ” Compute recommendations similar to standard kNN CF

expert = individual that we can trust to have produced thoughtful, consistent and reliable evaluations (ratings) of items in a given domain

Expert-based Collaborative Filtering

Find neighbors from a reduced set of experts instead of regular users.

Identify domain experts with reliable ratings

For each user, compute “ expert neighbors ”

Compute recommendations similar to standard kNN CF

Advantages of the Approach Noise Experts introduce less natural noise Malicious Ratings Dataset can be monitored to avoid shilling Data Sparsity Reduced set of domain experts can be motivated to rate items Cold Start problem Experts rate items as soon as they are available Scalability Dataset is several order of magnitudes smaller Privacy Recommendations can be computed locally

Noise

Experts introduce less natural noise

Malicious Ratings

Dataset can be monitored to avoid shilling

Data Sparsity

Reduced set of domain experts can be motivated to rate items

Cold Start problem

Experts rate items as soon as they are available

Scalability

Dataset is several order of magnitudes smaller

Privacy

Recommendations can be computed locally

Take home message Expert Collaborative Filtering Is a new approach to recommendation but it builds up on standard CF Addresses many of standard CF shortcomings At least in some conditions, users prefer it over standard CF approaches

Expert Collaborative Filtering

Is a new approach to recommendation but it builds up on standard CF

Addresses many of standard CF shortcomings

At least in some conditions, users prefer it over standard CF approaches

User study

User Study 57 participants, only 14.5 ratings/participant 50% of the users consider Expert-based CF to be good or very good Expert-based CF: only algorithm with an average rating over 3 (on a 0-4 scale)

57 participants, only 14.5 ratings/participant

50% of the users consider Expert-based CF to be good or very good

Expert-based CF: only algorithm with an average rating over 3 (on a 0-4 scale)

User Study Results to the questions: “The recommendation list includes movies I like/dislike” (1-4 Likert) Experts-CF clearly outperforms other methods

Results to the questions: “The recommendation list includes movies I like/dislike” (1-4 Likert)

Experts-CF clearly outperforms other methods

Expert Collaborative Filtering

Expert-based CF Given user u  U and d , find the set of experts E '  E such that: &quot; e  E '  sim ( u , e )  confidence threshold t = the minimum number of expert neighbors who must have rated the item in order to trust their prediction. Given an item i , find E ''  E ' s.t. &quot; e  E '' r ei  unrated . if n < t ⇒ no prediction, user mean is returned. if n  ⇒ rating can be predicted: similarity-weighted average of the ratings input from each expert e in E ''

Given user u  U and d , find the set of experts E '  E such that: &quot; e  E '  sim ( u , e ) 

confidence threshold t = the minimum number of expert neighbors who must have rated the item in order to trust their prediction.

Given an item i , find E ''  E ' s.t. &quot; e  E '' r ei  unrated .

if n < t ⇒ no prediction, user mean is returned.

if n  ⇒ rating can be predicted: similarity-weighted average of the ratings input from each expert e in E ''

Experts vs. Users Analysis

Mining the Web for Expert Ratings Collections of expert ratings can be obtained almost directly on the web: we crawled the Rotten Tomatoes movie critics mash-up Only those (169) with more than 250 ratings in the Neflix dataset were used

Collections of expert ratings can be obtained almost directly on the web: we crawled the Rotten Tomatoes movie critics mash-up

Only those (169) with more than 250 ratings in the Neflix dataset were used

Dataset Analysis (# ratings) Sparsity coefficient: 0.01 (users) vs. 0.07 (experts) Average movie has 1K user ratings vs. 100 expert ratings Average expert rated 400 movies, 10% rated > 1K

Sparsity coefficient: 0.01 (users) vs. 0.07 (experts)

Average movie has 1K user ratings vs. 100 expert ratings

Average expert rated 400 movies, 10% rated > 1K

Dataset Analysis ( average) Users: average movie rating ~0.55 (3.2⋆); 10%  0.45(2.8⋆),10%  0.7(3.8⋆) Experts: average movie rating ~0.6 (3.4⋆) 10%  0.4(2.6⋆), 10%  0.8 (4.2⋆) user ratings centered 0.7 (3.8⋆) expert ratings centered 0.6 (3.4⋆): small variability only 10% of the experts have a mean score  0.55 (3.2⋆) and another 10%  0.7 (3.8⋆)

Users: average movie rating ~0.55 (3.2⋆);

10%  0.45(2.8⋆),10%  0.7(3.8⋆)

Experts: average movie rating ~0.6 (3.4⋆)

10%  0.4(2.6⋆), 10%  0.8 (4.2⋆)

user ratings centered 0.7 (3.8⋆)

expert ratings centered 0.6 (3.4⋆): small variability

only 10% of the experts have a mean score  0.55 (3.2⋆) and another 10%  0.7 (3.8⋆)

Dataset Analysis (std) Users: per movie centered around 0.25 (1⋆), little variation per user centered around 0.25, larger variability Experts: lower std per movie (0.15) and larger variation. average std per expert = 0.2, small variability.

Users:

per movie centered around 0.25 (1⋆), little variation

per user centered around 0.25, larger variability

Experts:

lower std per movie (0.15) and larger variation.

average std per expert = 0.2, small variability.

Dataset Analysis. Summary Experts... are much less sparse rate movies all over the rating scale instead of being biased towards rating only “good” movies (different incentives). but, they seem to consistently agree on the good movies. have a lower overall standard deviation per movie: they tend to agree more than regular users. tend to deviate less from their personal average rating.

Experts...

are much less sparse

rate movies all over the rating scale instead of being biased towards rating only “good” movies (different incentives).

but, they seem to consistently agree on the good movies.

have a lower overall standard deviation per movie: they tend to agree more than regular users.

tend to deviate less from their personal average rating.

Experimental Results

Evaluation Procedure Use the 169 experts to predict ratings from 10.000 users sampled from the Netflix dataset Prediction MAE using a 80-20 holdout procedure (5-fold cross-validation) Top-N precision by classifying items as being “recommendable” given a threshold Still, take results with a grain of salt... we have a user study backing up the approach

Use the 169 experts to predict ratings from 10.000 users sampled from the Netflix dataset

Prediction MAE using a 80-20 holdout procedure (5-fold cross-validation)

Top-N precision by classifying items as being “recommendable” given a threshold

Still, take results with a grain of salt... we have a user study backing up the approach

Results. Prediction MAE Setting our parameters to  =10 and  =0.01, we obtain a MAE of 0.781 and a coverage of 97.7% expert-CF yields a significant accuracy improvement with respect to using the experts’ average Accuracy is worse than standard CF (with better coverage)

Setting our parameters to  =10 and  =0.01, we obtain a MAE of 0.781 and a coverage of 97.7%

expert-CF yields a significant accuracy improvement with respect to using the experts’ average

Accuracy is worse than standard CF (with better coverage)

Role of Thresholds MAE is inversely proportional to the similarity threshold (  ) until the 0.06 mark, when it starts to increase as we move to higher  values. below 0.0 it degrades rapidly: too many experts; Coverage decreases as we increase  . For the optimal MAE point of 0.06, coverage is still above 70%. MAE as a function of the confidence threshold (  )  =0.0 and  =0.01(optimal around 9)

MAE is inversely proportional to the similarity threshold (  ) until the 0.06 mark, when it starts to increase as we move to higher  values.

below 0.0 it degrades rapidly: too many experts;

Coverage decreases as we increase  .

For the optimal MAE point of 0.06, coverage is still above 70%.

MAE as a function of the confidence threshold (  )  =0.0 and  =0.01(optimal around 9)

Comparison to standard CF Standard NN CF has MAE around 10% but coverage is also 10% lower Expert-CF only works worse for the 10% of the users with lower MAE

Standard NN CF has MAE around 10% but coverage is also 10% lower

Expert-CF only works worse for the 10% of the users with lower MAE

Results2. Top-N Precision Precision of the Top-N Recommendations as a function of the “recommendable” threshold For a threshold of 4, NN-CF outperforms expert-based but if we lower it to 3 they are almost equal

Precision of the Top-N Recommendations as a function of the “recommendable” threshold

For a threshold of 4, NN-CF outperforms expert-based but if we lower it to 3 they are almost equal

Conclusions Different approach to the Recommendation problem At least in some conditions, users prefer recommendations from similar experts than similar users. Expert-based CF has the potential to address many of standard CF shortcomings

Different approach to the Recommendation problem

At least in some conditions, users prefer recommendations from similar experts than similar users.

Expert-based CF has the potential to address many of standard CF shortcomings

Future/Curent Work We are currently exploring its performance in other domains and implementing a distributed expert-based CF application (work with Jae-Wook Ahn, Pittsburgh U.)

We are currently exploring its performance in other domains and implementing a distributed expert-based CF application (work with Jae-Wook Ahn, Pittsburgh U.)

The Wisdom of the Few Thanks! Questions?

Thanks!

Questions?