Rate it Again
Increasing Recommendation Accuracy by
User re­Rating
Xavier Amatriain (with J.M. Pujol, N. Tintarev, N. Oliver)
Telefonica Research
Recsys 09
X. Amatriain et. al
Rate It Again

The Recommender Problem
● Two ways to address it
1. Improve the Algorithm
X. Amatriain et. al
Rate It Again

The Recommender Problem
● Two ways to address it
2. Improve the Input Data
Time for Data
Cleaning!
X. Amatriain et. al
Rate It Again

User Feedback is Noisy
● See our UMAP '09 Publication:
“I like it... I like it not” (Amatriain et al. '09)
X. Amatriain et. al
Rate It Again

Natural Noise Limits our User Model
DID YOU HEAR WHAT
I LIKE??!!
...and Our Prediction Accuracy
X. Amatriain et. al
Rate It Again

Experimental setup
● 118 participants rated movies in 3 trials
T1 (rand) <­> 24 h <­>T2 (pop.) <­> 15 days <­>T3 (rand)
● 100 Movies from Netflix dataset, stratified
random sampling on popularity
● Ratings on a 1 to 5 star scale with special “not
seen” symbol.
X. Amatriain et. al
Rate It Again

Users are Inconsistent
● What is the probability of making an inconsistency
given an original rating
X. Amatriain et. al
Rate It Again

Users are Inconsistent
Mild ratings are
noisier
● What is the percentage of inconsistencies given an
original rating
X. Amatriain et. al
Rate It Again

Users are Inconsistent
Negative
ratings are
noisier
● What is the percentage of inconsistencies given an
original rating
X. Amatriain et. al
Rate It Again

Prediction Accuracy
#Ti #Tj # RMSE
   
T1, T2 2185 1961 1838 2308 0.573 0.707
T1, T3 2185 1909 1774 2320 0.637 0.765
T2, T3 1969 1909 1730 2140 0.557 0.694
● Pairwise RMSE between trials considering
intersection and union of both sets
X. Amatriain et. al
Rate It Again

Max error in
Prediction Accuracy
trials that are
#Ti #Tj # RMSE
most distant in
time
   
T1, T2 2185 1961 1838 2308 0.573 0.707
T1, T3 2185 1909 1774 2320 0.637 0.765
T2, T3 1969 1909 1730 2140 0.557 0.694
● Pairwise RMSE between trials considering
intersection and union of both sets
X. Amatriain et. al
Rate It Again

Significant less
Prediction Accuracy
error when 2nd #Ti #Tj # RMSE
trial is involved
   
T1, T2 2185 1961 1838 2308 0.573 0.707
T1, T3 2185 1909 1774 2320 0.637 0.765
T2, T3 1969 1909 1730 2140 0.557 0.694
● Pairwise RMSE between trials considering
intersection and union of both sets
X. Amatriain et. al
Rate It Again

Algorithm Robustness to NN
Alg./Trial T1 T2 T3 Tworst /Tbest
User 1.2011 1.1469 1.1945 4.7%
Average
Item 1.0555 1.0361 1.0776 4%
Average
User­based 0.9990 0.9640 1.0171 5.5%
kNN
Item­based 1.0429 1.0031 1.0417 4%
kNN
SVD 1.0244 0.9861 1.0285 4.3%
RMSE for different Recommendation algorithms
●
when predicting each of the trials
X. Amatriain et. al
Rate It Again

Algorithm Robustness to NN
Trial 2 is
consistently the
Alg./Trial T1 T2 T3 Tworst /Tbest
least noisy
User 1.2011 1.1469 1.1945 4.7%
Average
Item 1.0555 1.0361 1.0776 4%
Average
User­based 0.9990 0.9640 1.0171 5.5%
kNN
Item­based 1.0429 1.0031 1.0417 4%
kNN
SVD 1.0244 0.9861 1.0285 4.3%
RMSE for different Recommendation algorithms
●
when predicting each of the trials
X. Amatriain et. al
Rate It Again

Algorithm Robustness to NN (2)
Training­Testing T1-T2 T1-T3 T2-T3
Dataset
User Average 1.1585 1.2095 1.2036
Movie Average 1.0305 1.0648 1.0637
User­based kNN 0.9693 1.0143 1.0184
Item­based kNN 1.0009 1.0406 1.0590
SVD 0.9741 1.0491 1.0118
● RMSE for different Recommendation algorithms
when predicting ratings in one trial (testing) from
ratings on another (training)
X. Amatriain et. al
Rate It Again

Algorithm Robustness to NN (2)
Training­Testing T1-T2 T1-T3 T2-T3
Dataset
User Average 1.1585 1.2095 1.2036
Movie Average 1.0305 1.0648 1.0637
User­based kNN 0.9693 1.0143 1.0184
Item­based kNN 1.0009 1.0406 1.0590
SVD
Noise is minimized 0.9741 1.0491 1.0118
when we predict
Trial 2
● RMSE for different Recommendation algorithms
when predicting ratings in one trial (testing) from
ratings on another (training)
X. Amatriain et. al
Rate It Again

Let's recap
● Users are inconsistent
● Inconsistencies can depend on many things
including how the items are presented
● Inconsistencies produce natural noise
● Natural noise reduces our prediction accuracy
independently of the algorithm
X. Amatriain et. al
Rate It Again

Hypothesis
● If we can somehow reduce natural noise due to
user inconsistencies we could greatly
improve recommendation accuracy.
● We can reduce natural noise by taking
advantage of user inconsistencies when re­
rating items.
X. Amatriain et. al
Rate It Again

Algorithm
● Given a rating dataset where (some) items
have been re­rated,
● Two fairness conditions:
1. Algorithm should remove as few ratings as
possible (i.e. only when there is some
certainty that the rating is only adding noise)
2.Algorithm should not make up new ratings but
decide on which of the existing ones are
valid.
X. Amatriain et. al
Rate It Again

Algorithm
● One source re­rating case:
● Given the following milding function:
X. Amatriain et. al
Rate It Again

Results
● One­source re­rating (Denoised⊚Denoising)
T1⊚T2 ΔT1 T1⊚T3 ΔT1 T2⊚T3 ΔT2
User­based kNN 0.8861 11.3% 0.8960 10.3% 0.8984 6.8%
SVD 0.9121 11.0% 0.9274 9.5% 0.9159 7.1%
● Two­source re­rating (Denoising T1with the other 2)
Datasets T1⊚(T2, T3) ΔT1
User­based kNN 0.8647 13.4%
SVD 0.8800 14.1%
X. Amatriain et. al
Rate It Again

Results Best results (above 10%!)
when denoising noisy trial
with less noisy
● One­source re­rating (Denoised⊚Denoising)
T1⊚T2 ΔT1 T1⊚T3 ΔT1 T2⊚T3 ΔT2
User­based kNN 0.8861 11.3% 0.8960 10.3% 0.8984 6.8%
SVD 0.9121 11.0% 0.9274 9.5% 0.9159 7.1%
● Two­source re­rating (Denoising T1with the other 2)
Datasets T1⊚(T2, T3) ΔT1
User­based kNN 0.8647 13.4%
SVD 0.8800 14.1%
X. Amatriain et. al
Rate It Again

Results Smaller (yet important)
improvement when
denoising less noisy set
● One­way re­rating (Denoised⊚Denoising)
T1⊚T2 ΔT1 T1⊚T3 ΔT1 T2⊚T3 ΔT2
User­based kNN 0.8861 11.3% 0.8960 10.3% 0.8984 6.8%
SVD 0.9121 11.0% 0.9274 9.5% 0.9159 7.1%
● Two­way re­rating (Denoising T1with the other 2)
Datasets T1⊚(T2, T3) ΔT1
User­based kNN 0.8647 13.4%
SVD 0.8800 14.1%
X. Amatriain et. al
Rate It Again

Results
● One­way re­rating (Denoised⊚Denoising)
T1⊚T2 ΔT1 T1⊚T3 ΔT1 T2⊚T3 ΔT2
User­based kNN 0.8861 11.3% 0.8960 10.3% 0.8984 6.8%
SVD 0.9121 11.0% 0.9274 9.5% 0.9159 7.1%
● Two­way re­rating (Denoising T1with the other 2)
Datasets T1⊚(T2, T3) ΔT1 Improvements
up to 14% with
User­based kNN 0.8647 13.4% 2 re­ratings!
SVD 0.8800 14.1%
X. Amatriain et. al
Rate It Again

But...
● We can't expect all users to re­rate all items
once or twice to improve accuracy!
● Need to devise methods to selectively choose
which ratings to denoise:
– Random selection
– Data­dependent (select ratings based on values)
– User­dependent (select ratings based on how
“noisy” user is)
X. Amatriain et. al
Rate It Again

Random re­rating
● Improvement in RMSE when doing once­source (left) and
two­source (right) re­rating as a function of the percentage
of randomly­selected denoised ratings (T1⊚T3 )
X. Amatriain et. al
Rate It Again

Random re­rating
● Improvement in RMSE when doing once­source (left) and
two­source (right) re­rating as a function of the percentage
of randomly­selected denoised ratings (T1⊚T3 )
X. Amatriain et. al
Rate It Again

Denoise Extreme Ratings
● Improvement in RMSE when doing once­source (left)
and two­source (right) re­rating as a function of the
percentage of denoised ratings: selecting only extreme
X. Amatriain et. al
Rate It Again

Denoise Extreme Ratings
● Improvement in RMSE when doing once­source (left)
and two­source (right) re­rating as a function of the
percentage of denoised ratings: selecting only extreme
X. Amatriain et. al
Rate It Again

Denoise outliers
● Improvement in RMSE when doing once­source (left) and two­
source (right) re­rating as a function of the percentage of denoised
ratings and users: selecting only noisy users and extreme ratings
X. Amatriain et. al
Rate It Again

Denoise outliers
● Improvement in RMSE when doing once­source (left) and two­
source (right) re­rating as a function of the percentage of denoised
ratings and users: selecting only noisy users and extreme ratings
X. Amatriain et. al
Rate It Again

Value of Rating
An extreme re­
rating improves
RMSE 10 times
more than adding a
new rating!
● Is it worth to add new ratings or re­rate existing items?
RMSE improvement as a function of new ratings added
in each case.
X. Amatriain et. al
Rate It Again

Conclusions
● Improving data can be more beneficial than
improving the algorithm
● Natural noise limits the accuracy of Recommender
Systems
● We can reduce natural noise by asking users to re­rate
items
● There are strategies to minimize the impact of the re­
rating process
● The value of a re­rate may be higher than that of a
new rating
X. Amatriain et. al
Rate It Again

Rate it Again
Increasing Recommendation Accuracy by
User re­Rating
Thanks!
X. Amatriain et. al
Rate It Again