The Magic Barrier of Recommender Systems - No Magic, Just Ratings
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

The Magic Barrier of Recommender Systems - No Magic, Just Ratings

on

  • 782 views

Recommender Systems need to deal with different types of users who represent their preferences in various ways. This difference in user behaviour has a deep impact on the final performance of the ...

Recommender Systems need to deal with different types of users who represent their preferences in various ways. This difference in user behaviour has a deep impact on the final performance of the recommender system, where some users may receive either better or worse recommendations depending, mostly, on the quantity and the quality of the information the system knows about the user. Specifically, the inconsistencies of the user impose a lower bound on the error the system may achieve when predicting ratings for that particular user.

In this work, we analyse how the consistency of user ratings (coherence) may predict the performance of recommendation methods. More specifically, our results show that our definition of coherence is correlated with the so-called magic barrier of recommender systems, and thus, it could be used to discriminate between easy users (those with a low magic barrier) and difficult ones (those with a high magic barrier).
We report experiments where the rating prediction error for the more coherent users is lower than that of the less coherent ones.
We further validate these results by using a public dataset, where the magic barrier is not available, in which we obtain similar performance improvements.

Statistics

Views

Total Views
782
Views on SlideShare
741
Embed Views
41

Actions

Likes
8
Downloads
19
Comments
0

3 Embeds 41

https://twitter.com 38
https://www.linkedin.com 2
https://www.rebelmouse.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

The Magic Barrier of Recommender Systems - No Magic, Just Ratings Presentation Transcript

  • 1. The  Magic  Barrier  of     Recommender  Systems:   No  Magic,  Just  ra;ngs   Alejandro  Bellogín,  Alan  Said,  Arjen  de  Vries   @abellogin,  @alansaid,  @arjenpdevries     UMAP  2014   2014-­‐07-­‐08  
  • 2. What  is  the  magic  barrier?   2  
  • 3. The  Magic  Barrier   The  upper  bound  on  recommenda;on   accuracy     Recommenda;on  accuracy  is  only  as   good  as  the  quality  of  the  underlying   data   Users  are  noisy  /  incoherent  in  their   ra;ng  behavior     3  
  • 4. How  to  find  the  magic   barrier?   1.  Collect  infinite  re-­‐ra;ngs  for  each   item  from  your  users.   2.  Find  standard  devia;on  of  ra;ng/ re-­‐ra;ng  inconsistencies  (noise)   [Said  et  al.  UMAP  ‘12]   Realis;cally,  an  infinite  amount  of   re-­‐ra;ngs  is  not  probable   4  
  • 5. Coherence   5  
  • 6. What  is  coherence?   Coherent  user   e slightly different. Specifically, the user in Figure 1a gives more consisten items sharing the same features, which in this case corresponds to the mov s. In the long term we argue that such a user will have a more consistent (le ehaviour, since her taste for each item feature seem to be well defined. Action Comedy Horror (a) A coherent user (c(u) = 1.28) Action Comedy Horror (b) A not coherent user (c(u) = Incoherent  user   htly different. Specifically, the user in Figure 1a gives more consistent rating sharing the same features, which in this case corresponds to the movie’s gen he long term we argue that such a user will have a more consistent (less noisy ur, since her taste for each item feature seem to be well defined. Action Comedy Horror A coherent user (c(u) = 1.28) Action Comedy Horror (b) A not coherent user (c(u) = 5.95) 6  
  • 7. DefiniAon  of  Coherence   Ra;ng  consistency  within  a  feature   space  (genres,  themes,  etc.)     c u( )= − σ f f ∈F ∑ u( ) σ f u( )= r u,i( )− rf u( )( ) 2 i∈I u, f( ) ∑ Item  features   User’s  ra;ng  devia;on  within  a  feature   7  
  • 8. Ra;ng  devia;on  within  a  feature  space   instead  of   infinite  re-­‐ra;ngs   Is  this  a  good   es;mate  of  the   magic  barrier?   8  
  • 9. Experiments   EX1:  Is  ra;ng  coherence  a  good   predictor  for  the  Magic  Barrier?         EX2:  Can  we  use  ra;ng  coherence  to   improve  recommenda;on  results? Especially  when  we  do  not  have  re-­‐ ra;ngs/magic  barrier?     9  
  • 10. RaAng  Coherence  vs.   Magic  Barrier   Rank  users  according  to  magic  barrier/ ra;ng  coherence:   •  High/low  magic  barrier   •  Low/high  ra;ng  coherence  in  the   “genre”  feature  space     Find  correla;on  (Spearman)  between   magic  barrier/coherence  ranking     *Dataset  from  moviepilot.de,  contains  re-­‐ra;ngs   from  306  users  [Said  et  al.  UMAP  ‘12]     10  
  • 11. Magic  Barrier  &  Ra;ng  Coherence  correla;on   11  
  • 12. Experiments   EX1:  Is  ra;ng  coherence  a  good   predictor  for  the  Magic  Barrier?         EX2:  Can  we  use  ra;ng  coherence  to   improve  recommenda;on  results?   Especially  when  we  do  not  have  re-­‐ ra;ngs/magic  barrier?   12  
  • 13. Use  raAng  coherence  for   recommendaAon   Divide  users  into  coherent  and   incoherent  –  with  higher  an  lower   magic  barrier   Movielens  1M     Train  each  group  separately     Evaluate  each  group  separately       13  
  • 14. RecommendaAon  and   EvaluaAon   Algorithm:  K-­‐NN,  Pearson     Training  sets   •  “Easy”  users  split   •  “Difficult”  users  split   •  All  users  split     Evalua;on  sets   •  “Easy”  users  split   •  “Difficult”  users  split   •  All  users  split   Training   Easy   Training   Difficult   Test   Easy   Test   Difficult   14  
  • 15. EvaluaAon   Comparisons     All  -­‐  All   All  –  Easy   All  –  Difficult   Easy  –  Easy   Difficult  –  Difficult       Training   Easy   Training   Difficult   Test   Easy   Test   Difficult   RMSE:  1.090   User  coverage:  100%     15  
  • 16. EvaluaAon   Comparisons     All  -­‐  All   All  –  Easy   All  –  Difficult   Easy  –  Easy   Difficult  –  Difficult       Training   Easy   Training   Difficult   Test   Easy   Test   Difficult   RMSE:  0.974   User  coverage:  50%     16  
  • 17. EvaluaAon   Comparisons     All  -­‐  All   All  –  Easy   All  –  Difficult   Easy  –  Easy   Difficult  –  Difficult       Training   Easy   Training   Difficult   Test   Easy   Test   Difficult   RMSE:  1.195   User  coverage:  50%     17  
  • 18. EvaluaAon   Comparisons     All  -­‐  All   All  –  Easy   All  –  Difficult   Easy  –  Easy   Difficult  –  Difficult       Training   Easy   Training   Difficult   Test   Easy   Test   Difficult   RMSE:  0.933   User  coverage:  50%     18  
  • 19. EvaluaAon   Comparisons     All  -­‐  All   All  –  Easy   All  –  Difficult   Easy  –  Easy   Difficult  –  Difficult       Training   Easy   Training   Difficult   Test   Easy   Test   Difficult   RMSE:  1.226   User  coverage:  50%     19  
  • 20. Results   Subset   RMSE   User  Coverage   All   1.090   100%   All-­‐Easy   0.974   50%   All-­‐Diff   1.195   50%   Easy-­‐Easy   0.933   50%   Diff-­‐Diff   1.226   50%   20  
  • 21. Results   Subset   RMSE   User  Coverage   All   1.090   100%   All-­‐Easy   0.974   50%  (easy)   All-­‐Diff   1.195   50%  (diff)   Easy-­‐Easy   0.933   50%  (easy)   Diff-­‐Diff   1.226   50%  (diff)   RMSE   average:   1.084   21  
  • 22. Results   Subset   RMSE   User  Coverage   All   1.090   100%   All-­‐Easy   0.974   50%  (easy)   All-­‐Diff   1.195   50%  (diff)   Easy-­‐Easy   0.933   50%  (easy)   Diff-­‐Diff   1.226   50%  (diff)   RMSE   average:   1.064   22  
  • 23. Experiments   EX1:  Is  ra;ng  coherence  a  good   predictor  for  the  Magic  Barrier?         EX2:  Can  we  use  ra;ng  coherence  to   improve  recommenda;on  results?   Especially  when  we  do  not  have  re-­‐ ra;ngs/magic  barrier?   23  
  • 24. Conclusions   Ra;ng  coherence  within  an  item’s   feature  space  is  related  to  the  magic   barrier     Ra;ng  coherence  can  be  used  to   predict  users’  ra;ng  inconsistencies     A  user’s  ra;ng  inconsistencies  tell  us   how  well  our  system  performs  for  that   user  –  i.e.  whether  we  need  to   op;mize  further   24  
  • 25. Thanks!     Ques;ons?                         More  recommender  systems:   www.recsyswiki.com     25  
  • 26. Image  credits   •  Slide  1  -­‐  hqp://www.burdens.net.au/product/esf-­‐barriers   •  Slide  2  -­‐  hqp://www.freepicturesweb.com/pictures/2010-­‐07/5881.html   •  Slide  4  -­‐  hqp://redgannet.blogspot.nl/2010/08/moreletakloof-­‐johannesburg-­‐jnb.html   •  Slide  5  -­‐  hqp://en.wikipedia.org/wiki/Standard_devia;on#Interpreta;on_and_applica;on   •  Slide  9,  12,  23  -­‐  hqp://www.withautodesk.com/html/science-­‐fair-­‐ideas-­‐for-­‐physical-­‐science-­‐inven;ons.html   •  Slide  13  -­‐  hqp://fineartamerica.com/featured/16-­‐cogs-­‐les-­‐cunliffe.html   •  Slide  24  -­‐  hqps://benchprep.com/blog/the-­‐ap-­‐english-­‐exam-­‐wri;ng-­‐compelling-­‐conclusions/   26