The Magic Barrier of Recommender Systems - No Magic, Just Ratings

1. The Magic Barrier of Recommender Systems: No Magic, Just ra;ngs Alejandro Bellogín, Alan Said, Arjen de Vries @abellogin, @alansaid, @arjenpdevries UMAP 2014 2014-‐07-‐08

2. What is the magic barrier? 2

3. The Magic Barrier The upper bound on recommenda;on accuracy Recommenda;on accuracy is only as good as the quality of the underlying data Users are noisy / incoherent in their ra;ng behavior 3

4. How to find the magic barrier? 1.  Collect infinite re-‐ra;ngs for each item from your users. 2.  Find standard devia;on of ra;ng/ re-‐ra;ng inconsistencies (noise) [Said et al. UMAP ‘12] Realis;cally, an infinite amount of re-‐ra;ngs is not probable 4

5. Coherence 5

6. What is coherence? Coherent user e slightly different. Specifically, the user in Figure 1a gives more consisten items sharing the same features, which in this case corresponds to the mov s. In the long term we argue that such a user will have a more consistent (le ehaviour, since her taste for each item feature seem to be well defined. Action Comedy Horror (a) A coherent user (c(u) = 1.28) Action Comedy Horror (b) A not coherent user (c(u) = Incoherent user htly different. Specifically, the user in Figure 1a gives more consistent rating sharing the same features, which in this case corresponds to the movie’s gen he long term we argue that such a user will have a more consistent (less noisy ur, since her taste for each item feature seem to be well defined. Action Comedy Horror A coherent user (c(u) = 1.28) Action Comedy Horror (b) A not coherent user (c(u) = 5.95) 6

7. DeﬁniAon of Coherence Ra;ng consistency within a feature space (genres, themes, etc.) c u( )= − σ f f ∈F ∑ u( ) σ f u( )= r u,i( )− rf u( )( ) 2 i∈I u, f( ) ∑ Item features User’s ra;ng devia;on within a feature 7

8. Ra;ng devia;on within a feature space instead of inﬁnite re-‐ra;ngs Is this a good es;mate of the magic barrier? 8

9. Experiments EX1: Is ra;ng coherence a good predictor for the Magic Barrier? EX2: Can we use ra;ng coherence to improve recommenda;on results? Especially when we do not have re-‐ ra;ngs/magic barrier? 9

10. RaAng Coherence vs. Magic Barrier Rank users according to magic barrier/ ra;ng coherence: •  High/low magic barrier •  Low/high ra;ng coherence in the “genre” feature space Find correla;on (Spearman) between magic barrier/coherence ranking *Dataset from moviepilot.de, contains re-‐ra;ngs from 306 users [Said et al. UMAP ‘12] 10

11. Magic Barrier & Ra;ng Coherence correla;on 11

13. Use raAng coherence for recommendaAon Divide users into coherent and incoherent – with higher an lower magic barrier Movielens 1M Train each group separately Evaluate each group separately 13

14. RecommendaAon and EvaluaAon Algorithm: K-‐NN, Pearson Training sets •  “Easy” users split •  “Difficult” users split •  All users split Evalua;on sets •  “Easy” users split •  “Difficult” users split •  All users split Training Easy Training Difficult Test Easy Test Difficult 14

15. EvaluaAon Comparisons All -‐ All All – Easy All – Difficult Easy – Easy Difficult – Difficult Training Easy Training Difficult Test Easy Test Difficult RMSE: 1.090 User coverage: 100% 15

20. Results Subset RMSE User Coverage All 1.090 100% All-‐Easy 0.974 50% All-‐Diff 1.195 50% Easy-‐Easy 0.933 50% Diff-‐Diff 1.226 50% 20

21. Results Subset RMSE User Coverage All 1.090 100% All-‐Easy 0.974 50% (easy) All-‐Diff 1.195 50% (diff) Easy-‐Easy 0.933 50% (easy) Diff-‐Diff 1.226 50% (diff) RMSE average: 1.084 21

22. Results Subset RMSE User Coverage All 1.090 100% All-‐Easy 0.974 50% (easy) All-‐Diff 1.195 50% (diff) Easy-‐Easy 0.933 50% (easy) Diff-‐Diff 1.226 50% (diff) RMSE average: 1.064 22

24. Conclusions Ra;ng coherence within an item’s feature space is related to the magic barrier Ra;ng coherence can be used to predict users’ ra;ng inconsistencies A user’s ra;ng inconsistencies tell us how well our system performs for that user – i.e. whether we need to op;mize further 24

25. Thanks! Ques;ons? More recommender systems: www.recsyswiki.com 25

26. Image credits •  Slide 1 -‐ hqp://www.burdens.net.au/product/esf-‐barriers •  Slide 2 -‐ hqp://www.freepicturesweb.com/pictures/2010-‐07/5881.html •  Slide 4 -‐ hqp://redgannet.blogspot.nl/2010/08/moreletakloof-‐johannesburg-‐jnb.html •  Slide 5 -‐ hqp://en.wikipedia.org/wiki/Standard_devia;on#Interpreta;on_and_applica;on •  Slide 9, 12, 23 -‐ hqp://www.withautodesk.com/html/science-‐fair-‐ideas-‐for-‐physical-‐science-‐inven;ons.html •  Slide 13 -‐ hqp://ﬁneartamerica.com/featured/16-‐cogs-‐les-‐cunliﬀe.html •  Slide 24 -‐ hqps://benchprep.com/blog/the-‐ap-‐english-‐exam-‐wri;ng-‐compelling-‐conclusions/ 26