RANKING AND DIVERSITY
IN RECOMMENDATIONS
Alexandros Karatzoglou
@alexk_z
Thanks

Linas Baltrunas
Telefonica

Yue Shi
TU Delft

Saul Vargas
Universidad Autonoma de Madrid

Pablo Castells
Universid...
Telefonica Research in Barcelona
•  User Modeling: Recommender Systems
•  Data Mining, Machine Learning
•  Multimedia Inde...
Recommendations in Telefonica
People You May Know@Tuenti
Recommendations in Telefonica
Firefox OS Marketplace
Collaborative Filtering

+
+

+
+

+

+
+

+
X ↵ik
fij = hUi Mj i i
hUk Mj i
, Mj +
|Fi |
+

k2Fi
Tensor Factorization
CP-Decomposition
HOSVD

Fijkijk S ⇥U Ui ,⇥M j , CkC Ck
F = = hU i M Mj ⇥ i
Karatzoglou et al. @ Recsy...
Publications in Ranking
CIKM 2013: GAPfm: Optimal Top-N Recommendations for Graded Relevance Domains
RecSys 2013: xCLiMF: ...
Recommendations are ranked lists
Popular Ranking Methods
•  In order to generate the ranked item list, we need some

relative utility score for each item
•...
Graphical Notation

Relevant	
  
Irrelevant	
  
Irrelevant	
  
Irrelevant	
  
Irrelevant	
  
Irrelevant	
  
Relevant	
  
R...
Ranking using latent
representation
•  If user = [-100, -100]
•  2d latent factor
•  We get the corresponding ranking
Matrix Factorization
(for ranking)
•  Randomly initialize item vectors
•  Randomly initialize user vectors
•  While not co...
Learning: Stochastic Gradient
Descent with Square Error Loss
SquareUser: [3, 1], RMSE=6.7
Loss
Learning	
  to	
  Rank	
  for	
  Top-­‐k	
  RecSys	
  
•  Usually	
  we	
  care	
  about	
  accurate	
  ranking	
  and	
  ...
Example: average precision (AP)
•  AP: we compute the precision at each relevant position

and average them

AP =

|S|
X

...
Why is hard? Non Smoothness
Example: AP

u:[-20,-20]

u:[20,20]
AP vs RMSE
The Non-smoothness of
Average Precision
AP =

|S|
X

P (k)

k=1

|S|

APm = PN

1

i=1

ymi

N
N
X ymi X
i=1

rmi

j=1

ym...
How can we get a smooth-AP?
•  We	
  replace	
  non	
  smooth	
  parts	
  of	
  MAP	
  with	
  smooth	
  

approxima=on	
 ...
How can we get a smooth-MAP?
•  We	
  replace	
  non	
  smooth	
  parts	
  of	
  MAP	
  with	
  smooth	
  

approxima=on	
...
Smooth version of MAP

u:[-20,-20]

u:[20,20]
Sometimes approximation is not very
good…
Ranking Inconsistencies
•  Achieving a perfect ranking for all users is not possible
•  Two Sources of Inconsistencies:
• ...
Summary on Ranking 101
Area Under the ROC Curve (AUC)

1
AU C := +
|S ||S |

S+ X
XS
i

j

I(Ri < Rj )
Reciprocal Rank (RR)

1
RR :=
Ri
Average Precision (AP)

AP =

|S|
X

P (k)

k=1

|S|
AP vs RR
Normalized Discounted Cumulative Gain
(nDCG)

DCG =

X 2score(i)
i

1
log2 (i + 2)
Relevance solved! Is that all?
•  Ranking “solves” the relevance problem
•  Can we be happy with the results?
Relevance solved! Is that all?
•  Coverage
•  Diversity
•  Novelty
•  Serendipity
Diversity in Recommendations
•  Diversity using Genres
•  Movies, Music, Books
•  Diversity should fulfill:
•  Coverage
• ...
Diversity methods for RecSys
•  Topic List Diversification or
•  Maximal Margin Relevance

fM M R (i; S) = (1

) rel(i) +
...
Diversity methods for RecSys
•  Intent-Aware IR metrics

ERR

IA =

X
s

p(s)ERR
Example

Action
Comedy
Sci-Fi
Western

Action
Thriller
Sci-Fi
Western

Adventure
Western
Genres

Action (517)
154

Comedy (1267)

125

8
11

6

60

37
4

146

2
639

187

21

9
0

30

8

3
9

4

4

232

171
11

...
Genres and Popularity
Binomial Diversity
•  We base a new Diversity Metric on the Binomial
Distribution

✓ ◆
N k
P (X = k) =
p (1
k

p)

N

k
User Genre Relevance
•  Fraction of Items of genre “g” user interacted
with
Iu

p00
g

kg
=
|Iu |

•  Global fraction of i...
Coverage
•  Product of the probabilities of the genres not represented
in the list not being picked by random

Coverage(R)...
Non-Redundancy
P (Xg

k | Xg > 0) = 1

N onRed(R) =

Y

g2G(R)

k 1
X
l=1

P (Xg

P (Xg = l | Xg > 0)
R
kg | Xg > 0)1/|G(R...
Non-Redundancy
Binomial Diversity

BinomDiv(R) = Coverage(R) · N onRed(R)
Re-Ranking
•  Re-Rank based on Relevance and Binomial Diversity

fBinomDiv (i; S) = (1

) normrel (rel(i)) + normdiv (div(...
Example
Thanks!
•  Questions?
Ranking and Diversity in Recommendations - RecSys Stammtisch at SoundCloud, Berlin
Upcoming SlideShare
Loading in...5
×

Ranking and Diversity in Recommendations - RecSys Stammtisch at SoundCloud, Berlin

2,077
-1

Published on

Slides from my talk at the RecSys Stammtisch at SoundCloud in Berlin. The presentation is split in two part one focusing on ranking and relevance and one on diversity and how to achieve it using genres. We introduce a novel diversity metric called Binomial Diversity.

Published in: Technology

Ranking and Diversity in Recommendations - RecSys Stammtisch at SoundCloud, Berlin

  1. 1. RANKING AND DIVERSITY IN RECOMMENDATIONS Alexandros Karatzoglou @alexk_z
  2. 2. Thanks Linas Baltrunas Telefonica Yue Shi TU Delft Saul Vargas Universidad Autonoma de Madrid Pablo Castells Universidad Autonoma de Madrid
  3. 3. Telefonica Research in Barcelona •  User Modeling: Recommender Systems •  Data Mining, Machine Learning •  Multimedia Indexing and Analysis •  HCI •  Mobile Computing •  Systems and Networking •  http://www.tid.es
  4. 4. Recommendations in Telefonica
  5. 5. People You May Know@Tuenti
  6. 6. Recommendations in Telefonica
  7. 7. Firefox OS Marketplace
  8. 8. Collaborative Filtering + + + + + + + + X ↵ik fij = hUi Mj i i hUk Mj i , Mj + |Fi | + k2Fi
  9. 9. Tensor Factorization CP-Decomposition HOSVD Fijkijk S ⇥U Ui ,⇥M j , CkC Ck F = = hU i M Mj ⇥ i Karatzoglou et al. @ Recsys 2010
  10. 10. Publications in Ranking CIKM 2013: GAPfm: Optimal Top-N Recommendations for Graded Relevance Domains RecSys 2013: xCLiMF: Optimizing Expected Reciprocal Rank for Data with Multiple Levels of Relevance RecSys 2012: CLiMF: Learning to Maximize Reciprocal Rank with Collaborative Less-is-More Filtering * Best Paper Award SIGIR 2012: TFMAP: Optimizing MAP for Top-N Context-aware Recommendation Machine Learning Journal, 2008: Improving Maximum Margin Matrix Factorization * Best Machine Learning Paper Award at ECML PKDD 2008 RecSys 2010: List-wise Learning to Rank with Matrix Factorization for Collaborative Filtering NIPS 2007: CoFiRank - Maximum Margin Matrix Factorization for Collaborative Ranking
  11. 11. Recommendations are ranked lists
  12. 12. Popular Ranking Methods •  In order to generate the ranked item list, we need some relative utility score for each item •  Popularity is the obvious baseline •  Score could depend on the user (personalized) •  Score could also depend on the other items in the list (list-wise) •  One popular way to rank the items in RS is to sort the items according to the rating prediction •  Works for the domains with ratings •  Wastes the modeling power for the irrelevant items
  13. 13. Graphical Notation Relevant   Irrelevant   Irrelevant   Irrelevant   Irrelevant   Irrelevant   Relevant   Relevant  
  14. 14. Ranking using latent representation •  If user = [-100, -100] •  2d latent factor •  We get the corresponding ranking
  15. 15. Matrix Factorization (for ranking) •  Randomly initialize item vectors •  Randomly initialize user vectors •  While not converged •  Compute rating prediction error •  Update user factors •  Update item factors •  Lets say user is [-100, -100] •  Compute the square error •  (5-<[-100, -100], [0.180, 0.19]>)2=1764 •  Update the user and item to the direction where the error is reduced (according to the gradient of the loss) 8 items with ratings and random factors
  16. 16. Learning: Stochastic Gradient Descent with Square Error Loss
  17. 17. SquareUser: [3, 1], RMSE=6.7 Loss
  18. 18. Learning  to  Rank  for  Top-­‐k  RecSys   •  Usually  we  care  about  accurate  ranking  and  not  ra=ng   predic=on   •  Square  Error  loss  op=mizes  to  accurately  predict  1s  and  5s.   •  RS  should  get  the  top  items  right  -­‐>  Ranking  problem   •  Why  not  to  learn  how  to  rank  directly?   •  Learning  to  Rank  methods  provide  up  to  30%  performance   improvements  in  off-­‐line  evalua=ons   •  It  is  possible,  but  a  more  complex  task  
  19. 19. Example: average precision (AP) •  AP: we compute the precision at each relevant position and average them AP = |S| X P (k) k=1 |S| P@1+ P@2 + P@4 1 /1+ 2 / 2 + 3 / 4 = = 0.92 3 3
  20. 20. Why is hard? Non Smoothness Example: AP u:[-20,-20] u:[20,20]
  21. 21. AP vs RMSE
  22. 22. The Non-smoothness of Average Precision AP = |S| X P (k) k=1 |S| APm = PN 1 i=1 ymi N N X ymi X i=1 rmi j=1 ymj I(rmj  rmi ) ymi is 1 if item i is relevant for user m and 0 otherwise I(·) indicator function (1 if it is true, 0 otherwise) rmi Rank of item i for user m
  23. 23. How can we get a smooth-AP? •  We  replace  non  smooth  parts  of  MAP  with  smooth   approxima=on   1 rmi ⇡ g(fmi ) = g(hUm , Vi i) g(x) = 1/(1 + e x )
  24. 24. How can we get a smooth-MAP? •  We  replace  non  smooth  parts  of  MAP  with  smooth   approxima=on   I(rmj  rmi ) ⇡ g(fmj fmi ) = g(hUm , Vj Vi i) g(x) = 1/(1 + e x )
  25. 25. Smooth version of MAP u:[-20,-20] u:[20,20]
  26. 26. Sometimes approximation is not very good…
  27. 27. Ranking Inconsistencies •  Achieving a perfect ranking for all users is not possible •  Two Sources of Inconsistencies: •  1) Factor Models (all models) have limited expressive power and cannot learn the perfect ranking for all users •  2) Ranking functions approximations are inconsistent e.g. A >B & B>C but C > A
  28. 28. Summary on Ranking 101
  29. 29. Area Under the ROC Curve (AUC) 1 AU C := + |S ||S | S+ X XS i j I(Ri < Rj )
  30. 30. Reciprocal Rank (RR) 1 RR := Ri
  31. 31. Average Precision (AP) AP = |S| X P (k) k=1 |S|
  32. 32. AP vs RR
  33. 33. Normalized Discounted Cumulative Gain (nDCG) DCG = X 2score(i) i 1 log2 (i + 2)
  34. 34. Relevance solved! Is that all? •  Ranking “solves” the relevance problem •  Can we be happy with the results?
  35. 35. Relevance solved! Is that all? •  Coverage •  Diversity •  Novelty •  Serendipity
  36. 36. Diversity in Recommendations •  Diversity using Genres •  Movies, Music, Books •  Diversity should fulfill: •  Coverage •  Redundancy •  Size Awareness
  37. 37. Diversity methods for RecSys •  Topic List Diversification or •  Maximal Margin Relevance fM M R (i; S) = (1 ) rel(i) + min dist(i, j) j2S
  38. 38. Diversity methods for RecSys •  Intent-Aware IR metrics ERR IA = X s p(s)ERR
  39. 39. Example Action Comedy Sci-Fi Western Action Thriller Sci-Fi Western Adventure Western
  40. 40. Genres Action (517) 154 Comedy (1267) 125 8 11 6 60 37 4 146 2 639 187 21 9 0 30 8 3 9 4 4 232 171 11 1 75 Drama (1711) 86 10 202 870 Romance (583) 46 Thriller (600)
  41. 41. Genres and Popularity
  42. 42. Binomial Diversity •  We base a new Diversity Metric on the Binomial Distribution ✓ ◆ N k P (X = k) = p (1 k p) N k
  43. 43. User Genre Relevance •  Fraction of Items of genre “g” user interacted with Iu p00 g kg = |Iu | •  Global fraction of items of genre “g” •  Mix P Iu u kg 0 pg = P u |Iu | pg = (1 ↵) 0 pg +↵ 00 pg
  44. 44. Coverage •  Product of the probabilities of the genres not represented in the list not being picked by random Coverage(R) = Y g 2G(R) / P (Xg = 0) 1/|G|
  45. 45. Non-Redundancy P (Xg k | Xg > 0) = 1 N onRed(R) = Y g2G(R) k 1 X l=1 P (Xg P (Xg = l | Xg > 0) R kg | Xg > 0)1/|G(R)|
  46. 46. Non-Redundancy
  47. 47. Binomial Diversity BinomDiv(R) = Coverage(R) · N onRed(R)
  48. 48. Re-Ranking •  Re-Rank based on Relevance and Binomial Diversity fBinomDiv (i; S) = (1 ) normrel (rel(i)) + normdiv (div(i; S)) normX (x) = x µX X
  49. 49. Example
  50. 50. Thanks! •  Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×