Institut de Recherche en Informatique de Toulouse (IRIT) - UMR 5505                  Bridging the gap between users and sy...
Diversity in recommender systems  How to recommend documents for a visited one              Maximizing the chances of re...
Diversity in recommender systems  How to recommend documents for a visited one              Maximizing the chances of re...
What is diversity?  Definitions from the literature    Topicality              Related to a particular topic [Xu and Ch...
Approaches to diversify IR results  Topical diversity     Clustering              Identify aspects              Reorde...
Approaches to diversify IR results  Topical diversity     Sliding Windows              Reorder the retrieved documents ...
Approaches to diversify IR results  Serendipity     Alternative to topical diversity     Similarity not only based on t...
Analysis of the TREC Web 2009 results  Hypothesis    Diversity of approaches              No one approach for all users...
Analysis of the TREC Web 2009 results  Experimental framework     Reference IR corpus (TREC Web 2009)        Two IR con...
Analysis of the TREC Web 2009 results  Adhoc Task    Top 10 documents      Overlap: 22.4%      Precision: 0.384    Ov...
Analysis of the TREC Web 2009 results  Diversity Task    Top 10 documents      Overlap: 6.3%    Overlap max < 15%27/10...
Analysis of the TREC Web 2009 results  Conclusions     Two distinct approaches are unlikely to return the same      (rel...
Users’ Study  Our intuitions    Most of the time, users want topicality              Get focused information        So...
Users’ Study  Goals     Verify our intuitions     Prove that diversified recommendations answer a larger      range of ...
Users’ Study  Experimental Framework     Select a document27/10/11      Candillier L. – Chevalier M. – Dudognon D. – Mot...
Users’ Study  Experimental Framework     Read the selected      document27/10/11      Candillier L. – Chevalier M. – Dud...
Users’ Study  Experimental Framework     Compute the recommendation lists       Approach 1                              ...
Users’ Study  Experimental Framework     Compute the recommendation lists       Approach 1                              ...
Users’ Study  Experimental Framework     Compute the recommendation lists       Approach 1                              ...
Users’ Study  Experimental Framework     Compute the recommendation lists       Approach 1                              ...
Users’ Study  Experimental Framework     Compute the recommendation lists       Approach 1                              ...
Users’ Study  Experimental Framework     Present recommendation lists for assessment                   Which list best m...
Users’ Study  Experimental Framework     Present recommendation lists for assessment                   Which list is the...
Users’ Study  Experimental Framework     Assessment of all      documents27/10/11      Candillier L. – Chevalier M. – Du...
Users’ Study  Approaches used     searchsim              Vector-space model              Document title as query      ...
Users’ Study  Approaches used                                                                  kmeans          K-...
Users’ Study  Approaches used                                                                                    ...
Users’ Study  Approaches used    Same analysis than TREC     experiments      Same results      Overlap is low (< 10%)...
Users’ Study     Results        Distribution of relevant documents   blogart                           fused            ...
Users’ Study     Results        Distribution of relevant documents                                                      ...
Users’ Study  Results     Distribution of relevant documentsblogart                           fused           35%       ...
Users’ Study  Results     Distribution of relevant documents              Relevant mainly retrieved by topical approach...
Conclusions and future work  Conclusions     Diversity of users’ expectations              No one approach to rule them...
Conclusions and future work  Future work     Real scale experiment              OverBlog platform        Renew the use...
Conclusions and future work  Future work     Improve the model              Refining the fusing process              A...
Thank you for your attention                               Questions ?27/10/11         Candillier L. – Chevalier M. – Dudo...
References W. Bi, X. Yu, Y. Liu, F. Guan, Z. Peng, H. Xu, and X. Cheng, “ICTNET at Web Track 2009 diversity task”, Text RE...
Upcoming SlideShare
Loading in …5
×

Diversity in recommender systems - Bridging the gap between users and systems

433 views
403 views

Published on

Conference Centric 2011, Barcelona, Spain

Full paper available at : http://www.thinkmind.org/index.php?view=article&articleid=centric_2011_2_30_30049

Abstract :
Recommender systems aim at automatically providing objects related to user’s interests. The angular stone of such systems is a way to identify documents to be recommended. Indeed, the quality of these systems depends on the accuracy of its recommendation selection method. Thus, the selection method should be carefully chosen in order to improve end-user satisfaction. In this paper, we first compare two sets of approaches from the literature to underline that their results are significantly different. We also provide the conclusion of a survey done by thirty four students showing that diversity is considered as important in recommendation lists. Finally, we show that combining existing recommendation selection methods is a good mean to obtain diversity in recommendation lists.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
433
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
16
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Diversity in recommender systems - Bridging the gap between users and systems

  1. 1. Institut de Recherche en Informatique de Toulouse (IRIT) - UMR 5505 Bridging the gap between users and systems Laurent CANDILLIER – Max CHEVALIER – Damien DUDOGNON – Josiane MOTHE27/10/11
  2. 2. Diversity in recommender systems  How to recommend documents for a visited one  Maximizing the chances of retrieving at least one relevant document per user [Santos et al., 2010]  Cover a large range of users’ interests  Context  Blog platform  Unknown user => no profile  Diversity of users, diversity of their expectations27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 2
  3. 3. Diversity in recommender systems  How to recommend documents for a visited one  Maximizing the chances of retrieving at least one relevant document per user [Santos et al., 2010]  Cover a large range of users’ interests  Context  Blog platform  Unknown user => no profile  Diversity of users, diversity of their expectations => Diversify the recommendations27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 3
  4. 4. What is diversity?  Definitions from the literature  Topicality  Related to a particular topic [Xu and Chen, 2006]  Diversity  Topical diversity  Extrinsic: solve ambiguity [Radlinski et al., 2009]  Intrinsic: avoid redundancy [Clarke et al., 2008]  Serendipity  Attractive and surprising documents [Herlocker et al., 2004]27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 4
  5. 5. Approaches to diversify IR results  Topical diversity  Clustering  Identify aspects  Reorder depending on the aspects covered  Examples  K-Means [Bi et al., 2009]  Hierarchical Clustering [Meij et al., 2010]27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 5
  6. 6. Approaches to diversify IR results  Topical diversity  Sliding Windows  Reorder the retrieved documents  Select documents using metrics  Similarity with the visited document  Similarity with the current recommended document list  Examples  MMR [Carbonell and Goldstein, 1998]  Intra-list similarity [Ziegler et al., 2005]27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 6
  7. 7. Approaches to diversify IR results  Serendipity  Alternative to topical diversity  Similarity not only based on the content  Examples  Organizational similarity [Cabanac et al., 2007]  Temporal diversity [Lathia et al., 2010]27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 7
  8. 8. Analysis of the TREC Web 2009 results  Hypothesis  Diversity of approaches  No one approach for all users’ needs  Approaches are complementary  Valuable to combine them  Goals  Analyse results obtained with approaches having  Same goal  Similar performances => To identify if diversity exists27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 8
  9. 9. Analysis of the TREC Web 2009 results  Experimental framework  Reference IR corpus (TREC Web 2009)  Two IR contexts  Adhoc task  Diversity task  Compare results (runs) of the 4 best approaches of each task  Similar performances according to IR metrics  MAP for adhoc task  NDCG for diversity task  Overlap for each pair of runs underlying diversity27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 9
  10. 10. Analysis of the TREC Web 2009 results  Adhoc Task  Top 10 documents  Overlap: 22.4%  Precision: 0.384  Overlap max < 30%27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 10
  11. 11. Analysis of the TREC Web 2009 results  Diversity Task  Top 10 documents  Overlap: 6.3%  Overlap max < 15%27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 11
  12. 12. Analysis of the TREC Web 2009 results  Conclusions  Two distinct approaches are unlikely to return the same (relevant) documents  Low average overlap  Diversity of approaches  No approach significantly better than others  A combination can be valuable  TREC tasks focused on topicality and topical diversity  Can’t be used to evaluate other types of diversity  Users’ study necessary [Hayes et al., 2002]27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 12
  13. 13. Users’ Study  Our intuitions  Most of the time, users want topicality  Get focused information  Sometime, they want diversity  Topical diversity  Enlarge the subject  Serendipity  Discover new information27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 13
  14. 14. Users’ Study  Goals  Verify our intuitions  Prove that diversified recommendations answer a larger range of users’ needs  Context of experimentation  34 students in M. Sc. (Management field)  Blog post recommendations27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 14
  15. 15. Users’ Study  Experimental Framework  Select a document27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 15
  16. 16. Users’ Study  Experimental Framework  Read the selected document27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 16
  17. 17. Users’ Study  Experimental Framework  Compute the recommendation lists Approach 1 List 1 (random) Approach 2 Approach 3 Approach 4 List 2 (fused) Approach 527/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 17
  18. 18. Users’ Study  Experimental Framework  Compute the recommendation lists Approach 1 List 1 (random) Approach 2 Approach 3 Approach 4 List 2 (fused) Approach 527/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 18
  19. 19. Users’ Study  Experimental Framework  Compute the recommendation lists Approach 1 List 1 (random) Approach 2 Approach 3 Approach 4 List 2 (fused) Approach 527/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 19
  20. 20. Users’ Study  Experimental Framework  Compute the recommendation lists Approach 1 List 1 (random) Approach 2 Approach 3 Approach 4 List 2 (fused) Approach 527/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 20
  21. 21. Users’ Study  Experimental Framework  Compute the recommendation lists Approach 1 List 1 (random) Approach 2 Approach 3 Approach 4 List 2 (fused) Approach 527/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 21
  22. 22. Users’ Study  Experimental Framework  Present recommendation lists for assessment Which list best meets your needs?27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 22
  23. 23. Users’ Study  Experimental Framework  Present recommendation lists for assessment Which list is the most diversified?27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 23
  24. 24. Users’ Study  Experimental Framework  Assessment of all documents27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 24
  25. 25. Users’ Study  Approaches used  searchsim  Vector-space model  Document title as query  mlt Topicality  Apache Solr MoreLikeThis module  Document content as query27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 25
  26. 26. Users’ Study  Approaches used        kmeans  K-means classification Topical diversity  One element per cluster27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 26
  27. 27. Users’ Study  Approaches used           blogart  Random selection from the same blog  topcateg Serendipity  Popular documents in the same category27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 27
  28. 28. Users’ Study  Approaches used  Same analysis than TREC experiments  Same results  Overlap is low (< 10%) => High diversity27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 28
  29. 29. Users’ Study  Results  Distribution of relevant documents blogart fused kmeans fused 35% 65% 52.5% 21.3% 0% 26.2% mlt fused 54.7% 32.8% 12.5%searchsim fused topcateg fused 52.4% 38.9% 8.8% 91.2% 8.7% 0% 27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 29
  30. 30. Users’ Study  Results  Distribution of relevant documents kmeans fused 35% 65% 52.5% 21.3% 0% 26.2% mlt fused 54.7% 32.8% 12.5%searchsim fused 52.4% 38.9% 8.8% 91.2% 8.7% 0% 27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 30
  31. 31. Users’ Study  Results  Distribution of relevant documentsblogart fused 35% 65% 52.5% 21.3% 0% 26.2% 54.7% 32.8% 12.5% topcateg fused 52.4% 38.9% 8.8% 91.2% 8.7% 0%27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 31
  32. 32. Users’ Study  Results  Distribution of relevant documents  Relevant mainly retrieved by topical approaches  But at least 20% are retrieved only by fused  Fused matches with a larger range of needs27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 32
  33. 33. Conclusions and future work  Conclusions  Diversity of users’ expectations  No one approach to rule them all  A diversity of approaches  Complementary  Fused  Diversity helps RS to fit more users’ needs27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 33
  34. 34. Conclusions and future work  Future work  Real scale experiment  OverBlog platform  Renew the user survey  More users (international call for participation)  Avoid revealed biases  e.g. More detailed form => Deeper analysis27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 34
  35. 35. Conclusions and future work  Future work  Improve the model  Refining the fusing process  Adding a learning process to weight each approach  For every visited document  Find the proportion of documents coming from each approach (log analysis)  Better match with the real users’ needs27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 35
  36. 36. Thank you for your attention Questions ?27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 36
  37. 37. References W. Bi, X. Yu, Y. Liu, F. Guan, Z. Peng, H. Xu, and X. Cheng, “ICTNET at Web Track 2009 diversity task”, Text REtrieval Conf., 2009 G. Cabanac, M. Chevalier, C. Chrisment, and C. Julien, “An Original Usage-based Metrics for Building a Unified View of Corporate Documents”, Inter. Conf. on Database and Expert Systems Applications, 2007, LNCS V. 4653, 2007, pp. 202–212 J. Carbonell and J. Goldstein, “The use of MMR, diversity-based reranking for reordering documents and producing summaries”, ACM Conf. on Research and Development in Information Retrieval, 1998, pp. 335-336 C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I.n MacKinnon, “Novelty and Diversity in Information Retrieval Evaluation”, ACM Conf. on Research and Development in Information Retrieval, 2008, pp. 659-666 C. Hayes, P. Massa, P. Avesani, and P. Cunningham, « An online evaluation framework for recommender systems», Workshop on Personalization and Recommendation in E-Commerce, 2002 J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, “Evaluating Collaborative Filtering Recommender Systems”, ACM Trans. Information Systems, 22(1), 2004, pp. 5-53 N. Lathia, S. Hailes, L. Capra, and X. Amatriain, “Temporal diversity in recommender systems”, ACM Conf. on Research and Development in Information Retrieval, 2010, pp. 210-217 E. Meij, J. He, W. Weerkamp, and M. de Rijke, “Topical Diversity and Relevance Feedback”, Text REtrieval Conf., 2010 F. Radlinski, P. N. Bennett, B. Carterette, and T. Joachims. “Redundancy, diversity and interdependent document relevance”, SIGIR Forum, 43(2), 2009, pp. 46–52 R. L. T. Santos, C. Macdonald, and I. Ounis, “Selectively Diversifying Web Search Results”, ACM Inter. Conf. on Information and Knowledge Management, 2010 Y. C. Xu and Z. Chen, “Relevance judgment: What do information users consider beyond topicality”, Journal of the American Society for Information Science and Technology, 57(7), 2006, pp. 961–973 C. Ziegler, S. McNee, J. A. Konstan, and G. Lausen, “Improving recommendation lists through topic diversification”, Inter. Conf. on World Wide Web, 2005, pp. 22–3227/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 37

×