Personalized Diversification of Search Results

1,131 views

Published on

Search personalization and diversification are often seen as oppos-ing alternatives to cope with query uncertainty, where, given an ambiguous query, it is either preferable to adapt the search result to a specific aspect that may interest the user (personalization) or to regard multiple aspects in order to maximize the probability that some query aspect is relevant to the user (diversification). In this work, we question this antagonistic view, and hypothesize that these two directions may in fact be effectively combined and enhance each other. We research the introduction of the user as an explicit random variable in state of the art diversification methods, thus developing a generalized framework for personalized diversi-fication. In order to evaluate our hypothesis, we conduct an evalu-ation with real users using crowdsourcing services. The obtained results suggest that the combination of personalization and diver-sification achieves competitive performance, improving the base-line, plain personalization, and plain diversification approaches in terms of both diversity and accuracy measures.

Published in: Technology
1 Comment
4 Likes
Statistics
Notes
  • very helpful, thanx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
1,131
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
39
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide
  • Personalizedia-selecttopicweightissustitutedbyuserpreferencetotopic
  • Personalizedia-selecttopicweightissustitutedbyuserpreferencetotopic
  • Personalizedia-selecttopicweightissustitutedbyuserpreferencetotopic
  • Onephrasemaliciousworkers
  • Randlinski and Dumais
  • Personalized Diversification of Search Results

    1. 1. Personalized Diversification of Search Results David Vallet, Pablo Castells Universidad Autónoma de Madrid {david.vallet,pablo.castells}@uam.es Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012
    2. 2. Classic Web Search Model Query: Queen ? Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 2
    3. 3. Classic Web Search Model Query: Queen ? Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 3
    4. 4. Personalized Web Search Model Query: Queen Personalized ordering User Profile Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 4
    5. 5. Diversified Web Search Model Query: Queen Diversification model ? Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 5
    6. 6. Personalized Diversification of Web Search? Query: Queen  Personalization User Tailor the results to the specific interests of the user Profile Relies on accurate user profile Risk of being perceived as intrusive by the user  Diversification Cover all interpretations of the query in the first results Maximize probability of showing an interpretation relevant to the user Diversification Outliers model Users not finding a relevant result in the top ? positions Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 6
    7. 7. Diversified Web Search Model: SoA diversification approaches document relevance novelty Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 7
    8. 8. Diversified Web Search Model: SoA diversification approaches document query document topic relevance relevance novelty Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 8
    9. 9. Diversified Web Search Model: Diversification VS Personalization baseline Diversification Personalization (IA-Select) (BM-25) Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 9
    10. 10. Diversified Web Search Model: Diversification VS Personalization baseline Diversification Diversified Personalization (IA-Select) personalization (BM-25) Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 10
    11. 11. Diversified Web Search Model: Diversification VS Personalization Small study: what do users prefer? • If profile is perfectly (explicitly) defined  personalization over any diversification • If profile has errors  personalized diversity over full personalization or diversification baseline Diversification diversified Personalization (IA-Select) personalization (BM-25) Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 11
    12. 12. Personalized Diversification of Web Search: Personalized IA-Select  IA-SELECT document relevance novelty  Personalized IA-SELECT – Adding a user component Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 12
    13. 13. Personalized Diversification of Web Search: Personalized xQuAD document query document topic  xQuAD relevance relevance novelty  Personalized xQuAD – Adding a user component Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 13
    14. 14. Personalized Diversification of Web Search: Model  Personalized IA-SELECT  Personalized xQuAD  Estimations Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 14
    15. 15. Personalized Diversification of Web Search: Model estimations delicious Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 15
    16. 16. Personalized Diversification of Web Search: Model estimations 1 http://www.textwise.com Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 16
    17. 17. How to evaluate? delicious Evaluation topic ?  Accuracy assessments  Diversity assessments – Document relevance assessments – Document/aspect assessments  Crowdsourced evaluation Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 17
    18. 18. Personalized Diversification of Web Search: Online Evaluation Social User Profile u Social tagging t1,d t2,d tl,d … • User annotations • Search result annotations 4 t1 1 d1 2 3 d1 Depth-5 t2 Top k Search pooling popular tags results d2 d2 Web Search Result reordering … … … dn d‘P’ t1 • User classification • Search result classification … Document classifier c1,d c2,d cc,d Users get to evaluate known topics Must be done live, cannot make workers wait Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 18
    19. 19. Crowdsourced Evaluation: UI Desing  Assessments – Q1: user relevance Accuracy (F-measure) – Q2: topic relevance – Q3: document-category classification: subjective to users, foster the reuse of categories (avg. 5 categories per topic)  Include a training task for workers (they appreciate it) Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 19
    20. 20. Personalized Diversification of Web Search: Crowdsourced Evaluation  Collected data – Period: 4 weeks – 35 Delicious users: bookmarks, top tags, tag assignments, etc. – 180 search topics – Over 3800 relevance judgments  Provided as evaluation and development dataset – http://ir.ii.uam.es/dvallet/persdivers Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 20
    21. 21. Personalized Diversification of Web Search: Results Accuracy metrics Topic relevance User relevance F(Topic,User) nDCG@5 nDCG@5 nDCG@5 P@5 P@5 P@5 Bing 0.393 0.918 0.330 0.702 0.359 0.796 IA-Select 0.363 0.911 0.294 0.664 0.325 0.768 Diversity xQuAD 0.381 0.911 0.320 0.670 0.348 0.772 Personalization 0.388 0.933 0.357 0.746 0.372 0.829 PIA-Select 0.331 0.861 0.306 0.652 0.318 0.742 Personalized PIA-SelectBM25 0.363 0.892 0.345 0.670 0.354 0.766 Diversity PxQuAD 0.395 0.930 0.337 0.714 0.364 0.807 PxQuADBM25 0.405 0.939 0.361 0.730 0.382 0.821 Statistically significant Statistically significant with respect to Baseline with respect to xQuAD Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 21
    22. 22. Personalized Diversification of Web Search: Results Diversity metrics Topic relevance User relevance α-nDCG@5 α-nDCG@5 ERR-IA@5 ERR-IA@5 S-recall@5 S-recall@5 Bing 0.274 0.787 0.500 0.254 0.626 0.475 Diversity IA-Select 0.262 0.758 0.463 0.237 0.582 0.426 xQuAD 0.275 0.793 0.510 0.257 0.624 0.478 Personalization 0.267 0.778 0.486 0.262 0.646 0.483 PIA-Select 0.251 0.742 0.489 0.239 0.593 0.481 Personalized PIA-SelectBM25 0.279 0.803 0.543 0.273 0.656 0.521 Diversity PxQuAD 0.275 0.796 0.503 0.268 0.649 0.489 PxQuADBM25 0.281 0.810 0.519 0.267 0.658 0.496 Statistically significant Statistically significant with respect to Baseline with respect to xQuAD Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 22
    23. 23. Conclusions  Adapted SoA diversification techniques to include a personalized factor  Presented an example of personalized diversification of Web search – Personalization: social tagging – Diversification: ODP Web categories  Effectively combined both benefits of personalization and diversification  Crowdsourced evaluation approach for diversity and personalized techniques – Great if you have a lot of restrictions – Repeatable – Cheap – …Although can be difficult to set up Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012 23
    24. 24. Thank you!!! Thank you!  References – [Vallet-10]: David Vallet, Iván Cantador, Joemon M. Jose: Personalizing Web Search with Folksonomy-Based User and Document Profiles. ECIR 2010: 420-431 Personalized Diversification of Search ResultsIRGIR Group @ UAM 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, 15th August 2012

    ×