Search personalization and diversification are often seen as oppos-ing alternatives to cope with query uncertainty, where, given an ambiguous query, it is either preferable to adapt the search result to a specific aspect that may interest the user (personalization) or to regard multiple aspects in order to maximize the probability that some query aspect is relevant to the user (diversification). In this work, we question this antagonistic view, and hypothesize that these two directions may in fact be effectively combined and enhance each other. We research the introduction of the user as an explicit random variable in state of the art diversification methods, thus developing a generalized framework for personalized diversi-fication. In order to evaluate our hypothesis, we conduct an evalu-ation with real users using crowdsourcing services. The obtained results suggest that the combination of personalization and diver-sification achieves competitive performance, improving the base-line, plain personalization, and plain diversification approaches in terms of both diversity and accuracy measures.

1. Personalized Diversification of
Search Results
David Vallet, Pablo Castells
Universidad Autónoma de Madrid
{david.vallet,pablo.castells}@uam.es
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012

2. Classic Web Search Model
Query: Queen
?
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 2

3. Classic Web Search Model
Query: Queen
?
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 3

4. Personalized Web Search Model
Query: Queen
Personalized
ordering
User
Profile
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 4

5. Diversified Web Search Model
Query: Queen
Diversification
model
?
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 5

6. Personalized Diversification of Web Search?
Query: Queen
 Personalization
User Tailor the results to the specific interests of the user
Profile Relies on accurate user profile
Risk of being perceived as intrusive by the user
 Diversification
Cover all interpretations of the query in the first
results
Maximize probability of showing an interpretation
relevant to the user
Diversification Outliers
model Users not finding a relevant result in the top
? positions
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 6

7. Diversified Web Search Model: SoA diversification approaches
document relevance novelty
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 7

8. Diversified Web Search Model: SoA diversification approaches
document query document topic
relevance relevance novelty
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 8

9. Diversified Web Search Model: Diversification VS Personalization
baseline Diversification Personalization
(IA-Select) (BM-25)
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 9

10. Diversified Web Search Model: Diversification VS Personalization
baseline Diversification Diversified Personalization
(IA-Select) personalization (BM-25)
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 10

11. Diversified Web Search Model: Diversification VS Personalization
Small study: what do users prefer?
• If profile is perfectly (explicitly) defined
 personalization over any diversification
• If profile has errors
 personalized diversity over full
personalization or diversification
baseline Diversification diversified Personalization
(IA-Select) personalization (BM-25)
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 11

12. Personalized Diversification of Web Search: Personalized IA-Select
 IA-SELECT document relevance novelty
 Personalized IA-SELECT
– Adding a user component
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 12

13. Personalized Diversification of Web Search: Personalized xQuAD
document query document topic
 xQuAD relevance relevance novelty
 Personalized xQuAD
– Adding a user component
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 13

14. Personalized Diversification of Web Search: Model
 Personalized IA-SELECT
 Personalized xQuAD
 Estimations
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 14

15. Personalized Diversification of Web Search: Model estimations
delicious
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 15

16. Personalized Diversification of Web Search: Model estimations
1 http://www.textwise.com
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 16

17. How to evaluate?
delicious Evaluation topic
?
 Accuracy assessments  Diversity assessments
– Document relevance assessments – Document/aspect assessments
 Crowdsourced evaluation
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 17

18. Personalized Diversification of Web Search: Online Evaluation
Social User
Profile u Social tagging t1,d t2,d tl,d
…
• User annotations
• Search result annotations 4
t1 1 d1
2 3 d'1
Depth-5
t2 Top k Search pooling
popular tags results d2 d'2
Web Search Result reordering
…
…
…
dn d‘P’
t1 • User classification
• Search result classification
…
Document classifier c1,d c2,d cc,d
Users get to evaluate known topics
Must be done live, cannot make workers wait
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 18

19. Crowdsourced Evaluation: UI Desing
 Assessments
– Q1: user relevance
Accuracy (F-measure)
– Q2: topic relevance
– Q3: document-category classification: subjective to users, foster the reuse of
categories (avg. 5 categories per topic)
 Include a training task for workers (they appreciate it)
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 19

20. Personalized Diversification of Web Search: Crowdsourced Evaluation
 Collected data
– Period: 4 weeks
– 35 Delicious users: bookmarks, top tags, tag assignments, etc.
– 180 search topics
– Over 3800 relevance judgments
 Provided as evaluation and development dataset
– http://ir.ii.uam.es/dvallet/persdivers
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 20

21. Personalized Diversification of Web Search: Results
Accuracy metrics
Topic relevance User relevance F(Topic,User)
nDCG@5
nDCG@5
nDCG@5
P@5
P@5
P@5
Bing 0.393 0.918 0.330 0.702 0.359 0.796
IA-Select 0.363 0.911 0.294 0.664 0.325 0.768
Diversity
xQuAD 0.381 0.911 0.320 0.670 0.348 0.772
Personalization 0.388 0.933 0.357 0.746 0.372 0.829
PIA-Select 0.331 0.861 0.306 0.652 0.318 0.742
Personalized PIA-SelectBM25 0.363 0.892 0.345 0.670 0.354 0.766
Diversity PxQuAD 0.395 0.930 0.337 0.714 0.364 0.807
PxQuADBM25 0.405 0.939 0.361 0.730 0.382 0.821
Statistically significant Statistically significant
with respect to Baseline with respect to xQuAD
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 21

22. Personalized Diversification of Web Search: Results
Diversity metrics
Topic relevance User relevance
α-nDCG@5
α-nDCG@5
ERR-IA@5
ERR-IA@5
S-recall@5
S-recall@5
Bing 0.274 0.787 0.500 0.254 0.626 0.475
Diversity IA-Select 0.262 0.758 0.463 0.237 0.582 0.426
xQuAD 0.275 0.793 0.510 0.257 0.624 0.478
Personalization 0.267 0.778 0.486 0.262 0.646 0.483
PIA-Select 0.251 0.742 0.489 0.239 0.593 0.481
Personalized PIA-SelectBM25 0.279 0.803 0.543 0.273 0.656 0.521
Diversity PxQuAD 0.275 0.796 0.503 0.268 0.649 0.489
PxQuADBM25 0.281 0.810 0.519 0.267 0.658 0.496
Statistically significant Statistically significant
with respect to Baseline with respect to xQuAD
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 22

23. Conclusions
 Adapted SoA diversification techniques to include a personalized factor
 Presented an example of personalized diversification of Web search
– Personalization: social tagging
– Diversification: ODP Web categories
 Effectively combined both benefits of personalization and diversification
 Crowdsourced evaluation approach for diversity and personalized
techniques
– Great if you have a lot of restrictions
– Repeatable
– Cheap
– …Although can be difficult to set up
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012 23

24. Thank you!!!
Thank you!
 References
– [Vallet-10]: David Vallet, Iván Cantador, Joemon M. Jose: Personalizing
Web Search with Folksonomy-Based User and Document Profiles. ECIR
2010: 420-431
Personalized Diversification of Search Results
IRG
IR Group @ UAM
34th International ACM SIGIR Conference on Research and Development in Information Retrieval,
Portland, Oregon, 15th August 2012