From Dango to Japanese Cakes: Query Reformulation Models and Patterns

Francesco Bonchi, Carlos Castillo Yahoo! Research · Barcelona, Spain From “Dango” to “Japanese Cakes” Query Reformulation Models and Patterns Paolo Boldi, Sebastiano Vigna Università degli Studi · Milan, Italy

(Physical) User session Basic concepts
A query log is a collection of user sessions
More information:
timestamps
views
clicks
ebay autotrader brcelona barcelona barcelona f.c. barcelona hotels

(Physical) User session Basic concepts Chain #1 Chain #2
For more information on how sessions can be segmented into chains (even intertwined), see:
A. Gionis, P. Boldi, F. Bonchi, C. Castillo, D. Donato, S. Vigna: Query-flow graphs (CIKM 2008)
ebay autotrader brcelona barcelona barcelona f.c. barcelona hotels

User chain Basic concepts brcelona barcelona barcelona f.c. barcelona hotels

Basic concepts
Understanding query-reformulation type is the main goal of this paper
User chain brcelona barcelona barcelona f.c. barcelona hotels correction specialization parallel move

Related works
Lau & Horvitz, Patterns of search: analyzing and modeling web query reﬁnement . UM'99
Rieh & Xie, Analysis of multiple query reformulations on the web: the interactive information retrieval context. Inf. Process. Manage. 2006
Jansen et al., Query modiﬁcations patterns during web searching , ITNG'07

1. Query-reformulation types 2. Building a classifier 3. Patterns 4. An application to query suggestion 5. Conclusions

Rieh, S. Y. and Xie, H: “Analysis of multiple query reformulations on the web”. IPM 32 (3) 2006. Query-reformulation types barcelona cheap barcelona hotels barcelona hotels luxury barcelona hotels Specialize Generalize brcelona Correct barcelona f.c. Generalize Parallel move Specialize Specialize

Reformulation types
Correction
startford cinema -> stratford cinema
Generalization (“zoom out”)
barcelona hotels -> barcelona
Specialization (“zoom in”)
barcelona soccer -> barcelona camp nou
Zoom-in, zoom-out, pan, names comes from Y!SAMA Rieh and Xie: “Analysis of multiple query reformulations”. IPM 2006.

Reformulation types
Rephrasing
wikipedia english -> english wikipedia
robbs celebrities -> robbs celebs
Parallel move
barcelona -> rome
Rieh and Xie: “Analysis of multiple query reformulations”. IPM 2006.

G S C P

Query-reformulation classifier

Classifier: 92% accuracy

Example 1/4: “cat”

Example 2/4: “peanut”

Example 3/4: “surf board”

Example 4/4: “bruce springsteen”

Finding query-reformulation patterns
Sessions can be seen as strings of QRTs:
XSSX
XCGSSX
XCSGSGSGPX
Patterns can be identified in these strings
Data: UK and US query logs, early 2008

Chain length distribution

QRT distribution

Interesting patterns

General observations on QRTs
Parallel moves (50%-60%)
The most frequent class
Specializations (30%-40%)
Generalizations (5%-10%)
Frequently appear together in alternating order
Corrections (5%-10%)
More frequent at the beginning or end of a chain

Query recommendation setting
Dataset: “Spring 2006 Microsoft data”
Provided by WSCD'09 workshop organizers; 15M queries from a 1-month period
Query-flow weighted graph + Reformulations types
(see “ Query-flow graphs ”, CIKM 2008)

Implemented systems
Slices
Queryflow-{G, S, P, C, SP, GSP, GSPC, ...}
Composed graphs
Queryflow-{(SS T ), (SG), ...}
Weight is weight of heaviest length-2 path
Recommendations based on random walks
WebGraph (graph) SUX4J (hashing)

Random walks
Number of steps: 1, 5, 10 (2, 6, 12)
Scoring: PPR ( Absolute ) or PPR/PR ( Relative )
Self-transition with prob. 0.9, no random jumps

For comparison: click graphs
Weights based on clicks
Two weighting schemes:
N. Craswell and M. Szummer: “Random walks on the click graph” SIGIR 2007. d 1 d 2 d 3 q 1 q 2 d 4 d 5 q 3 q 4

Example: “sudoku”
sudoku
free sudoku
sudoku shack
daily sudoku
msn sudoku
rci
sudoku
sudoku #5 easy
brainy puzzle
sudoku game
SuDoKu
How to play Sudoku
QueryFlow-S, 10, Abs. QueryDocument-Bwd, 6, Rel.

Example: “myspace layouts”
myspace layouts
free myspace layouts
myspace contact tables
myspace codes
premade myspace layouts
myspace icons
myspace layouts
myspace codes + hide last login
myspace and layouts
dissect virtual frogs
Myspace Editors Cars
MYSPACE GENERATOR
QueryFlow-S, 10, Abs. QueryDocument-Bwd, 6, Rel.

Evaluation method
Sampled 114 queries (freq 700 - 15,000)
Run all systems for each query
Create a pool: union of top-5 recommendations
~ 6,000 query recommendations in total
Evaluate each recommendation in the pool
Useful, somewhat useful, not useful

Assessment
Example, query “cnn news”:

Distribution of assessments n = 6 093

Inter-assessor agreement n = 560

Inter-assessor agreement n = 560 Good Bad

Precision @ 5

Diversity score
Take one query
Get top-5 recommendations that are good
Get top-5 search results for each of them
Count how many distinct URLs appear
Discard URLs from initial query, this is in [0,25]

Diversity scores

Conclusions (1/3)
A definition of query reformulation types
Refine the categories depending on the application
An accurate classifier
Interesting typical patterns

Conclusions (2/3)
Query reformulation types are useful
Fewer slices are better
Even more useful than clicks!
Few iterations are enough

Conclusions (3/3) (history-based suggestions)

Q&A barcelona cheap barcelona hotels barcelona hotels luxury barcelona hotels Specialize Generalize brcelona Correct barcelona f.c. Generalize Parallel move Specialize Specialize

That's all folks

User session Basic concepts Radlinksi & Joachims: “Query chains: learning to rank from implicit feedback”. KDD 2005. ebay autotrader used fox vw ryanair barcelona places barcelona rent barcelona Chain/Mission #1 Chain/Mission #2

Whence come the chains? ... ebay autotrader used fox vw barcelona places barcelona rent barcelona barcelona soccer barcelona fc soccer

Whence come the chains? Query-flow graph ... ebay autotrader used fox vw barcelona places barcelona rent barcelona barcelona soccer barcelona fc soccer

Query-Flow Graph P. Boldi, F. Bonchi, C. Castillo, D. Donato, A. Gionis, S. Vigna: “The Query-Flow Graph”. CIKM 2008. ebay autotrader used fox vw barcelona hotel barcelona rent barcelona barcelona soccer barcelona fc soccer s t ... ...

The query-flow graph
Directed graph
Built from a query log
Nodes are queries

Frequencies of query transitions

Transitions are non-symmetrical 93% of the time 93% of the time 7% of the time And when it happens, the frequencies are not correlated barclona barcelona barcelona rent barcelona

The query-flow graph
Directed graph
Built from sessions/chains
Nodes are queries
Edges hold information, e.g.:
Aggregate features from query transitions
Weights derived from those features

Statistics on query transitions
Data for (q,q')
Frequency
Relative to other transitions from q
Bags of terms and character n-grams
Cosine similarity, Jaccard coefficient, ...
Average time
Average position in session
...
Aggregate multiple users R. Jones & K. Klinkner “Beyond session timeout”. CIKM 2008. Radlinksi & Joachims: “Query chains: learning to rank from implicit feedback”. KDD 2005. barcelona places barcelona rent barcelona

Our query-flow graph
Directed graph
Built from sessions/chains
Nodes are queries
Edges hold information, e.g.:
Aggregate features from query transitions
Weights derived from those features

Weighting schemes
Interested in weights such as:
w(q,q') = freq(q,q') / freq(q)
-or-
w(q,q') = chain(q,q')

Weighting schemes
-or-
freq(q,q') is the frequency of the transition
chain(q,q') is the chaining probability
Pr(q, q' in same chain | q, q' in same session)

Example with observed transition probabilities

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

Weighting schemes
-or-
freq(q,q') is the frequency of the transition
chain(q,q') is the chaining probability
Pr(q, q' in same chain | q, q' in same session)

Estimating chain(q,q')
Data for (q,q')
Frequency
N-grams and terms similarity
Average delta time
Clicks between q and q'
...
R. Jones & K. Klinkner “Beyond session timeout”. CIKM 2008. barcelona places barcelona rent barcelona

Data for (q,q')
Frequency
Average delta time
...
Training labels
5,000 consecutive query pairs
Manually labelled
2/3 are same-chain
+
barcelona places barcelona rent barcelona

Data for (q,q')
Frequency
Average delta time
...
freq(q,q') = 1: different-chain if α time - β sim.ngrams.jaccard - γ sim.terms.jaccard + δ > θ same-chain otherwise freq(q,q') > 1: C5.0 rules, 8 rules in total
+ Training labels
barcelona places barcelona rent barcelona

There are intertwined chains (photos: commons.wikimedia.org) pointui forum audi ipswich golfers elbow cox ipswich

Efficient session breaking
1. sort the session
2. break into contiguous chains

Sorting: ATSP formulation pointui forum audi ipswich golfers elbow cox ipswich

ATSP formulation pointui forum audi ipswich golfers elbow cox ipswich

ATSP formulation Weights given by chain(q,q') pointui forum audi ipswich golfers elbow cox ipswich

ATSP solution Maximize Π chain(q i ,q i+1 ) s.t. TSP constraints Greedy algorithm with d-steps lookahead pointui forum audi ipswich golfers elbow cox ipswich

ATSP solution pointui forum audi ipswich golfers elbow cox ipswich

Breaking using a threshold pointui forum audi ipswich golfers elbow cox ipswich

Breaking using a threshold Chain #1 Chain #2 pointui forum audi ipswich golfers elbow cox ipswich

Testing set
Queries from 1,458 users from UK
Collected queries over 3 months
Selected users with >1 query
Partitioned into chains manually
415 (28%) were single-chain
3.6 chains per user on average

Evaluation wrt ground truth
Sorting strategy
Rand index of omniscient breaking strategy
Breaking strategy
Rand index
Rand index = pair-wise agreements / ( )
n 2

Evaluation of reordering strategy
Average over all users

Evaluation of reordering + breaking
Baseline: 30 min. timeout

Evaluation (cont.) Baseline: 30' timeout Ours Better rand index for the “difficult” chains. Ours and baseline score 0.85 and 0.71 if we exclude the cases where baseline has score of 1

Session segmentation remarks
Simple time-outs OK ... but we can do much better
Global information in the query-flow graph can help

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

(Physical) User session Basic concepts ebay autotrader brcelona barcelona barcelona f.c. barcelona hotels

http://freesearch.pe.kr	89
http://www.freesearch.pe.kr	8
http://blog.daum.net	8
http://www.hanrss.com	7
http://cafe391.daum.net	3
http://emti.net	1
http://gogamza.mireene.co.kr	1
http://www.slideshare.net	1

From Dango to Japanese Cakes: Query Reformulation Models and Patterns

by Carlos Castillo on Sep 18, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

8 Embeds 118

Statistics

From Dango to Japanese Cakes: Query Reformulation Models and Patterns — Presentation Transcript