Francesco Bonchi, Carlos Castillo Yahoo! Research · Barcelona, Spain From “Dango” to “Japanese Cakes” Query Reformulation Models and Patterns Paolo Boldi, Sebastiano Vigna Università degli Studi · Milan, Italy

(Physical) User session Basic concepts A query log is a collection of user sessions More information: timestamps views clicks ebay autotrader brcelona barcelona barcelona f.c. barcelona hotels

A query log is a collection of user sessions

More information:

timestamps

views

clicks

(Physical) User session Basic concepts Chain #1 Chain #2 For more information on how sessions can be segmented into chains (even intertwined), see: A. Gionis, P. Boldi, F. Bonchi, C. Castillo, D. Donato, S. Vigna: Query-flow graphs (CIKM 2008) ebay autotrader brcelona barcelona barcelona f.c. barcelona hotels

For more information on how sessions can be segmented into chains (even intertwined), see:

A. Gionis, P. Boldi, F. Bonchi, C. Castillo, D. Donato, S. Vigna: Query-flow graphs (CIKM 2008)

User chain Basic concepts brcelona barcelona barcelona f.c. barcelona hotels

Basic concepts Understanding query-reformulation type is the main goal of this paper User chain brcelona barcelona barcelona f.c. barcelona hotels correction specialization parallel move

Understanding query-reformulation type is the main goal of this paper

Related works Lau & Horvitz, Patterns of search: analyzing and modeling web query refinement . UM'99 Rieh & Xie, Analysis of multiple query reformulations on the web: the interactive information retrieval context. Inf. Process. Manage. 2006 Jansen et al., Query modifications patterns during web searching , ITNG'07

Lau & Horvitz, Patterns of search: analyzing and modeling web query refinement . UM'99

Rieh & Xie, Analysis of multiple query reformulations on the web: the interactive information retrieval context. Inf. Process. Manage. 2006

Jansen et al., Query modifications patterns during web searching , ITNG'07

1. Query-reformulation types 2. Building a classifier 3. Patterns 4. An application to query suggestion 5. Conclusions

1. Query-reformulation types 2. Building a classifier 3. Patterns 4. An application to query suggestion 5. Conclusions

Rieh, S. Y. and Xie, H: “Analysis of multiple query reformulations on the web”. IPM 32 (3) 2006. Query-reformulation types barcelona cheap barcelona hotels barcelona hotels luxury barcelona hotels Specialize Generalize brcelona Correct barcelona f.c. Generalize Parallel move Specialize Specialize

Reformulation types Correction startford cinema -> stratford cinema Generalization (“zoom out”) barcelona hotels -> barcelona Specialization (“zoom in”) barcelona soccer -> barcelona camp nou Zoom-in, zoom-out, pan, names comes from Y!SAMA Rieh and Xie: “Analysis of multiple query reformulations”. IPM 2006.

Correction

startford cinema -> stratford cinema

Generalization (“zoom out”)

barcelona hotels -> barcelona

Specialization (“zoom in”)

barcelona soccer -> barcelona camp nou

Reformulation types Rephrasing wikipedia english -> english wikipedia robbs celebrities -> robbs celebs Parallel move barcelona -> rome Rieh and Xie: “Analysis of multiple query reformulations”. IPM 2006.

Rephrasing

wikipedia english -> english wikipedia

robbs celebrities -> robbs celebs

Parallel move

barcelona -> rome

G S C P

1. Query-reformulation types 2. Building a classifier 3. Patterns 4. An application to query suggestion 5. Conclusions

Query-reformulation classifier

Classifier: 92% accuracy

Example 1/4: “cat”

Example 2/4: “peanut”

Example 3/4: “surf board”

Example 4/4: “bruce springsteen”

1. Query-reformulation types 2. Building a classifier 3. Patterns 4. An application to query suggestion 5. Conclusions

Finding query-reformulation patterns Sessions can be seen as strings of QRTs: XSSX XCGSSX XCSGSGSGPX Patterns can be identified in these strings Data: UK and US query logs, early 2008

Sessions can be seen as strings of QRTs:

XSSX

XCGSSX

XCSGSGSGPX

Patterns can be identified in these strings

Data: UK and US query logs, early 2008

Chain length distribution

QRT distribution

Interesting patterns

General observations on QRTs Parallel moves (50%-60%) The most frequent class Specializations (30%-40%) Generalizations (5%-10%) Frequently appear together in alternating order Corrections (5%-10%) More frequent at the beginning or end of a chain

Parallel moves (50%-60%)

The most frequent class

Specializations (30%-40%)

Generalizations (5%-10%)

Frequently appear together in alternating order

Corrections (5%-10%)

More frequent at the beginning or end of a chain

1. Query-reformulation types 2. Building a classifier 3. Patterns 4. An application to query suggestion 5. Conclusions

Query recommendation setting Dataset: “Spring 2006 Microsoft data” Provided by WSCD'09 workshop organizers; 15M queries from a 1-month period Query-flow weighted graph + Reformulations types (see “ Query-flow graphs ”, CIKM 2008)

Dataset: “Spring 2006 Microsoft data”

Provided by WSCD'09 workshop organizers; 15M queries from a 1-month period

Query-flow weighted graph + Reformulations types

(see “ Query-flow graphs ”, CIKM 2008)

Implemented systems Slices Queryflow-{G, S, P, C, SP, GSP, GSPC, ...} Composed graphs Queryflow-{(SS T ), (SG), ...} Weight is weight of heaviest length-2 path Recommendations based on random walks WebGraph (graph) SUX4J (hashing)

Slices

Queryflow-{G, S, P, C, SP, GSP, GSPC, ...}

Composed graphs

Queryflow-{(SS T ), (SG), ...}

Weight is weight of heaviest length-2 path

Recommendations based on random walks

WebGraph (graph) SUX4J (hashing)

Random walks Number of steps: 1, 5, 10 (2, 6, 12) Scoring: PPR ( Absolute ) or PPR/PR ( Relative ) Self-transition with prob. 0.9, no random jumps

Number of steps: 1, 5, 10 (2, 6, 12)

Scoring: PPR ( Absolute ) or PPR/PR ( Relative )

Self-transition with prob. 0.9, no random jumps

For comparison: click graphs Weights based on clicks Two weighting schemes: N. Craswell and M. Szummer: “Random walks on the click graph” SIGIR 2007. d 1 d 2 d 3 q 1 q 2 d 4 d 5 q 3 q 4

Weights based on clicks

Two weighting schemes:

Example: “sudoku” sudoku free sudoku sudoku shack daily sudoku msn sudoku rci sudoku sudoku #5 easy brainy puzzle sudoku game SuDoKu How to play Sudoku QueryFlow-S, 10, Abs. QueryDocument-Bwd, 6, Rel.

sudoku

free sudoku

sudoku shack

daily sudoku

msn sudoku

rci

sudoku

sudoku #5 easy

brainy puzzle

sudoku game

SuDoKu

How to play Sudoku

Example: “myspace layouts” myspace layouts free myspace layouts myspace contact tables myspace codes premade myspace layouts myspace icons myspace layouts myspace codes + hide last login myspace and layouts dissect virtual frogs Myspace Editors Cars MYSPACE GENERATOR QueryFlow-S, 10, Abs. QueryDocument-Bwd, 6, Rel.

myspace layouts

free myspace layouts

myspace contact tables

myspace codes

premade myspace layouts

myspace icons

myspace layouts

myspace codes + hide last login

myspace and layouts

dissect virtual frogs

Myspace Editors Cars

MYSPACE GENERATOR

Evaluation method Sampled 114 queries (freq 700 - 15,000) Run all systems for each query Create a pool: union of top-5 recommendations ~ 6,000 query recommendations in total Evaluate each recommendation in the pool Useful, somewhat useful, not useful

Sampled 114 queries (freq 700 - 15,000)

Run all systems for each query

Create a pool: union of top-5 recommendations

~ 6,000 query recommendations in total

Evaluate each recommendation in the pool

Useful, somewhat useful, not useful

Assessment Example, query “cnn news”:

Example, query “cnn news”:

Distribution of assessments n = 6 093

Inter-assessor agreement n = 560

Inter-assessor agreement n = 560 Good Bad

Precision @ 5

Precision @ 5

Diversity score Take one query Get top-5 recommendations that are good Get top-5 search results for each of them Count how many distinct URLs appear Discard URLs from initial query, this is in [0,25]

Take one query

Get top-5 recommendations that are good

Get top-5 search results for each of them

Count how many distinct URLs appear

Discard URLs from initial query, this is in [0,25]

Diversity scores

1. Query-reformulation types 2. Building a classifier 3. Patterns 4. An application to query suggestion 5. Conclusions

Conclusions (1/3) A definition of query reformulation types Refine the categories depending on the application An accurate classifier Interesting typical patterns

A definition of query reformulation types

Refine the categories depending on the application

An accurate classifier

Interesting typical patterns

Conclusions (2/3) Query reformulation types are useful Fewer slices are better Even more useful than clicks! Few iterations are enough

Query reformulation types are useful

Fewer slices are better

Even more useful than clicks!

Few iterations are enough

Conclusions (3/3) (history-based suggestions)

Q&A barcelona cheap barcelona hotels barcelona hotels luxury barcelona hotels Specialize Generalize brcelona Correct barcelona f.c. Generalize Parallel move Specialize Specialize

That's all folks

User session Basic concepts Radlinksi & Joachims: “Query chains: learning to rank from implicit feedback”. KDD 2005. ebay autotrader used fox vw ryanair barcelona places barcelona rent barcelona Chain/Mission #1 Chain/Mission #2

Whence come the chains? ... ebay autotrader used fox vw barcelona places barcelona rent barcelona barcelona soccer barcelona fc soccer

Whence come the chains? Query-flow graph ... ebay autotrader used fox vw barcelona places barcelona rent barcelona barcelona soccer barcelona fc soccer

Query-Flow Graph P. Boldi, F. Bonchi, C. Castillo, D. Donato, A. Gionis, S. Vigna: “The Query-Flow Graph”. CIKM 2008. ebay autotrader used fox vw barcelona hotel barcelona rent barcelona barcelona soccer barcelona fc soccer s t ... ...

The query-flow graph Directed graph Built from a query log Nodes are queries

Directed graph

Built from a query log

Nodes are queries

Frequencies of query transitions

Frequencies of query transitions

Transitions are non-symmetrical 93% of the time 93% of the time 7% of the time And when it happens, the frequencies are not correlated barclona barcelona barcelona rent barcelona

The query-flow graph Directed graph Built from sessions/chains Nodes are queries Edges hold information, e.g.: Aggregate features from query transitions Weights derived from those features

Directed graph

Built from sessions/chains

Nodes are queries

Edges hold information, e.g.:

Aggregate features from query transitions

Weights derived from those features

Statistics on query transitions Data for (q,q') Frequency Relative to other transitions from q Bags of terms and character n-grams Cosine similarity, Jaccard coefficient, ... Average time Average position in session ... Aggregate multiple users R. Jones & K. Klinkner “Beyond session timeout”. CIKM 2008. Radlinksi & Joachims: “Query chains: learning to rank from implicit feedback”. KDD 2005. barcelona places barcelona rent barcelona

Data for (q,q')

Frequency

Relative to other transitions from q

Bags of terms and character n-grams

Cosine similarity, Jaccard coefficient, ...

Average time

Average position in session

...

Our query-flow graph Directed graph Built from sessions/chains Nodes are queries Edges hold information, e.g.: Aggregate features from query transitions Weights derived from those features

Directed graph

Built from sessions/chains

Nodes are queries

Edges hold information, e.g.:

Aggregate features from query transitions

Weights derived from those features

Weighting schemes Interested in weights such as: w(q,q') = freq(q,q') / freq(q) -or- w(q,q') = chain(q,q')

Interested in weights such as:

w(q,q') = freq(q,q') / freq(q)

-or-

w(q,q') = chain(q,q')

Weighting schemes Interested in weights such as: w(q,q') = freq(q,q') / freq(q) -or- w(q,q') = chain(q,q') freq(q,q') is the frequency of the transition chain(q,q') is the chaining probability Pr(q, q' in same chain | q, q' in same session)

Interested in weights such as:

w(q,q') = freq(q,q') / freq(q)

-or-

w(q,q') = chain(q,q')

freq(q,q') is the frequency of the transition

chain(q,q') is the chaining probability

Pr(q, q' in same chain | q, q' in same session)

Example with observed transition probabilities

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

Weighting schemes Interested in weights such as: w(q,q') = freq(q,q') / freq(q) -or- w(q,q') = chain(q,q') freq(q,q') is the frequency of the transition chain(q,q') is the chaining probability Pr(q, q' in same chain | q, q' in same session)

Interested in weights such as:

w(q,q') = freq(q,q') / freq(q)

-or-

w(q,q') = chain(q,q')

freq(q,q') is the frequency of the transition

chain(q,q') is the chaining probability

Pr(q, q' in same chain | q, q' in same session)

Estimating chain(q,q') Data for (q,q') Frequency N-grams and terms similarity Average delta time Average position in session Clicks between q and q' ... R. Jones & K. Klinkner “Beyond session timeout”. CIKM 2008. barcelona places barcelona rent barcelona

Data for (q,q')

Frequency

N-grams and terms similarity

Average delta time

Average position in session

Clicks between q and q'

...

Estimating chain(q,q') Data for (q,q') Frequency N-grams and terms similarity Average delta time Average position in session Clicks between q and q' ... Training labels 5,000 consecutive query pairs Manually labelled 2/3 are same-chain + barcelona places barcelona rent barcelona

Data for (q,q')

Frequency

N-grams and terms similarity

Average delta time

Average position in session

Clicks between q and q'

...

Training labels

5,000 consecutive query pairs

Manually labelled

2/3 are same-chain

+

Estimating chain(q,q') Data for (q,q') Frequency N-grams and terms similarity Average delta time Average position in session Clicks between q and q' ... freq(q,q') = 1: different-chain if α time - β sim.ngrams.jaccard - γ sim.terms.jaccard + δ > θ same-chain otherwise freq(q,q') > 1: C5.0 rules, 8 rules in total + Training labels barcelona places barcelona rent barcelona

Data for (q,q')

Frequency

N-grams and terms similarity

Average delta time

Average position in session

Clicks between q and q'

...

+ Training labels

There are intertwined chains (photos: commons.wikimedia.org) pointui forum audi ipswich golfers elbow cox ipswich

Efficient session breaking 1. sort the session 2. break into contiguous chains

1. sort the session

2. break into contiguous chains

Sorting: ATSP formulation pointui forum audi ipswich golfers elbow cox ipswich

ATSP formulation pointui forum audi ipswich golfers elbow cox ipswich

ATSP formulation Weights given by chain(q,q') pointui forum audi ipswich golfers elbow cox ipswich

ATSP solution Maximize Π chain(q i ,q i+1 ) s.t. TSP constraints Greedy algorithm with d-steps lookahead pointui forum audi ipswich golfers elbow cox ipswich

ATSP solution pointui forum audi ipswich golfers elbow cox ipswich

Breaking using a threshold pointui forum audi ipswich golfers elbow cox ipswich

Breaking using a threshold Chain #1 Chain #2 pointui forum audi ipswich golfers elbow cox ipswich

Testing set Queries from 1,458 users from UK Collected queries over 3 months Selected users with >1 query Partitioned into chains manually 415 (28%) were single-chain 3.6 chains per user on average

Queries from 1,458 users from UK

Collected queries over 3 months

Selected users with >1 query

Partitioned into chains manually

415 (28%) were single-chain

3.6 chains per user on average

Evaluation wrt ground truth Sorting strategy Rand index of omniscient breaking strategy Breaking strategy Rand index Rand index = pair-wise agreements / ( ) n 2

Sorting strategy

Rand index of omniscient breaking strategy

Breaking strategy

Rand index

Rand index = pair-wise agreements / ( )

Evaluation of reordering strategy Average over all users

Average over all users

Evaluation of reordering + breaking Baseline: 30 min. timeout

Baseline: 30 min. timeout

Evaluation (cont.) Baseline: 30' timeout Ours Better rand index for the “difficult” chains. Ours and baseline score 0.85 and 0.71 if we exclude the cases where baseline has score of 1

Session segmentation remarks Simple time-outs OK ... but we can do much better Global information in the query-flow graph can help

Simple time-outs OK ... but we can do much better

Global information in the query-flow graph can help

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

(Physical) User session Basic concepts ebay autotrader brcelona barcelona barcelona f.c. barcelona hotels