Francesco Bonchi, Carlos Castillo , Debora Donato, Aris Gionis Yahoo! Research · Barcelona, Spain Query-Flow Graphs Paolo Boldi, Sebastiano Vigna Universita degli Studi · Milan, Italy

Before we begin ... Users trust in search engines We take their privacy very seriously Research datasets

Users trust in search engines

We take their privacy very seriously

Research datasets

R. Baeza-Yates: “Graphs from search engine queries”. SOFSEM 2007.

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

User session Basic concepts Radlinksi & Joachims: “Query chains: learning to rank from implicit feedback”. KDD 2005. ebay autotrader used fox vw ryanair barcelona places barcelona rent barcelona

Chain/Mission #1 Chain/Mission #2 User session Basic concepts Radlinksi & Joachims: “Query chains: learning to rank from implicit feedback”. KDD 2005. ebay autotrader used fox vw ryanair barcelona places barcelona rent barcelona

Whence come the chains? ... ebay autotrader used fox vw barcelona places barcelona rent barcelona barcelona soccer barcelona fc soccer

Whence come the chains? Query-flow graph ... ebay autotrader used fox vw barcelona places barcelona rent barcelona barcelona soccer barcelona fc soccer

Query-Flow Graph P. Boldi, F. Bonchi, C. Castillo, D. Donato, A. Gionis, S. Vigna: “The Query-Flow Graph”. CIKM 2008. ebay autotrader used fox vw barcelona hotel barcelona rent barcelona barcelona soccer barcelona fc soccer s t ... ...

The query-flow graph Directed graph Built from a query log Nodes are queries

Directed graph

Built from a query log

Nodes are queries

Frequencies of query transitions

Frequencies of query transitions

Transitions are non-symmetrical 93% of the time 93% of the time 7% of the time And when it happens, the frequencies are not correlated barclona barcelona barcelona rent barcelona

The query-flow graph Directed graph Built from sessions/chains Nodes are queries Edges hold information, e.g.: Aggregate features from query transitions Weights derived from those features

Directed graph

Built from sessions/chains

Nodes are queries

Edges hold information, e.g.:

Aggregate features from query transitions

Weights derived from those features

Statistics on query transitions Data for (q,q') Frequency Relative to other transitions from q Bags of terms and character n-grams Cosine similarity, Jaccard coefficient, ... Average time Average position in session ... Aggregate multiple users R. Jones & K. Klinkner “Beyond session timeout”. CIKM 2008. Radlinksi & Joachims: “Query chains: learning to rank from implicit feedback”. KDD 2005. barcelona places barcelona rent barcelona

Frequency

Relative to other transitions from q

Bags of terms and character n-grams

Cosine similarity, Jaccard coefficient, ...

Average time

Average position in session

...

Our query-flow graph Directed graph Built from sessions/chains Nodes are queries Edges hold information, e.g.: Aggregate features from query transitions Weights derived from those features

Directed graph

Built from sessions/chains

Nodes are queries

Edges hold information, e.g.:

Aggregate features from query transitions

Weights derived from those features

Weighting schemes Interested in weights such as: w(q,q') = freq(q,q') / freq(q) -or- w(q,q') = chain(q,q')

Interested in weights such as:

w(q,q') = freq(q,q') / freq(q)

-or-

w(q,q') = chain(q,q')

Weighting schemes Interested in weights such as: w(q,q') = freq(q,q') / freq(q) -or- w(q,q') = chain(q,q') freq(q,q') is the frequency of the transition chain(q,q') is the chaining probability Pr(q, q' in same chain | q, q' in same session)

Interested in weights such as:

w(q,q') = freq(q,q') / freq(q)

-or-

w(q,q') = chain(q,q')

freq(q,q') is the frequency of the transition

chain(q,q') is the chaining probability

Pr(q, q' in same chain | q, q' in same session)

Example with observed transition probabilities

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

Weighting schemes Interested in weights such as: w(q,q') = freq(q,q') / freq(q) -or- w(q,q') = chain(q,q') freq(q,q') is the frequency of the transition chain(q,q') is the chaining probability Pr(q, q' in same chain | q, q' in same session)

Interested in weights such as:

w(q,q') = freq(q,q') / freq(q)

-or-

w(q,q') = chain(q,q')

freq(q,q') is the frequency of the transition

chain(q,q') is the chaining probability

Pr(q, q' in same chain | q, q' in same session)

Estimating chain(q,q') Data for (q,q') Frequency N-grams and terms similarity Average delta time Average position in session Clicks between q and q' ... R. Jones & K. Klinkner “Beyond session timeout”. CIKM 2008. barcelona places barcelona rent barcelona

Frequency

N-grams and terms similarity

Average delta time

Average position in session

Clicks between q and q'

...

Estimating chain(q,q') Data for (q,q') Frequency N-grams and terms similarity Average delta time Average position in session Clicks between q and q' ... Training labels 5,000 consecutive query pairs Manually labelled 2/3 are same-chain + barcelona places barcelona rent barcelona

Frequency

N-grams and terms similarity

Average delta time

Average position in session

Clicks between q and q'

...

5,000 consecutive query pairs

Manually labelled

2/3 are same-chain

Estimating chain(q,q') Data for (q,q') Frequency N-grams and terms similarity Average delta time Average position in session Clicks between q and q' ... freq(q,q') = 1: different-chain if α time - β sim.ngrams.jaccard - γ sim.terms.jaccard + δ > θ same-chain otherwise freq(q,q') > 1: C5.0 rules, 8 rules in total + Training labels barcelona places barcelona rent barcelona

Frequency

N-grams and terms similarity

Average delta time

Average position in session

Clicks between q and q'

...

There are intertwined chains (photos: commons.wikimedia.org) pointui forum audi ipswich golfers elbow cox ipswich

Efficient session breaking 1. sort the session 2. break into contiguous chains

1. sort the session

2. break into contiguous chains

Sorting: ATSP formulation pointui forum audi ipswich golfers elbow cox ipswich

ATSP formulation pointui forum audi ipswich golfers elbow cox ipswich

ATSP formulation Weights given by chain(q,q') pointui forum audi ipswich golfers elbow cox ipswich

ATSP solution Maximize Π chain(q i ,q i+1 ) s.t. TSP constraints Greedy algorithm with d-steps lookahead pointui forum audi ipswich golfers elbow cox ipswich

ATSP solution pointui forum audi ipswich golfers elbow cox ipswich

Breaking using a threshold pointui forum audi ipswich golfers elbow cox ipswich

Breaking using a threshold Chain #1 Chain #2 pointui forum audi ipswich golfers elbow cox ipswich

Testing set Queries from 1,458 users from UK Collected queries over 3 months Selected users with >1 query Partitioned into chains manually 415 (28%) were single-chain 3.6 chains per user on average

Queries from 1,458 users from UK

Collected queries over 3 months

Selected users with >1 query

Partitioned into chains manually

415 (28%) were single-chain

3.6 chains per user on average

Evaluation wrt ground truth Sorting strategy Rand index of omniscient breaking strategy Breaking strategy Rand index Rand index = pair-wise agreements / ( ) n 2

Sorting strategy

Rand index of omniscient breaking strategy

Breaking strategy

Rand index

Rand index = pair-wise agreements / ( )

Evaluation of reordering strategy Average over all users

Average over all users

Evaluation of reordering + breaking Baseline: 30 min. timeout

Baseline: 30 min. timeout

Evaluation (cont.) Baseline: 30' timeout Ours Better rand index for the “difficult” chains. Ours and baseline score 0.85 and 0.71 if we exclude the cases where baseline has score of 1

Session segmentation remarks Simple time-outs OK ... but we can do much better Global information in the query-flow graph can help

Simple time-outs OK ... but we can do much better

Global information in the query-flow graph can help

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

Rieh, S. Y. and Xie, H: “Analysis of multiple query reformulations on the web”. IPM 32 (3) 2006. Query-reformulation types barcelona cheap barcelona hotels barcelona hotels luxury barcelona hotels Specialize Generalize brcelona Correct barcelona f.c. Generalize Parallel move Specialize Specialize

Reformulation types Correction startford cinema -> stratford cinema Generalization (“zoom out”) barcelona hotels -> barcelona Specialization (“zoom in”) barcelona soccer -> barcelona camp nou Zoom-in, zoom-out, pan, names comes from Y!SAMA Rieh and Xie: “Analysis of multiple query reformulations”. IPM 2006.

Correction

startford cinema -> stratford cinema

Generalization (“zoom out”)

barcelona hotels -> barcelona

Specialization (“zoom in”)

barcelona soccer -> barcelona camp nou

Reformulation types Rephrasing wikipedia english -> english wikipedia robbs celebrities -> robbs celebs Parallel move barcelona -> rome Rieh and Xie: “Analysis of multiple query reformulations”. IPM 2006.

Rephrasing

wikipedia english -> english wikipedia

robbs celebrities -> robbs celebs

Parallel move

barcelona -> rome

P. Boldi, F. Bonchi, C. Castillo, S. Vigna: “From 'dango' to 'japanese cakes'”. 2009. G S C P

P. Boldi, F. Bonchi, C. Castillo, S. Vigna: “From 'dango' to 'japanese cakes'”. 2009. Query-reformulation classifier

Classifier: 92% accuracy

Example 1/4: “ cat ”

Example 2/4: “ peanut ”

Example 3/4: “ surf board ”

Example 4/4: “ bruce springsteen ”

Finding query-reformulation patterns Sessions can be seen as strings of QRTs: XSSX XCGSSX XCSGSGSGPX Patterns can be identified in these strings Data: UK and US query logs, early 2008

Sessions can be seen as strings of QRTs:

XSSX

XCGSSX

XCSGSGSGPX

Patterns can be identified in these strings

Data: UK and US query logs, early 2008

Chain length distribution

QRT distribution

Interesting patterns

General observations on QRTs Parallel moves (50%-60%) The most frequent class Specializations (30%-40%) Generalizations (5%-10%) Frequently appear together in alternating order Corrections (5%-10%) More frequent at the beginning or end of a chain

Parallel moves (50%-60%)

The most frequent class

Specializations (30%-40%)

Generalizations (5%-10%)

Frequently appear together in alternating order

Corrections (5%-10%)

More frequent at the beginning or end of a chain

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

Query recommendation setting Dataset: “Spring 2006 Microsoft data” Provided by WSCD'09 workshop organizers 15M queries from a 1-month period Automatic chains and reformulations labels Obtained with models from previous works Recommendations based on random walks WebGraph (graph) SUX4J (hashing)

Dataset: “Spring 2006 Microsoft data”

Provided by WSCD'09 workshop organizers

15M queries from a 1-month period

Automatic chains and reformulations labels

Obtained with models from previous works

Recommendations based on random walks

WebGraph (graph) SUX4J (hashing)

Implemented systems Slices Queryflow-{G, S, P, C, SP, GSP, GSPC, ...} Composed graphs Queryflow-{(SS T ), (SG), ...} Weight is weight of heaviest length-2 path

Slices

Queryflow-{G, S, P, C, SP, GSP, GSPC, ...}

Composed graphs

Queryflow-{(SS T ), (SG), ...}

Weight is weight of heaviest length-2 path

Random walks Number of steps: 1, 5, 10 (2, 6, 12) Scoring: PPR ( Absolute ) or PPR/PR ( Relative ) Self-transition with prob. 0.9, no random jumps

Number of steps: 1, 5, 10 (2, 6, 12)

Scoring: PPR ( Absolute ) or PPR/PR ( Relative )

Self-transition with prob. 0.9, no random jumps

Click graphs Weights based on clicks Two weighting schemes: N. Craswell and M. Szummer: “Random walks on the click graph” SIGIR 2007. d 1 d 2 d 3 q 1 q 2 d 4 d 5 q 3 q 4

Example: “ sudoku ” sudoku free sudoku sudoku shack daily sudoku msn sudoku rci sudoku sudoku #5 easy brainy puzzle sudoku game SuDoKu How to play Sudoku QueryFlow-S, 10, Abs. QueryDocument-Bwd, 6, Rel.

sudoku

free sudoku

sudoku shack

daily sudoku

msn sudoku

rci

sudoku

sudoku #5 easy

brainy puzzle

sudoku game

SuDoKu

How to play Sudoku

Example: “ myspace layouts ” myspace layouts free myspace layouts myspace contact tables myspace codes premade myspace layouts myspace icons myspace layouts myspace codes + hide last login myspace and layouts dissect virtual frogs Myspace Editors Cars MYSPACE GENERATOR QueryFlow-S, 10, Abs. QueryDocument-Bwd, 6, Rel.

myspace layouts

free myspace layouts

myspace contact tables

myspace codes

premade myspace layouts

myspace icons

myspace layouts

myspace codes + hide last login

myspace and layouts

dissect virtual frogs

Myspace Editors Cars

MYSPACE GENERATOR

Evaluation method Sampled 114 queries (freq 700 - 15,000) Run all systems for each query Create a pool: union of top-5 recommendations ~ 6,000 query recommendations in total Evaluate each recommendation in the pool Useful, somewhat useful, not useful

Sampled 114 queries (freq 700 - 15,000)

Run all systems for each query

Create a pool: union of top-5 recommendations

~ 6,000 query recommendations in total

Evaluate each recommendation in the pool

Useful, somewhat useful, not useful

Assessment Example, query “ cnn news ”:

Example, query “ cnn news ”:

Distribution of assessments n = 6 093

Inter-assessor agreement n = 560

Inter-assessor agreement n = 560 Good Bad

Precision @ 5 P. Boldi, F. Bonchi, C. Castillo, D. Donato, S. Vigna: “From 'dango' to 'japanese cakes'”. WSCD 2009.

Precision @ 5

Diversity score Take one query Get top-5 recommendations that are good Get top-5 search results for each of them Count how many distinct URLs appear Discard URLs from initial query, this is in [0,25]

Take one query

Get top-5 recommendations that are good

Get top-5 search results for each of them

Count how many distinct URLs appear

Discard URLs from initial query, this is in [0,25]

Diversity scores

1. Building a query-flow graph 2. Application to session segmentation 3. Query-reformulation types 4. Application to query recommendation 5. Conclusions

Conclusions (1/3) Query reformulation types Classification can be improved using the graph itself Learn topic simultaneously Parallel moves are interesting, even if risky Refine the categories depending on the application

Query reformulation types

Classification can be improved using the graph itself

Learn topic simultaneously

Parallel moves are interesting, even if risky

Refine the categories depending on the application

Conclusions (2/3) Query recommendations Recognizing chains is important Fewer slices is better than more Few iterations are enough Why 5? Let the system decide how many...

Query recommendations

Recognizing chains is important

Fewer slices is better than more

Few iterations are enough

Why 5? Let the system decide how many...

Conclusions (3/3) (history-based recommendation)

Q&A barcelona cheap barcelona hotels barcelona hotels luxury barcelona hotels Specialize Generalize brcelona Correct barcelona f.c. Generalize Parallel move Specialize Specialize