Salvatore_Orlando

Beyond Query Suggestions:
Recommending Tasks to SE Users
Claudio Lucchese*, Gabriele Tolomei*+, Salvatore Orlando+, Raffaele Perego*, Fabrizio Silvestri*
*ISTI - CNR, Pisa, Italy
+Università Ca’ Foscari Venezia, Italy

TechTalk at Moscow - August 22, 2011

C. Lucchese, S. Orlando, R. Perego, F. Silvestri, G. Tolomei. Identifying Task-based Sessions in Search Engine Query Logs. ACM WSDM, Hong Kong, February 9-12, 2011
G. Tolomei, S. Orlando, and F. Silvestri. Towards a task-based search and recommender systems. The 26th IEEE International Conference on Data Engineering, ICDE '10 Workshops, pages 333-336, 2010.
C. Lucchese, S. Orlando, R. Perego, F. Silvestri, G. Tolomei. Beyond Query Suggestions: Recommending Tasks to Search Engine Users, submitted to WSDM 2012.
mercoledì 24 agosto 2011 1

Contents

• Introduction and Motivations
• Web Task Discovery
• Web Task Recommendation
• Mining Query Logs for Web Task Discovery
• Mining Query Logs for Web Task Recommendation


Web Task Discovery
• What is a Web task?
• Any (atomic) user activity that can be achieved by exploiting the information/services available on
the Web, e.g., “find a recipe”, “book a flight”, “read news”, etc.

• Why do we use WSE Query Logs to identify Web tasks?
• “Addiction to Web search” : Users rely on WSEs for satisfying their information needs by issuing
possibly interleaved stream of related queries

• How do we discover Web tasks in query logs?

• Task-based Session Discovery
• sessions are sets of possibly non contiguous queries, issued by a user whose aim is to
carry out specific “Web tasks”

Web Task Discovery
• What is a Web task?
• Any (atomic) user activity that can be achieved by exploiting the information/services available on
the Web, e.g., “find a recipe”, “book a flight”, “read news”, etc.

• Why do we use WSE Query Logs to identify Web tasks?
• “Addiction to Web search” : Users rely on WSEs for satisfying their information needs by issuing
possibly interleaved stream of related queries

• How do we discover Web tasks in query logs? we argue that there
is task interleaving
• Task-based Session Discovery
• sessions are sets of possibly non contiguous queries, issued by a user whose aim is to
carry out specific “Web tasks”

The Big Picture of Task Discovery


query
...

hong kong
ﬂights


query
...

ﬂy to
hong kong


query
...

nba sport
news


query
...

pisa to
hong kong


long-term session
... ... ...


Δt > tφ long-term session
... ... ...

...
1 2 n



1 2 ... n



1 2 ... n

ﬂy to nba news shopping in
Hong Kong Hong Kong


Task/Query Recommendation
How to exploit Task-based Session Knowledge

• We are interested in supplying novel recommendations to WSE users,
on the basis of the knowledge of the task-based query sessions
• from intra-task query recommendation

• to task recommendation

• to ﬁnally arrive at inter-task query recommendation


How to exploit Task-based Session Knowledge

• We are interested in supplying novel recommendations to WSE users,
on the basis of the knowledge of the task-based query sessions
• from intra-task query recommendation

• to task recommendation

• to ﬁnally arrive at inter-task query recommendation

We aim to recommend another related task, where the current and
the recommended ones are part of a mission

From tasks to missions
• From Web task to Web mission
• We argue that single search tasks may be subsumed by composite tasks, namely
missions, which user aim to accomplish through a WSE

• Example
• Alice starts interacting with her favorite WSE by submitting queries we recognize
as part of the same task, related to “booking of a hotel room in New York”

• The current Alice's task could be included in a mission, composed of more tasks,
concerned with “planning a travel to New York”

R. Jones and K.L. Klinkner. 2008. Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs. In CIKM ’08. ACM, 699–708.


From intra- to inter-task query suggestions
• Assume we recognize that Alice is performing a query task (several queries)
related to “booking of a hotel room in New York”
• query suggestion mechanisms may provide alternative related queries (intra-task query
suggestions), such as “cheap new york hotels”, “times square hotel”, “waldorf astoria”,
etc. this is similar to current query suggestion mechanisms

• Assume we discover that many users performed a task similar to the Alice's
one, along with other tasks ➠ they could be part of a composite mission
• e.g., a mission that is concerned with “planning a travel to New York”

• we may also recommend to Alice other tasks, whose underpinning queries look like:
“mta subway”, or “broadway shows”, or “JFK airport shuttle” (inter-task query
suggestions)

Task Discovery


Data Set: AOL Query Log
✓ 3-months collection
Original Data Set ✓ ~20M queries
✓ ~657K users

✓ 1-week collection
✓ ~100K queries
✓ 1,000 users
✓ removed empty queries
✓ removed “non-sense” queries
Sample Data Set ✓ removed stop-words
✓ applied Porter stemming
algorithm


Data Analysis: query time gap

84.1% of adjacent query
pairs are issued within 26
minutes

tφ = 26 min.


Task-based Session Discovery
a combined approach
1) TimeSplitting-t 2) QueryClustering-m
The idea is that if two consecutive queries Queries in a time-based sessions are
are far away, they are also likely unrelated f u r t h e r g ro u p e d u s i n g c l u s t e r i n g
algorithms, which exploit several query
TS-t, where t is the time gap between features.
consecutive queries must be not greater Two queries (qi, qj) are in the same task-
than t based session iff they belong to the same
cluster.
This technique alone produces time-based
sessions, but is unable to deal with multi- QC-m, where m is one of the clustering
tasking method
Methods: QC-MEANS, QC-SCAN, QC-WCC, and
Methods: TS-5, TS-15, TS-26, etc. QC-HTC

QC-m: Query Features and Similarities
Content-based (µcontent) Semantic-based (µsemantic)
✓ two queries (qi, qj) sharing common terms are ✓ using Wikipedia and Wiktionary for
likely related “expanding” a query q
✓ µjaccard: Jaccard index on query character 3- ✓ “wikiﬁcation” of q using vector-space model
grams

✓ relatedness between (qi, qj) computed using
✓ µlevenshtein: normalized Levenshtein (edit) cosine-similarity
distance


Distance Functions: µ1 vs. µ2
✓Convex combination µ1

✓ Conditional formula µ2
Idea: if two queries are close in term of lexical content, the semantic expansion could be
unhelpful. Vice-versa, nothing can be said when queries do not share any content feature

✓ Both µ1 and µ2 rely on the estimation of some parameters, i.e., α, t, and b
✓ Use the ground-truth for tuning parameters


Ground-truth: construction e use
• ~500 long-term sessions are ﬁrst split using the threshold tφ devised before
(i.e., 26 minutes), obtaining several time-gap sessions

• Human annotators put together task-related queries that they claim to be
part of the same task

• This ground-truth is used for

• to tune some parameter of our method

• more importantly, to evaluate the automatic clustering for task-based
session discovery


Ground-truth: statistics
✓ 4.49 avg. queries per time-gap
session
✓ more than 70% time-gap session
contains at most 5 queries

✓ 1.80 tasks per time-gap session (avg.)
✓ 2.57 avg. queries per task-based ✓ ~47% time-gap session contains more
sessions than one task (multi-tasking)
✓ ~75% tasks contains at most 3
✓ 1,046 over 1,424 queries (i.e., ~74%)
queries included in multi-tasking sessions


QC-WCC
WEIGHTED CONNECTED COMPONENTS

time-gap session φ 1 2 3 4 5 6 7 8


QC-WCC


1 2 Build the similarity graph Gφ - O(N2)

8 3

7 4

6 5


QC-WCC


1 2

8 3

Drop “weak edges”
7 4

6 5


QC-WCC


1 2

8 3

7 4

6 5


QC-HTC
HEAD-TAIL COMPONENTS
• Variation of QC-WCC, that does not need to compute the full similarity graph
• Performs into 3 steps:
1. computes the similarities between chronologically consecutive queries in φ and
insert the corresponding labeled edge in the graph Gφ - O(N)
2. Drop “weak edges”, thus obtaining clusters corresponding to sub-sequences
• each cluster represented by the chronologically ﬁrst and last queries: head and
tail
3. Pairwise merging of the sub-sequences so obtained,
• by only inserting edges between the heads/tails if the corresponding queries are
similar enough

Experiments Setup

• Run and compare all the proposed approaches with:
• TS-26: time-splitting technique (baseline)

• QFG: session extraction method based on the query-ﬂow graph model
(state of the art)

P. Boldi, F. Bonchi, C. Castillo, D. Donato, A. Gionis, S.Vigna: The query-ﬂow graph: model and applications. CIKM 2008: 609-618


Evaluation
• Measure the degree of correspondence between the true tasks (manually-extracted
ground-truth), and the predicted tasks (algorithms’ outputs)

• we can use all the external measures commonly used to evaluate a clustering algorithm
when used to extract disjoint partitions in a dataset annotated with class labels

a) F-MEASURE b) RAND c) JACCARD
✓ evaluates the extent to ✓ pairs of queries instead ✓ pairs of queries instead
which a predicted task of singleton of singleton
contains only and all the ✓ f00, f01, f10, f11 ✓ f01, f10, f11
queries of a true task
✓ combines p(i, j) and r(i, j)
the precision and recall
of task i w.r.t. class j


Evaluation
• =Measureof objʼs w/ different class and task the true tasks (manually-extracted
f00 #pairs
the degree of correspondence between
ground-truth), and the predicted tasks (algorithms’ outputs)
f01 = #pairs of objʼs w/ different class and same task
• we can objʼs w/ external measures different used
f10 = #pairs ofuse all thesame class and commonlytask to evaluate a clustering algorithm
f11 = #pairs of objʼs extract disjoint partitions in a dataset annotated with class labels
when used to w/ same class and task

a) F-MEASURE b) RAND c) JACCARD
✓ evaluates the extent to ✓ pairs of queries instead ✓ pairs of queries instead
which a predicted task of singleton of singleton
contains only and all the ✓ f00, f01, f10, f11 ✓ f01, f10, f11
queries of a true task
✓ combines p(i, j) and r(i, j)
the precision and recall
of task i w.r.t. class j


Results: best
Supervised method
Query Flow Graph (QFG)

K-means (centroid-based)
and DB-Scan (density-
based)

P. Boldi, F. Bonchi, C. Castillo, D. Donato, A. Gionis, S.Vigna: The query-ﬂow graph: model and applications. CIKM 2008: 609-618


Task Recommendation


Crowd-based Task Synthesis
User 1
All the tasks appear
User 2
different
User 3
....

• We need to ﬁnd recurrent behaviors in the crowd
• We need to identify similar tasks by clustering the associated
“virtual documents”
• where each virtual document include the queries performed in
the task by a user

Crowd-based Task Synthesis
same Th
• We can rewrite each task-oriented User 1
session in terms of the new
User 2
synthesized tasks ids: Th
where Th = {T1 , ... , TK} User 3

....

• Th can be considered as a representative for an aggregation
composed of similar tasks, performed by several distinct users
• The various long term sessions thus become sets/sequences
of synthesized tasks in which we can ﬁnd recurrences


Task-based Model Generation
• Produce a Task Recommendation Model
• a weighted directed graph GT = (T, E, w), where the weighting function
w(.) measures the “inter-task relatedness”
• if they are related, they are probably part of the same mission

wh,i wk,i

wi,j

GT = (T, E, w)

Task-based Recommendation
• Generate a Task-oriented Recommendations
• given a user who is interested in (has just performed) a task Ti
• retrieve from GT the set Rm(Ti), which includes the m-top related nodes/
tasks to Ti
• the graph nodes in Rm(Ti) are directly connected to node Ti and are the
m ones labeled with the highest weights

Ti

Task-based Recommendation
• Generate a Task-oriented Recommendations
• given a user who is interested in (has just performed) a task Ti
• retrieve from GT the set Rm(Ti), which includes the m-top related nodes/
tasks to Ti
• the graph nodes in Rm(Ti) are directly connected to node Ti and are the
m ones labeled with the highest weights

R2(Ti)

Ti

How to Generate the Model
• Various methods to generate edges in GT and the
associated weights
wh,i wk,i • Random-based (baseline): an edge for each pair,
whose weights are uniform
wi,j

GT = (T, E, w) • Sequence-based: the frequency of the pairs wrt a
given support threshold, by considering the relative
order in the original sequences

• Association-Rule based (support): the frequency of the rule wrt a given support
threshold. We do not consider the relative order in the original sequences to
extract the rules
• Association-Rule based (confidence): the confidence of the rules wrt a given
confidence threshold. We do not consider the relative order in the original
sequences to extract the rules

Data Set: AOL 2006 Query Log
✓ ~20M queries
Original Data Set ✓ ~657K users

✓ Top-600 longest user sessions
✓ ~58K queries
✓ avg 14 queries per user/day

✓set A : 500 user sessions (training)
✓ set B : 100 user sessions (test)
Sample Data Set


Data Set: AOL 2006 Query Log
✓ ~20M queries
Original Data Set ✓ ~657K users

✓ Top-600 longest user sessions
✓ ~58K queries
✓ avg 14 queries per user/day

✓set A : 500 user sessions (training)
✓ set B : 100 user sessions (test)
Sample Data Set
From both A and B we extracted the
tasks with our QC-HTC
Set A was used to generate the graph
model using various mining techniques

Experimental results
• We used the log subset B (test query log) for evaluation
wh,i wk,i
• we divided each long term session in B (with
wi,j synthesized tasks) into a 1/3 prefix and 2/3 suffix

GT = (T, E, w) • the prefix is used to retrieve from GT the tasks:
Rm(prefix)
• We measured
• precision (proportion of suggestions that actually occur in the 2/3 suffix)
• coverage (proportion of tasks in the 1/3 prefix that are able to provide at least one
suggestion)
• changing the weighting in each model, by tuning the corresponding parameters, modifies
the coverage ...
• we thus plot precision vs coverage to permit the different models to be fairly compared

Recommendation Models


Anecdotal Evidence

actually
}
performed
queries


Anecdotal Evidence


Questions?


Salvatore_Orlando

Recommended

Recommended

More Related Content

Similar to Salvatore_Orlando

Similar to Salvatore_Orlando (20)

More from yaevents

More from yaevents (20)

Recently uploaded

Recently uploaded (20)

Salvatore_Orlando