1. Session, focus and engagement
Mounia Lalmas
Yahoo! Research Barcelona
mounia@acm.org
2. A bit about myself
1999-2008: Lecturer (assistant professor) to Professor at
Queen Mary, University of London
2008-2010 Microsoft Research/RAEng Research
Professor at the University of Glasgow (and lived outside
London)
2011- Visiting Principal Scientist at Yahoo! Research
Barcelona
Research topics
XML retrieval and evaluation (INEX)
Quantum theory to model interactive information retrieval
Aggregated search
Bridging the digital divide
Models and measures of user engagement
3. Message and Outline
Interaction and search
Beyond result relevance
Beyond search session
Towards “engagement”
1. Motivations
2. Engagement
3. Future directions
5. Information Retrieval Over Query Sessions
Retrieval Models & Ranking: How to analyze/model/predict user
interactions and use these findings to improve retrieval
performance? How can we adapt ranking/retrieval models and IR
theory in the light of a sequence of user interactions.
Evaluation & Test Collections: How can we evaluate retrieval
system performance over entire query sessions? How can we build
reusable test collections to study this IR task? How can we
model/simulate user interactions over a session?
User Interaction & Interfaces: How can we model user
interactions so we can predict and improve the user experience
over sessions? How can we design and perform user studies that
reveal new information about users? How can we make use of
implicit feedback from users?
6. Relevance in multimedia search
Multimedia search
activities often
driven by
entertainment
needs, not by
M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011
information needs
7. Relevance in focused retrieval
Relevance in context
Table of Content Focused retrieval is about putting
results (element, fact, passage) in
context, to understand and trust them
Courtesy Jaap Kamps, Zoltan Szlavik, Norbert Goevert
8. Courtesy of Janette Lehmann
Beyond search session
On month browsing data,
sample of sites
(INT=Yahoo site,
EXT=non Yahoo site)
On month browsing data, sample of Yahoo! sites
users spend more and more of their online session multi-tasking, e.g. emailing,
reading news, searching for information ONLINE MULTI-TASKING
navigating between sites, using browser tabs, etc
seamless integration of social networks platforms into many services
11. User Engagement – connecting three sides
User engagement is a quality of user experience that
emphasizes the positive aspects of interaction – in
particular the fact of being captivated by the technology.
Successful technologies are not just used, they are
engaged with.
user feelings: happy, sad, user mental states: concentrated, user interactions: click, read
excited, bored, … challenged, lost, interested … comment, recommend, buy, …
The emotional, cognitive and/or behavioural connection
that exists, at any point in time and over time, between a
user and a technological resource
12. Characteristics of user engagement (I)
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),
WSDM Workshop on User Modelling for Web Applications, 2011.
13. Characteristics of user engagement (II)
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),
WSDM Workshop on User Modelling for Web Applications, 2011.
14. The four I’s
Measuring Engagement, Forrester Research, June 2008
17. Models of user engagement
Online sites differ concerning their engagement!
Games Search
Users spend Users come
much time per frequently and
visit do not stay long
Social media Special
Users come Users come on
frequently and average once per
stay long time considered
Service News
Users visit site, Users come
when needed periodically
Is it possible to model these differences?
18. Data and Metrics
Interaction data, 2M users, July 2011, 80 US sites
Popularity #Users Number of distinct users
#Visits Number of visits
#Clicks Number of clicks
Activity ClickDepth Average number of page views per visit.
DwellTimeA Average time per visit
Loyalty ActiveDays Number of days a user visited the site
ReturnRate Number of times a user visited the site
DwellTimeL Average time a user spend on the site.
19. Diversity in user engagement
Engagement of a site depends on users and time
Users and Loyalty Time and Popularity
Sites have different user groups Site engagement can be periodic
Proportion of user groups is site- or contains peaks
dependent
mail, social
media
media
(special events)
media,
entertainment
daily activity, shopping,
navigation entertainment
20. Methodology
General models User-based models Time-based models
Dimensions 5 user groups weekdays, weekend
8 metrics 8 metrics per user 8 metrics per time span
group
#Dimensions 8 40 16
Kernel k-means with
Kendall tau rank correlation kernel
Nb of clusters based on eigenvalue distribution of kernel matrix
Significant metric values with Kruskal-Wallis/Bonferonni
#Clusters
(Models) 6 7 5
Analysing cluster centroids = models
21. Models of user engagement
Models based on engagement metrics
• 6 general models
• Popularity, activity and loyalty are
independent from each other
• Popularity and loyalty are influenced by
external and internal factors
e.g. frequency of publishing new
information, events, personal interests
• Activity depends on the structure of the site
interest-
specific
periodic
media
e-commerce,
configuration
22. Models of user engagement
Models based on engagement metrics, user and time
User-based [7 models] Time-based [5 models]
Models based on engagement per Models based on engagement
user group over weekdays and weekend
navigation game, sport
hobbies, daily news
interest-specific
Sites of the same type (e.g. mainstream media) do not necessarily belong to
the same model
The groups of models describe different aspects of engagement, i.e. they are
independent from each other
23. Recap & Next
User engagement is complex and standard
metrics capture only a part of it
First step towards a taxonomy of models of user
engagement … and associated metrics
Next
Interaction between models
Interaction between sites (multi-tasking)
User demographics, time of the day, geo-location, etc
J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
24. 3. Outline
1. Motivations
• Engagement
• Future directions
• The three sides of user engagement
• Interactive IR
• Towards engagement
25. Let us revisit … connecting three sides
+ layout +links The three sides
+ saliency + content
+ emotional
+ cognitive
+ behavioral
user engagement
within and across site Measurements and methodologies
+ online analytics metrics (dwell time, CTR, …)
+ complex networks metrics
Goals
+ Models of user engagement + questionnaires, surveys, …
+ Metrics of user engagement + crowd-sourcing
+ biometrics (eye tracking, mouse tracking, …)
26. Let us revisit … Interactive IR
P Ingwersen, Human Aspects in IR, ESSIR 2011.
27. session, interaction, multi-tasking,
network, search, relevance, …
•I Aapakis, K Athanasakos, J Jose, A comparison of general vs personalised affective models for
the prediction of topical relevance, SIGIR 2010.
•J Huang, R White, S Dumais, No clicks, no problem: using cursor movements to understand and improve
search, CHI 2011.
• P Ingwersen & K Järvelin, The turn: integration of information seeking and retrieval in context, 2005.
TOWARDS ENGAGEMENT
28. Information Retrieval Over Query Sessions
Retrieval Models & Ranking: How to analyze/model/predict user
T e
EN nc
interactions and use these findings to improve retrieval
EM eva
performance? How can we adapt ranking/retrieval models and IR
theory in the light of a sequence of user interactions.
AG rel
E aNG nd
Evaluation & Test Collections: How can we evaluate retrieval
system performance over S
D ion
entire query sessions? How can we build
R s over a session?
reusable test collections to study this IR task? How can we
A es
model/simulate user interactions
W s
TO nd
o
User Interaction & Interfaces: How can we model user
ywe can predict and improve the user experience over
interactions e
b so
sessions? How can we design and perform user studies that reveal
new information about users? How can we make use of implicit
feedback from users?
29. Thank you
mounia@acm.org T e
EN nc
EM eva
www.dcs.gla.ac.uk/~mounia
AG rel
NG nd
E a
D S ion
R s
A es
W s
TO nd
yo
be
Editor's Notes
Characteristics elaborate the notion of engagement over 3 broad dimensions: emotional, cognitive and behavioural. We identified 8 of them based on previous works / suggested by us. Attention -> exclusivity Affect -> lack of fun can lead as a barrier to shopping (O’Brian) Aesthetics -> promotes attention, stimulate curiosity Endurability -> does the experience meet the expectation?
Novelty -> freshness of content / has to be carefully balanced Richness = complexity of thoughts and actions, increased complexity/ Affect = relates to the emotion experienced during the interaction (fun and enjoyment)