This document discusses user engagement in information retrieval over query sessions. It begins with an introduction of the author and their research topics related to engagement. The main topics covered are: 1) Motivations for looking beyond result relevance and search sessions to consider engagement. 2) A definition of engagement as the emotional, cognitive and behavioral connection between a user and technology. Characteristics of engagement like concentration, interest and actions are discussed. 3) Models of user engagement are proposed based on metrics of popularity, activity and loyalty for general engagement models as well as user-based and time-based models. The models capture different aspects of engagement for sites. Future directions discussed include analyzing interactions between models

1. Session, focus and engagement
Mounia Lalmas
Yahoo! Research Barcelona
mounia@acm.org

2. A bit about myself
 1999-2008: Lecturer (assistant professor) to Professor at
Queen Mary, University of London
 2008-2010 Microsoft Research/RAEng Research
Professor at the University of Glasgow (and lived outside
London)
 2011- Visiting Principal Scientist at Yahoo! Research
Barcelona
 Research topics
 XML retrieval and evaluation (INEX)
 Quantum theory to model interactive information retrieval
 Aggregated search
 Bridging the digital divide
 Models and measures of user engagement

3. Message and Outline
 Interaction and search
 Beyond result relevance
 Beyond search session
 Towards “engagement”
1. Motivations
2. Engagement
3. Future directions

4. 1. Outline
1. Motivations
• Relevance in multimedia search
• Relevance in focused retrieval
• Online multi-tasking
• Engagement
• Future directions

5. Information Retrieval Over Query Sessions
 Retrieval Models & Ranking: How to analyze/model/predict user
interactions and use these findings to improve retrieval
performance? How can we adapt ranking/retrieval models and IR
theory in the light of a sequence of user interactions.
 Evaluation & Test Collections: How can we evaluate retrieval
system performance over entire query sessions? How can we build
reusable test collections to study this IR task? How can we
model/simulate user interactions over a session?
 User Interaction & Interfaces: How can we model user
interactions so we can predict and improve the user experience
over sessions? How can we design and perform user studies that
reveal new information about users? How can we make use of
implicit feedback from users?

6. Relevance in multimedia search
Multimedia search
activities often
driven by
entertainment
needs, not by
M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011
information needs

7. Relevance in focused retrieval
Relevance in context
Table of Content Focused retrieval is about putting
results (element, fact, passage) in
context, to understand and trust them
Courtesy Jaap Kamps, Zoltan Szlavik, Norbert Goevert

8. Courtesy of Janette Lehmann
Beyond search session
On month browsing data,
sample of sites
(INT=Yahoo site,
EXT=non Yahoo site)
On month browsing data, sample of Yahoo! sites
users spend more and more of their online session multi-tasking, e.g. emailing,
reading news, searching for information  ONLINE MULTI-TASKING
navigating between sites, using browser tabs, etc
seamless integration of social networks platforms into many services

9. Interactive IR …
P Ingwersen, Human Aspects in IR, ESSIR 2011.

10. 2. Outline
1. Motivations
• (User) Engagement
• Definition
• Characteristics
• Measuring
• Models
• Future directions

11. User Engagement – connecting three sides
 User engagement is a quality of user experience that
emphasizes the positive aspects of interaction – in
particular the fact of being captivated by the technology.
 Successful technologies are not just used, they are
engaged with.
user feelings: happy, sad, user mental states: concentrated, user interactions: click, read
excited, bored, … challenged, lost, interested … comment, recommend, buy, …
The emotional, cognitive and/or behavioural connection
that exists, at any point in time and over time, between a
user and a technological resource

12. Characteristics of user engagement (I)
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),
WSDM Workshop on User Modelling for Web Applications, 2011.

13. Characteristics of user engagement (II)
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),
WSDM Workshop on User Modelling for Web Applications, 2011.

14. The four I’s
Measuring Engagement, Forrester Research, June 2008

15. Measuring user engagement

16. Objective measures – Online activities
Proxy of user engagement

17. Models of user engagement
Online sites differ concerning their engagement!
Games Search
Users spend Users come
much time per frequently and
visit do not stay long
Social media Special
Users come Users come on
frequently and average once per
stay long time considered
Service News
Users visit site, Users come
when needed periodically
Is it possible to model these differences?

18. Data and Metrics
Interaction data, 2M users, July 2011, 80 US sites
Popularity #Users Number of distinct users
#Visits Number of visits
#Clicks Number of clicks
Activity ClickDepth Average number of page views per visit.
DwellTimeA Average time per visit
Loyalty ActiveDays Number of days a user visited the site
ReturnRate Number of times a user visited the site
DwellTimeL Average time a user spend on the site.

19. Diversity in user engagement
Engagement of a site depends on users and time
Users and Loyalty Time and Popularity
 Sites have different user groups  Site engagement can be periodic
 Proportion of user groups is site- or contains peaks
dependent
mail, social
media
media
(special events)
media,
entertainment
daily activity, shopping,
navigation entertainment

20. Methodology
General models User-based models Time-based models
Dimensions 5 user groups weekdays, weekend
8 metrics 8 metrics per user 8 metrics per time span
group
#Dimensions 8 40 16
Kernel k-means with
Kendall tau rank correlation kernel
Nb of clusters based on eigenvalue distribution of kernel matrix
Significant metric values with Kruskal-Wallis/Bonferonni
#Clusters
(Models) 6 7 5
Analysing cluster centroids = models

21. Models of user engagement
Models based on engagement metrics
• 6 general models
• Popularity, activity and loyalty are
independent from each other
• Popularity and loyalty are influenced by
external and internal factors
 e.g. frequency of publishing new
information, events, personal interests
• Activity depends on the structure of the site
interest-
specific
periodic
media
e-commerce,
configuration

22. Models of user engagement
Models based on engagement metrics, user and time
User-based [7 models] Time-based [5 models]
 Models based on engagement per  Models based on engagement
user group over weekdays and weekend
navigation game, sport
hobbies, daily news
interest-specific
 Sites of the same type (e.g. mainstream media) do not necessarily belong to
the same model
 The groups of models describe different aspects of engagement, i.e. they are
independent from each other

23. Recap & Next
 User engagement is complex and standard
metrics capture only a part of it
 First step towards a taxonomy of models of user
engagement … and associated metrics
 Next
 Interaction between models
 Interaction between sites (multi-tasking)
 User demographics, time of the day, geo-location, etc
J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.

24. 3. Outline
1. Motivations
• Engagement
• Future directions
• The three sides of user engagement
• Interactive IR
• Towards engagement

25. Let us revisit … connecting three sides
+ layout +links The three sides
+ saliency + content
+ emotional
+ cognitive
+ behavioral
user engagement
within and across site Measurements and methodologies
+ online analytics metrics (dwell time, CTR, …)
+ complex networks metrics
Goals
+ Models of user engagement + questionnaires, surveys, …
+ Metrics of user engagement + crowd-sourcing
+ biometrics (eye tracking, mouse tracking, …)

26. Let us revisit … Interactive IR
P Ingwersen, Human Aspects in IR, ESSIR 2011.

27. session, interaction, multi-tasking,
network, search, relevance, …
•I Aapakis, K Athanasakos, J Jose, A comparison of general vs personalised affective models for
the prediction of topical relevance, SIGIR 2010.
•J Huang, R White, S Dumais, No clicks, no problem: using cursor movements to understand and improve
search, CHI 2011.
• P Ingwersen & K Järvelin, The turn: integration of information seeking and retrieval in context, 2005.
TOWARDS ENGAGEMENT

28. Information Retrieval Over Query Sessions
 Retrieval Models & Ranking: How to analyze/model/predict user
T e
EN nc
interactions and use these findings to improve retrieval
EM eva
performance? How can we adapt ranking/retrieval models and IR
theory in the light of a sequence of user interactions.
AG rel

E aNG nd
Evaluation & Test Collections: How can we evaluate retrieval
system performance over S
D ion
entire query sessions? How can we build
R s over a session?
reusable test collections to study this IR task? How can we
A es
model/simulate user interactions
W s
TO nd
 o
User Interaction & Interfaces: How can we model user
ywe can predict and improve the user experience over
interactions e
b so
sessions? How can we design and perform user studies that reveal
new information about users? How can we make use of implicit
feedback from users?

29. Thank you
mounia@acm.org T e
EN nc
EM eva
www.dcs.gla.ac.uk/~mounia
AG rel
NG nd
E a
D S ion
R s
A es
W s
TO nd
yo
be