The document discusses user engagement in web search and its characteristics, highlighting its measurement and implications for search applications. It emphasizes the importance of positive user experiences and the emotional, cognitive, and behavioral connections between users and technology. Various studies described in the talk explore factors such as inter-session engagement, online multi-tasking, and serendipity in user engagement across different web applications.

1. An Engaging Click
Mounia Lalmas
Ya h o o L a b s
London
Search Solutions 2013

2. This talk
What is user engagement?
What are the characteristics of user engagement?
How to measure user engagement?
What is user engagement in web search?
What is an engaging click?
1. inter-session
2. online multi-tasking
3. downstream engagement
4. serendipity
work on user
engagement across
web applications
implications to web
search

3. This talk
What is user engagement?
What are the characteristics of user engagement?
How to measure user engagement?
What is user engagement in web search?
What is an engaging click?
1. inter-session
2. online multi-tasking
3. downstream engagement
4. serendipity
Work on user
engagement across
web applications
Implications to search

4. What is user engagement?
User engagement is a quality of the user experience
that emphasizes the positive aspects of interaction – in
particular the fact of being captivated by the
technology (Attfield et al, 2011).
user feelings: happy, sad,
excited, …
user mental states: flow,
presence, immersion, …
user interactions: click,
read, comment, buy…
emotional, cognitive and behavioural connection
that exists, at any point in time and over time, between
a user and a technological resource

5. Why is it important to engage users?
§ In today’s wired world, users have enhanced expectations
about their interactions with technology
… resulting in increased competition amongst the
purveyors and designers of interactive systems.
§ In addition to utilitarian factors, such as usability, we must
consider the hedonic and experiential factors of interacting
with technology, such as fun, fulfillment, play, and user
engagement.
(O’Brien, Lalmas & Yom-Tov, 2013)

6. Characteristics of user engagement
Endurability
Aesthetics
(Read, MacFarlane, & Casey,
(Jacques et al, 1995; O’Brien,
2008)
2002; O’Brien, 2008)
Motivation, interests,
incentives, and benefits
(Jacques et al., 1995; O’Brien &
Toms, 2008)
Focused attention
(Webster & Ho, 1997; O’Brien,
2008)
Novelty
(Webster & Ho, 1997; O’Brien,
2008)
Reputation, trust and
expectation (Attfield et al,
2011)
Richness and control
Positive Affect
(O’Brien & Toms, 2008)
(O’Brien, Lalmas & Yom-Tov, 2013)
(Jacques et al, 1995; Webster &
Ho, 1997)

7. Measuring user engagement
Measures
Characteristics
SelfQuestionnaire, interview, report,
Subjective
reported
product reaction cards, think-aloud Short- and long-term
engagement
Lab and field
Small-scale
Cognitive
Task-based methods (time spent,
engagement follow-on task)
Objective
Short-term
Lab and field
Physiological measures (e.g. EEG, Small-scale and largeSCL, fMRI, eye tracking, mousescale
tracking)
Interaction
Web analytics
engagement
metrics + models
Objective
Short- and long-term
Field
Large-scale

8. User engagement in web search – around
“returning relevant results to the users”
(Slaney, 2011)
satisfying user information needs
§ Clickthrough rate (CTR)
§ Dwell time (on landing page)
§ Time to first click
§ Skipping
§ Multimedia search activities
often driven by
entertainment needs, not by
information needs.
§ Displaying rich information
on result pages (restaurant
phone number) means that
users do not need to click.
§ Abandonment rate
§ Number of query reformulations
§ Search engine switching
§ Cumulative gain family of
metrics, precision at rank k, …

9. This talk
What is user engagement?
What are the characteristics of user engagement?
How to measure user engagement?
What is user engagement in web search?
What is an engaging click?
1. inter-session
2. online multi-tasking
3. downstream engagement
4. serendipity
work on user
engagement across
web applications
implications to web
search

10. (Dupret & Lalmas, 2013)
•
•
Domain: Yahoo Answers Japan
Study: Inter-session engagement metric
If users find a web application interesting,
engaging or useful, they will return to it sooner.

11. Absence time and survival analysis
survival Analysis: high hazard rate = short absence
short absence is
a sign of loyalty
important indication
of user engagement

12. Using absence time to compare 6 ranking
functions (buckets) on Yahoo Answers Japan
1. Returning relevant results is important, but is not enough to
keep returning to the search application
2. Clicks after the 5th results reflect poorer user experience;
users cannot find what they are looking for
Endurability
3. No click means a bad user experience
4. Clicking lower in the ranking suggests more careful choice
from the user
5. Clicking at bottom is a sign of low quality overall ranking
6. Users finding their answers quickly (click sooner) return
sooner to the search application
7. Returning to the same search result page is a worse user
experience than reformulating the query.

13. (Lehmann et al, 2013)
•
•
Domain: 700+ web applications
Study: Online multi-tasking
Online multi-tasking affects the way users interact
(or engage) with sites.

14. Online multi-tasking – and search
181K users, 2 months browser
data, 600 sites, 4.8M sessions
• only 40% of the sessions have no
site revisitation
• commonly accessed sites between visits à search 22%, navigation 12%, social 8%
• for some sites (e-commerce) same sites are accessed between visits à one task?
• no patterns for sites such as mail, social à anchor, habit?
• longer time between visits à a different task (new search)
• more vs less times spent at each revisit à increased vs shift of attention

15. Revisitation patterns
average attention
mail sites [decreasing attention]
search sites [increasing attention]
auction sites [complex attention]
100% 54% 36% 26% 20% 17% 14% 12% 10%
100% 62% 41% 29% 21% 16% 13% 10% 8%
100% 69% 54% 44% 38% 33% 29% 26% 23%
100% 67% 54% 46% 41% 35% 31% 29% 26%
p-value < 0.05
p-value < 0.05
m = -0.288
p-value < 0.05
m = 0.063
p-value = 0.24
m = 0.142
●
0.8
●
12
11
11.2
0.4
●
●
●
●
●
●
10
●
●
1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9
0.4
●
●
1 2 3 4 5 6 7 8 9
0.8
●
●
●
11.0
●
Motivation, interests,
incentives, and benefits
k [kth visit on site]
0.0
0.0
§ 10% users accessed a site 9+ times (23% for search sites); 28% at least four
1 2 3 4 5 6 7 8 9
1 2 3 4
1 times (44%7for search sites) 5 6 7 8 9
2 3 4 5 6
8 9
1 2 3 4 5 6 7 8 9
0.0
0.0
●
§ 48% sites visited at least 9 times
§ Revisitation “level” depends on site
0.4
0.8
1 2 3 4 5 6 7 8 9
% of navigation type
●
● ●
●
0.4
● ●
●
10.8
12
●
●
●
●
●
0.8
●
●
11
11.2
● ●
13
●
●
10
●
10.8
% of total page views on site
proportion of users
k [kth visit on site]
k [kth visit on site]
k [kth visit on site]
Hyperlinking
§ Activity on site decreases with each Teleporting but activity on many search
revisit Backpaging
sites increases

16. (Yom-Tov et al, 2013)
•
•
Domain: Network of sites
Study: Downstream engagement
Success of a site depends on itself, but also on
how it is reached from other sites.

17. Downstream user engagement:
engagement across a network of sites
Large online providers (AOL, Google, Yahoo!, MSN, etc.)
offer not one service (site), but a network of sites
Downstream engagement
for site A
(% remaining session time)
Site A
Provider sites
User session

18. Influential features (50 Yahoo sites, 250K+ users, 1.9M sessions)
o Time of day
o Number of (non-image/non-video) links to Yahoo! sites in HTML body
o Average rank of Yahoo! links on page
o Number of (non-image/non-video) links to non-Yahoo! sites in HTML body
o Number of span tags (tags that allow adding style to content or manipulating
content, e.g. JavaScript)
o Link placements and number of Yahoo links can influence downstream
engagement
o Not new, but here shown to hold also across sites
Richness and control
o Links to non-Yahoo sites have a positive effect on downstream
engagement
o Possibly because when users are faced with abundance of outside links
they decide to focus their attention on a central content provider, rather than
visiting multitude of external sites

19. (Bordino, Mejova & Lalmas, 2013)
•
•
Domain: social media (Yahoo! Answers and Wikipedia)
Study: serendipity (in entity search)
Interesting search results may promote
serendipitous browsing.

20. Yahoo! Answers
community-driven question &
answer portal
§ 67 336 144 questions & 261
minimally curated
770 047 answers
opinions, gossip, personal info
§ January 1, 2010 –of view
variety of points
December 31, 2011
§ English-language
Entity
Search
vs
Wikipedia
community-driven
encyclopedia
• 3 795 865 articles
curated
• as of endknowledge
high-quality of
December 2011
variety of niche topics
• English Wikipedia
we build an entity-driven serendipitous search system
based on entity networks extracted from Wikipedia and
Yahoo! Answers
Serendipity
finding something good or useful while not
specifically looking for it, serendipitous search
systems provide relevant and interesting results

21. retrieve entities most related to a
query entity using random walk
Wikipedia
Yahoo! Answers
•
•
Annotator agreement (overlap): 0.85
Average overlap in top 5 results: <1

22. Serendipity
“making fortunate discoveries by
accident”
Serendipity = unexpectedness + relevance
“Expected” result baselines from web search
Baseline
Top:
5
en&&es
that
occur
most
frequently
in
top
5
search
from
Bing
and
Google
Top
–WP:
same
as
above,
but
excluding
Wikipedia
page
from
results
Rel:
top
5
en&&es
in
the
related
query
sugges&ons
provided
by
Bing
and
Google
Rel
+
Top:
union
of
Top
and
Rel
| relevant & unexpected | / | unexpected |
number of serendipitous results out of all
of the unexpected results retrieved
Data
WP
YA
WP
YA
WP
YA
WP
YA
0.63
(0.58)
0.69
(0.63)
0.63
(0.58)
0.70
(0.64)
0.64
(0.61)
0.70
(0.65)
0.61
(0.54)
0.68
(0.57)
| relevant & unexpected | / | retrieved |
serendipitous out of all retrieved

23. Interestingness ≠ Relevance
Interesting > Relevant
Oil Spill à
Robert Pattinson à
Egypt à
Penguins in Sweaters WP Water for Elephants WP Ptolemaic Kingdom WP & YA
Relevant > Interesting
Lady Gaga à Britney Spears
Novelty
WP
Egypt à Cairo Conference WP
Netflix à Blu-ray Disc YA

24. Take-away message
§ Search is not just about specific information needs
§ People search for many other reasons
›
Navigation
›
›
Transaction
Fun (ECIR 2012 workshop)
›
Etc.
§ Engagement in search is to view search activities as part of
the current overall task of a user
§ We never know what we get if we are ready to explore
›
Users do things that no one expects, not even them!
›
(like staying inside Yahoo! in spite of having many links to go elsewhere)
So a link is not everything, for search too!
§ Summarizing, in search we need to look at user
engagement in a broader way

25. Thank you
Acknowledgements: Ricardo Baeza-Yates, Ilaria Bordino, George
Dupret, Janette Lehmann, Yelena Mejova and Elad Yom-Tov.
Blog: labtomarket.wordpress.com
A first version of this talk was given by Ricardo Baeza-Yates, SIGIR 2013 Industry Day