Measuring Web User Engagement: a
cauldron of web analytics, focus attention, positive affect, user interest, saliency, mouse movement & multi-tasking Mounia Lalmas Yahoo! Labs Barcelona

Click-through rate as proxy of
user engagement!

Click-through rate as proxy of
relevance! Multimedia search activities often driven by entertainment needs, not by M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011. information needs

Click-through rate as proxy of
user user satisfaction! I just wanted the phone number … I am totally satisfied 

In this talk – results,
messages & questions 1. Big data and in-depth focused user studies “a must”! 2. Users “multi-task” online, what does this mean? 3. Mouse movement hard to “experiment with” and/or “interpret”. 4. Using crowd-sourcing “I think” worked fine.

This talk is not about
aesthetics … but see later http://www.lowpriceskates.com/ (e-commerce – skating) Source: http://www.webpagesthatsuck.com/

This talk is not about
usability http://chiptune.com/ (music repository) Source: http://www.webpagesthatsuck.com/

User Engagement – connecting three
sides  User engagement is a quality of the user experience that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology.  Successful technologies are not just used, they are engaged with. user feelings: happy, sad, user mental states: concentrated, user interactions: click, read excited, … lost, involved, … comment, recommend, buy, … The emotional, cognitive and behavioural connection that exists, at any point in time and over time, between a user and a technological resource S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.

Characteristics of user engagement Focused
Attention Novelty Positive Affect Richness & Control Aesthetics Reputation, Trust & Expectation Endurability Motivation, Interests, Incentives & Benefits H.L. O'Brien. Defining and Measuring Engagement in User Experiences with Technology. PhD Thesis, 2008. H.L. O'Brien & E.G. Toms. JASIST 2008, JASIST 2010.

Measuring user engagement

Connecting three measurement approaches nt
ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog

Models of user engagement …
towards a taxonomy? nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog

Models of user engagement Online
sites differ concerning their engagement! Games Search Users spend Users come much time per frequently and do visit not stay long Social media Special Users come Users come on frequently and average once stay long Service News Users visit site, Users come when needed periodically

Data and Metrics Interaction data,
2M users, July 2011, 80 US sites Popularity #Users Number of distinct users #Visits Number of visits #Clicks Number of clicks Activity ClickDepth Average number of page views per visit. DwellTimeA Average time per visit Loyalty ActiveDays Number of days a user visited the site ReturnRate Number of times a user visited the site DwellTimeL Average time a user spend on the site.

Methodology General models Time-based models
Dimensions weekdays, weekend 8 metrics 8 metrics per time span #Dimensions 8 16 Kernel k-means with Kendall tau rank correlation kernel Nb of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni #Clusters (Models) 6 5 Analysing cluster centroids = models

Models of user engagement [6
general] • Popularity, activity and loyalty are independent from each other • Popularity and loyalty are influenced by external and internal factors  e.g. frequency of publishing new information, events, personal interests • Activity depends on the structure of the site interest-specific periodic media e-commerce media (daily) search models based on engagement metrics only

Time-based [5 models] Models based
on engagement over weekdays and weekend work-related daily news hobbies, interest-specific weather time-based models ≠ general models next put all and more together! let machine learning tell you more!

Models of user engagement –
Recap & Next User engagement is complex and standard metrics capture only a part of it User engagement depends on time (and users) First step towards a taxonomy of models of user engagement … and associated metrics Next  More sites, more models  User demographics, time of the day, geo-location, etc.  Online multi-tasking J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.

Online multi-tasking nt ageme ti
on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog

Online multi-tasking 181K users, 2
months browser data, 600 sites, 4.8M sessions •only 40% of the sessions have no site revisitation •hyperlinking, backpaging and teleporting leaving a site is not a “bad thing!” (fictitious navigation between sites within an online session) users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information  ONLINE MULTI-TASKING navigating between sites, using browser tabs, bookmarks, etc seamless integration of social networks platforms into many services

Navigating between sites – hyperliking,
backpaging and teleporting Number of backpaging actions is an under-estimate!

Revisitation and navigation patterns

Online multi-tasking – Some results
48% sites visited at least 9 times Revisitation “level” depends on site 10% users accessed a site 9+ times (23% for search sites); 28% at least four times (44% for search sites) Activity on site decreases with each revisit but activity on many search and adult sites increases Backpaging usually increases with each revisit but hyperlinking remains important means to navigate between sites

Online multi-tasking – Recap &
Next J. Lehmann, M. Lalmas & G. Dupret. Online Multi-Tasking and User Engagement. Submitted for publication, 2013.

Focus attention, positive affect &
saliency nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog

Saliency, attention and positive affect
How the visual catchiness (saliency) of “relevant” information impacts user engagement metrics such as focused attention and emotion (affect) focused attention refers to the exclusion of other things affect relates to the emotions experienced during the interaction Saliency model of visual attention developed by Itti and Koch L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 2000.

Manipulating saliency non-salient condition salient
condition Web page screenshot Saliency maps

Study design  8 tasks
= finding latest news or headline on celebrity or entertainment topic  Affect measured pre- and post- task using the Positive e.g. “determined”, “attentive” and Negative e.g. “hostile”, “afraid” Affect Schedule (PANAS)  Focused attention measured with 7-item focused attention subscale e.g. “I was so involved in my news tasks that I lost track of time”, “I blocked things out around me when I was completing the news tasks” and perceived time  Interest level in topics (pre-task) and questionnaire (post-task) e.g. “I was interested in the content of the web pages”, “I wanted to find out more about the topics that I encountered on the web pages”  189 (90+99) participants from Amazon Mechanical Turk

Saliency and positive affect When
headlines are visually non-salient  users are slow at finding them, report more distraction due to web page features, and show a drop in affect When headlines are visually catchy or salient  user find them faster, report that it is easy to focus, and maintain positive affect Saliency is helpful in task performance, focusing/avoiding distraction and in

Saliency and focused attention Adapted
focused attention subscale from the online shopping domain to entertainment news domain Users reported “easier to focus in the salient condition” BUT no significant improvement in the focused attention subscale or differences in perceived time spent on tasks User interest in web page content is a good predictor of focused attention, which in turn is a good predictor of positive affect

Saliency and user engagement –
Recap & Next Interaction of saliency, focused attention, and affect, together with user interest, is complex Next: include web page content as a quality of user engagement in focused attention scale more “realistic” user (interactive) reading experience bio-metrics (mouse-tracking, eye-tracking, facial expression, etc) L. McCay-Peet, M. Lalmas, V. Navalpakkam. On saliency, affect and focused attention, CHI 2012

Mouse tracking, positive effect, attention
nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog

Mouse tracking … and user
engagement  324 users from Amazon Mechanical Turk (between subject design)  Two domains (BBC and Wikipedia)  Two tasks (reading and quiz)  “Normal vs Ugly” interface  Questionnaires (qualitative data)  focus attention, positive effect, novelty, interest, usability, aesthetics  + demographics, handeness & hardware  Mouse tracking (quantitative data)  movement speed, movement rate, click rate, pause length, percentage of time still

“Ugly” vs “Normal” Interface (BBC
News)

“Ugly” vs “Normal” (Wikipedia)

Mouse tracking can tell about
Age Hardware Mouse Trackpad Task  Searching: There are many different types of phobia. What is Gephyrophobia a fear of?  Reading: (Wikipedia) Archimedes, Section 1: Biography

Mouse tracking could not tell
much on

Mouse tracking and user engagement
— Recap & Next High level of ecological validity Age, task, and hardware Do we have a Hawthorne Effect??? “Usability” vs engagement “Even uglier” interface? I don’t think so Within- vs between-subject design? Next Sequence of movements Automatic clustering D. Warnock and M. Lalmas. An Exploration of Cursor tracking Data. Submitted for publication, 2013.

Connecting three measurement approaches on
interacti The value of a click? self- repo r te d e gn itiv co

Thank you Collaborators: Ioannis Arapakis
Ricardo Baeza-Yates Georges Dupret Janette Lehmann Lori McCay-Peet (Dalhousie University) Vidhya Navalpakkam David Warnock (Glasgow University) Elad Yom-Tov Contact: mounia@acm.org and many others at Yahoo! Labs