This document presents research on predicting user engagement with direct displays like knowledge panels using mouse cursor data. The researchers conducted a crowdsourcing study tracking users' mouse cursors during search tasks. They developed predictive models to determine when users notice, find useful, and perceive faster task completion from direct displays. Their models outperformed baselines in accuracy and other metrics, showing mouse cursor data can predict user engagement without explicit feedback. The researchers conclude this approach offers an efficient way to analyze interactions and optimize direct display placement and content.

1. Predicting User Engagement with Direct Displays Using
Mouse Cursor Information
Ioannis Arapakis (Eurecat), Luis Leiva (Sciling)

2. Contents
§Introduction & motivation
§Merits of mouse cursor analysis
§Experimental setup
§Predictive modelling
§Performance assessment
§Results
§Conclusions

3. Introduction
§ In recent years direct displays (DDs)
have become a standard component
on the SERPs of all major web search
engines
§ DDs serve two main purposes:
• Provide well-structured summary of
information which is difficult or time-
consuming to access
• Help tidy up the SERP section that contains
the universal search results

4. Knowledge Module
§ One such prominent example is the Knowledge Module (KM)
display which provides users with information about the named
entities they are searching
§ The content presented in the KM display is obtained in a semi-
structured format from curated entity databases (e.g., Freebase,
Wikipedia)
§ This raw information is further enriched by the search engine,
e.g., ranking of related entities, explanations of their relationship
or with related multimedia and social media content

5. Motivation
§ In this context, most research has focused on general backend
system tasks, the most important being knowledge base
construction, or more specific backend tasks such as related
entity recommendation
§ This work attempts to understand how users engage with a DD
like the KM display in entity-centric search tasks
§ We are interested in predicting user engagement with a DD in
the absence of explicit feedback (e.g., self-report data)

6. Addressing the gap
§ Existing modelling techniques make a simplifying assumption
when analysing web search log data: the user is assumed to
be equally engaged with all parts of the SERP
§ In practice this assumption is not always true:
• A user may click on certain links on the page, but not all links
• May read a certain result snippet in the SERP but not necessarily the entire
list of results
• May ignore the SERP content completely and focus only on the images
shown in the KM display or other DDs

7. Mouse cursor tracking
§ Navigation & interaction with a digital
environment usually involves the use of a
mouse (i.e., selecting, hovering, clicking)
§ Can be easily performed in a non-invasive
manner, without removing users from their
natural setting
§ Several works have shown that the mouse
cursor is a proxy of gaze (attention)
§ Low-cost, scalable alternative to eye-
tracking

9. Crowdsourcing study
§ We conducted a crowdsourcing study and examine how users
engage with DDs like the knowledge module (KM) display
§ We collected and analysed more than 115K mouse cursor
positions from 300 users
§ With this study we aim to predict:
• When a user notices the KM display on the SERP
• If it is perceived as a useful aid to their search tasks
• Whether interacting with the KM display alters the users’ perception of how
fast they complete the search tasks

10. Experimental design
§ Repeated-measures design
§ One independent variable: KM display (with two levels: “visible”
or “hidden”)
§ Three dependent variables: (i) KM display noticeability, (ii) KM
display usefulness and (iii) perceived task accomplishment
speed
§ Two short search tasks were completed using the Yahoo search
engine: one task with the KM display on the SERP and one
without it*
* The KM display visibility was controlled with client-side scripting.

11. Search UI
§ Participants accessed the search engine through a custom proxy
which did not alter the original look and feel of the SERPs
§ This allowed us to capture user interactions with the SERP
without interfering with the actual web search engine interface in
production
§ For each search task, participants were presented with a
question and were suggested a search query to begin with

12. Search query sample
§ Query set consisted of 32 unique query patterns (144 different queries in total)
§ The selected query patterns belonged to four different topics (famous people,
movies, athletes, sport teams) and required either single or multiple answers

13. Mouse cursor tracking tool
§ To collect mouse cursor data we used EVTRACK*, an open
source JavaScript event tracking library that is part of the smt2ε
system
§ EVTRACK allows to specify what browser events should be
captured and how they should be captured, i.e., via event
listeners (the event is captured as soon as it is fired) or via event
polling
* https://github.com/luileito/evtrack

14. Self-reported measures of engagement
§ A mini-questionnaire on the SERPs gathered
ground truth labels for the mouse cursor data
§ The mini-questionnaire was initially hidden
and was shown to the user just before
leaving the SERP
§ It comprised 3 questions:
• Did you notice the knowledge module?
• To what extent did you find the knowledge module
useful in answering the question?
• To what extent did the knowledge module help you
answer the question faster?

15. Procedure
§ Participants were asked to evaluate two different backend
systems of Yahoo web search by performing two search tasks
§ For each task, participants had to answer a question by
searching for relevant information on the proxified search engine
§ In one task the KM display would be hidden (control condition)
and in the other task it would be visible (experimental condition)
§ The order of the tasks was randomized for each participant
§ Participants were presented with a suggested query to begin
their search but were free to submit additional queries
§ We used informational, entity-centric queries to introduce a
common starting point across all participants

16. Modelling user engagement
§ Our final dataset consists of ~115K cursor positions, collected
during 600 search task sessions
§ Out of those 600 search task sessions we analysed the 300
cases that correspond to the experimental condition with the
visible KM in the SERP
§ Our dataset is generally balanced, with 176 users having
reported noticing the KM display
§ We normalised the values for each feature so that feature values
that those that fall in greater numeric ranges do not dominate
over those in smaller numeric ranges

17. Feature Engineering
* These functions are computed for most base and meta-features.

18. Users who did not engage with the KG Users who engaged with the KG

19. Feature Engineering (cont.)

20. Predictive Modelling
§ We trained 10 RF* models (90% of data) and used
them to obtain the predictions for each of the held-
out set (10% of data) among the ten folds**
§ Excluded highly correlated and linearly dependent
features
§ Performed feature selection using recursive
feature elimination
§ We used a subset of our training data for fine-
tuning the classifier’s hyperparameters
* R packages “Caret” and “randomForest”.
** With stratified sampling.

21. Performance evaluation
§ Baselines:
• If the user clicked on the KM display (hasClickedKM, binary)
• If the mouse cursor hovered over the KM display (hasHoveredKM, binary)
• Time spent on the page (dwellTime) as a feature to the RF classifier
§ Performance evaluation:
• Precision / Recall
• Accuracy
• F-Measure
• AUC

22. Results

23. Attention
Precision Recall F-Measure Accuracy AUC
0
0.2
0.4
0.6
0.8
1
Attention
Click
Hover
DwellTime
Ours

24. Usefulness
Precision Recall F-Measure Accuracy AUC
0
0.2
0.4
0.6
0.8
1
Usefulness
Click
Hover
DwellTime
Ours

25. Perceived Task Duration
Precision Recall F-Measure Accuracy AUC
0
0.2
0.4
0.6
0.8
1
Task Duration
Click
Hover
DwellTime
Ours

26. Computational complexity
§ Mouse gesture techniques that rely on PCA
preprocessing and k-means clustering
• Covariance matrix computation + eigenvalue
decomposition ☞ O(p2N + p3)
• K-means ☞ O(icN)
§ Cursor Motifs that use Dynamic Time Warping
(DTW) and k-nearest neighbours (kNN)
• DTW ☞ O(N2)
• kNN ☞ O(N2k2)
§ Proposed method has linear ☞ O(N) or
quasilinear cost ☞ O(NlogN)

27. Conclusions
§ We conducted a crowdsourcing study that revealed the potential
benefits of using mouse cursor data to predict user
engagement with DDs
§ We demonstrated that our feature selection model outperforms
the standard baselines to measure three user engagement
proxies with the KM display
§ Our initial results suggest that it is possible to predict when the
user attention is captured by a DD using only simple, yet highly
discriminative features derived from mouse cursor activity

28. Conclusions (cont.)
§ Predicting accurately if a DD was truly noticed can:
• Increase the true negative prediction rate
• Reduce the false negative rate
§ Knowing when a user finds a DD useful has important
implications on the methodology for understanding the impact
of launching a new DD, modifying its existing design, and how
that change may affect search UIs

29. Conclusions (cont.)
§ Information about perceived task duration can be combined with
the previous grounds truths to understand better how users
engage with ads or multimedia content
§ The main practical use of our models is perhaps to
automatically select or lay out the DDs
§ DDs are optional for the SERPs and the user behaviour could
provide signals about whether DDs should be shown or not in
particular queries
§ Our method offers a computationally efficient way to analyse
mouse cursor data

30. Acknowledgments
§ We thank B. Barla Cambazoglu and Marios Koulakis for fruitful
discussions

31. Thank you for your attention!
iarapakis
arapakis.ioannis@gmail.com
https://es.linkedin.com/in/ioannisarapakis
http://www.slideshare.net/iarapakis/sigir16