Relating Eye-tracking Measures with changes in Knowledge On Search Tasks - ACM ETRA 2018
Relating Eye-Tracking Measures With Changes In Knowledge
On Search Tasks
Nilavra Bhattacharya Jacek Gwizdka
nilavra @ ieee.org etra2018 @ gwizdka.com
School of Information, The University of Texas at Austin
I N T R O D U C T I O N
Can human learning be inferred from eye movements? We
investigate whether differences in the knowledge-change of
online information searchers are reflected in their eye-
tracking measures related to reading. We conducted a lab-
based eye-tracking study, and examined the associations
between changes in knowledge of participants (measured in
a content-independent way), and their eye-gaze measures.
Our results show that participants with high knowledge-
change diﬀer significantly in terms of their total reading-
sequence-length, reading-sequence-duration, and number of
reading fixations, when compared to participants with low
R E S E A R C H Q U E S T I O N
Are the changes in verbal knowledge, from before to after a
search task, observable in eye-tracking measures?
M E T H O D
• 30 university students (16 women, 14 men)
• Good data for 26 participants
• Mean age 24.5 years
• Native English speakers with uncorrected 20/20 vision
• Within-subject design
• Eye-tracker: Tobii TX300
• Three information search tasks on health-related topics
• Two multi-faceted assigned tasks (A1, A2)
• One self-generated task (S)
• Google search with modified interface
• Calculated separately for each task:
• Independent: 2 knowledge-change measures
• Dependent: 6 eye-tracking measures related to reading
• Unit of analysis: participant-task pair
• Split into two groups – Lo and Hi – based on median
scores on knowledge-change measures
TA S K S
Assigned Task A1: Vitamin A
Your teenage cousin has asked your advice in regard to taking vitamin A for
health improvement purposes. You have heard conflicting reports about the
effects of vitamin A, and you want to explore this topic in order to help your
cousin. Specifically, you want to know:
1. What is the recommended dosage of vitamin A for underweight teenagers?
2. What are the health benefits of taking vitamin A? Please find at least 3
benefits and 3 disadvantages of vitamin A.
3. What are the consequences of vitamin A deficiency or excess? Please find 3
consequences of vitamin A deficiency and 3 consequences of its excess.
4. Please find at least 3 food items that are considered as good sources of
Assigned Task A2: Hypotension
Your friend has hypotension. You are curious about this issue and want to
investigate more. Specifically, you want to know:
1. What are the causes of hypotension?
2. What are the consequences of hypotension?
3. What are the differences between hypotension and hypertension in terms
of symptoms? Please find at least 3 differences in symptoms between
4. What are some medical treatments for hypotension? Which solution would
you recommend to your friend if he/she also has a heart condition? Why?
Example Self-generated ‘S’ task
Crohn’s disease: I know someone who was recently diagnosed, and am
curious about the disease.
R E S U L T S & D I S C U S S I O N
• We expected that people who did ‘more’ reading on
RELEVANT CONTENT pages would gain more topical
• However, we found that people scoring higher in our
knowledge-change scores (i.e. learning more) generally did
• Total fixation count of reading-sequences, and total duration
of reading-sequences were significantly smaller for the
respective Hi group, than the corresponding Lo group,
across both task types.
• For assigned tasks alone, total-regression-length was
significantly lower for the Hi group as well.
• Small number of similar tasks
• Choice of knowledge-change measures
• Investigate other factors which may cause differences in
• Examine different knowledge-change measures to better
reflect the learning process
Two kinds of webpages visited by participants: modified Google SERPs (left), and CONTENT pages (right)
RELEVANT CONTENT Page: A CONTENT page which the participants bookmarked and took notes from
(i.e. considered RELEVANT to the task). Analyses below consider only these kind of pages, as new
knowledge is most likely acquired from pages with which the participants interacted.
gauging existing knowledge
free-recall to identify knowledge-gain
Eye Tracking Measures per task (D.V.)
Rseq_N number of reading-sequences
total length of the (mostly horizontal) scan-paths
formed by joining the reading fixation points (px)
total count of reading fixations making up a
total duration of all fixations comprising reading
Reg_N total count of backward-regressions
Reg_px_tot total length of regressions (px)
Knowledge Change Measures per task (I.V.)
We used word-ranks of approximately 1/3 million most frequent English
words, taken from Google’s Web Trillion Word Corpus (https://goo.gl/2JZn7d)
𝑖𝑡𝑒𝑚𝑠 𝑝𝑜𝑠𝑡 − 𝑖𝑡𝑒𝑚𝑠 𝑝𝑟𝑒
Range of difference of the two knowledge-change
measures, across three tasks: A1, A2, S
Relating Eye Tracking with Knowledge Change measures
A1 A2 S
Lo 0 – 0.78 0 – 0.79 0 – 0.6
Hi 0.78 – 0.96 0.8 – 0.94 0.6 – 0.94
A1 A2 S
Lo 4 k – 8.8 k 4.9 k – 10.9 k 1.4 k – 9.2 k
Hi 9.4 k – 17.6 k 11.3 k – 22.1 k 9.6 k – 24 k
Funded, in part, by IMLS Career Development Grant #RE-04-11-0062-11 to Jacek Gwizdka
Post-task knowledge measurement
Fixation heatmap on a SERP page. The task-prompt is in top-left.
Fixation heatmap while taking notes.
Pre-task knowledge measurement
A simple line-oriented classifier labelled fixations as reading or scanning.
Reading fixations along a line make a reading sequence.
the no. of
after a task