1) The study examined whether changes in participants' knowledge after an online search task were reflected in their eye-tracking measures related to reading. 2) It found that participants who showed higher gains in knowledge assessments generally had fewer total fixations and shorter reading sequences compared to those with lower knowledge gains. 3) Limitations included the small number of similar tasks and choices of knowledge measurement, while future work could investigate other behavioral factors and knowledge measures.

1. Relating Eye-Tracking Measures With Changes In Knowledge
On Search Tasks
Nilavra Bhattacharya Jacek Gwizdka
nilavra @ ieee.org etra2018 @ gwizdka.com
School of Information, The University of Texas at Austin
I N T R O D U C T I O N
Can human learning be inferred from eye movements? We
investigate whether differences in the knowledge-change of
online information searchers are reflected in their eye-
tracking measures related to reading. We conducted a lab-
based eye-tracking study, and examined the associations
between changes in knowledge of participants (measured in
a content-independent way), and their eye-gaze measures.
Our results show that participants with high knowledge-
change differ significantly in terms of their total reading-
sequence-length, reading-sequence-duration, and number of
reading fixations, when compared to participants with low
knowledge-change.
R E S E A R C H Q U E S T I O N
Are the changes in verbal knowledge, from before to after a
search task, observable in eye-tracking measures?
M E T H O D
Participants
• 30 university students (16 women, 14 men)
• Good data for 26 participants
• Mean age 24.5 years
• Native English speakers with uncorrected 20/20 vision
Experiment Design
• Within-subject design
• Eye-tracker: Tobii TX300
• Three information search tasks on health-related topics
• Two multi-faceted assigned tasks (A1, A2)
• One self-generated task (S)
• Google search with modified interface
Measures
• Calculated separately for each task:
• Independent: 2 knowledge-change measures
• Dependent: 6 eye-tracking measures related to reading
• Unit of analysis: participant-task pair
• Split into two groups – Lo and Hi – based on median
scores on knowledge-change measures
TA S K S
Assigned Task A1: Vitamin A
Your teenage cousin has asked your advice in regard to taking vitamin A for
health improvement purposes. You have heard conflicting reports about the
effects of vitamin A, and you want to explore this topic in order to help your
cousin. Specifically, you want to know:
1. What is the recommended dosage of vitamin A for underweight teenagers?
2. What are the health benefits of taking vitamin A? Please find at least 3
benefits and 3 disadvantages of vitamin A.
3. What are the consequences of vitamin A deficiency or excess? Please find 3
consequences of vitamin A deficiency and 3 consequences of its excess.
4. Please find at least 3 food items that are considered as good sources of
vitamin A.
Assigned Task A2: Hypotension
Your friend has hypotension. You are curious about this issue and want to
investigate more. Specifically, you want to know:
1. What are the causes of hypotension?
2. What are the consequences of hypotension?
3. What are the differences between hypotension and hypertension in terms
of symptoms? Please find at least 3 differences in symptoms between
them.
4. What are some medical treatments for hypotension? Which solution would
you recommend to your friend if he/she also has a heart condition? Why?
Example Self-generated ‘S’ task
Crohn’s disease: I know someone who was recently diagnosed, and am
curious about the disease.
R E S U L T S & D I S C U S S I O N
Findings
• We expected that people who did ‘more’ reading on
RELEVANT CONTENT pages would gain more topical
knowledge.
• However, we found that people scoring higher in our
knowledge-change scores (i.e. learning more) generally did
less reading.
• Total fixation count of reading-sequences, and total duration
of reading-sequences were significantly smaller for the
respective Hi group, than the corresponding Lo group,
across both task types.
• For assigned tasks alone, total-regression-length was
significantly lower for the Hi group as well.
Limitations
• Small number of similar tasks
• Choice of knowledge-change measures
Future Work
• Investigate other factors which may cause differences in
behaviour
• Examine different knowledge-change measures to better
reflect the learning process
Two kinds of webpages visited by participants: modified Google SERPs (left), and CONTENT pages (right)
book-marking
and
note-taking
area
RELEVANT CONTENT Page: A CONTENT page which the participants bookmarked and took notes from
(i.e. considered RELEVANT to the task). Analyses below consider only these kind of pages, as new
knowledge is most likely acquired from pages with which the participants interacted.
Pre Task
Questionnaire
gauging existing knowledge
Task
Session
Post Task
Questionnaire
free-recall to identify knowledge-gain
Next
Task
Eye Tracking Measures per task (D.V.)
Rseq_N number of reading-sequences
Rseq_px_tot
total length of the (mostly horizontal) scan-paths
formed by joining the reading fixation points (px)
Rseq_fixn_ct_tot
total count of reading fixations making up a
reading sequence
Rseq_dur_tot
total duration of all fixations comprising reading
sequences (ms)
Reg_N total count of backward-regressions
Reg_px_tot total length of regressions (px)
Knowledge Change Measures per task (I.V.)
We used word-ranks of approximately 1/3 million most frequent English
words, taken from Google’s Web Trillion Word Corpus (https://goo.gl/2JZn7d)
𝑟𝑒𝑙_𝑐ℎ𝑎𝑛𝑔𝑒_𝑖𝑛_𝑖𝑡𝑒𝑚𝑠 =
𝑖𝑡𝑒𝑚𝑠 𝑝𝑜𝑠𝑡 − 𝑖𝑡𝑒𝑚𝑠 𝑝𝑟𝑒
𝑖𝑡𝑒𝑚𝑠 𝑝𝑜𝑠𝑡
𝑚𝑒𝑎𝑛_𝑟𝑎𝑛𝑘_𝑃𝑂𝑆𝑇_𝑛𝑜𝑢𝑛𝑠 =
σ𝑖=1
𝑛
𝑟𝑎𝑛𝑘𝑖
𝑛
Range of difference of the two knowledge-change
measures, across three tasks: A1, A2, S
Relating Eye Tracking with Knowledge Change measures
A1 A2 S
Lo 0 – 0.78 0 – 0.79 0 – 0.6
Hi 0.78 – 0.96 0.8 – 0.94 0.6 – 0.94
A1 A2 S
Lo 4 k – 8.8 k 4.9 k – 10.9 k 1.4 k – 9.2 k
Hi 9.4 k – 17.6 k 11.3 k – 22.1 k 9.6 k – 24 k
Any
Funded, in part, by IMLS Career Development Grant #RE-04-11-0062-11 to Jacek Gwizdka
Post-task knowledge measurement
Fixation heatmap on a SERP page. The task-prompt is in top-left.
Fixation heatmap while taking notes.
Pre-task knowledge measurement
A simple line-oriented classifier labelled fixations as reading or scanning.
Reading fixations along a line make a reading sequence.
Relative
change in
the no. of
phrases
entered
before and
after a task
Mean
rank of
nouns
entered
after a
task