Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Relating Eye-tracking Measures with changes in Knowledge On Search Tasks - ACM ETRA 2018


Published on

Poster of our paper at the ACM ETRA 2018 @ Warsaw, Poland.
Full paper present at

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Relating Eye-tracking Measures with changes in Knowledge On Search Tasks - ACM ETRA 2018

  1. 1. Relating Eye-Tracking Measures With Changes In Knowledge On Search Tasks Nilavra Bhattacharya Jacek Gwizdka nilavra @ etra2018 @ School of Information, The University of Texas at Austin I N T R O D U C T I O N Can human learning be inferred from eye movements? We investigate whether differences in the knowledge-change of online information searchers are reflected in their eye- tracking measures related to reading. We conducted a lab- based eye-tracking study, and examined the associations between changes in knowledge of participants (measured in a content-independent way), and their eye-gaze measures. Our results show that participants with high knowledge- change differ significantly in terms of their total reading- sequence-length, reading-sequence-duration, and number of reading fixations, when compared to participants with low knowledge-change. R E S E A R C H Q U E S T I O N Are the changes in verbal knowledge, from before to after a search task, observable in eye-tracking measures? M E T H O D Participants • 30 university students (16 women, 14 men) • Good data for 26 participants • Mean age 24.5 years • Native English speakers with uncorrected 20/20 vision Experiment Design • Within-subject design • Eye-tracker: Tobii TX300 • Three information search tasks on health-related topics • Two multi-faceted assigned tasks (A1, A2) • One self-generated task (S) • Google search with modified interface Measures • Calculated separately for each task: • Independent: 2 knowledge-change measures • Dependent: 6 eye-tracking measures related to reading • Unit of analysis: participant-task pair • Split into two groups – Lo and Hi – based on median scores on knowledge-change measures TA S K S Assigned Task A1: Vitamin A Your teenage cousin has asked your advice in regard to taking vitamin A for health improvement purposes. You have heard conflicting reports about the effects of vitamin A, and you want to explore this topic in order to help your cousin. Specifically, you want to know: 1. What is the recommended dosage of vitamin A for underweight teenagers? 2. What are the health benefits of taking vitamin A? Please find at least 3 benefits and 3 disadvantages of vitamin A. 3. What are the consequences of vitamin A deficiency or excess? Please find 3 consequences of vitamin A deficiency and 3 consequences of its excess. 4. Please find at least 3 food items that are considered as good sources of vitamin A. Assigned Task A2: Hypotension Your friend has hypotension. You are curious about this issue and want to investigate more. Specifically, you want to know: 1. What are the causes of hypotension? 2. What are the consequences of hypotension? 3. What are the differences between hypotension and hypertension in terms of symptoms? Please find at least 3 differences in symptoms between them. 4. What are some medical treatments for hypotension? Which solution would you recommend to your friend if he/she also has a heart condition? Why? Example Self-generated ‘S’ task Crohn’s disease: I know someone who was recently diagnosed, and am curious about the disease. R E S U L T S & D I S C U S S I O N Findings • We expected that people who did ‘more’ reading on RELEVANT CONTENT pages would gain more topical knowledge. • However, we found that people scoring higher in our knowledge-change scores (i.e. learning more) generally did less reading. • Total fixation count of reading-sequences, and total duration of reading-sequences were significantly smaller for the respective Hi group, than the corresponding Lo group, across both task types. • For assigned tasks alone, total-regression-length was significantly lower for the Hi group as well. Limitations • Small number of similar tasks • Choice of knowledge-change measures Future Work • Investigate other factors which may cause differences in behaviour • Examine different knowledge-change measures to better reflect the learning process Two kinds of webpages visited by participants: modified Google SERPs (left), and CONTENT pages (right) book-marking and note-taking area RELEVANT CONTENT Page: A CONTENT page which the participants bookmarked and took notes from (i.e. considered RELEVANT to the task). Analyses below consider only these kind of pages, as new knowledge is most likely acquired from pages with which the participants interacted. Pre Task Questionnaire gauging existing knowledge Task Session Post Task Questionnaire free-recall to identify knowledge-gain Next Task Eye Tracking Measures per task (D.V.) Rseq_N number of reading-sequences Rseq_px_tot total length of the (mostly horizontal) scan-paths formed by joining the reading fixation points (px) Rseq_fixn_ct_tot total count of reading fixations making up a reading sequence Rseq_dur_tot total duration of all fixations comprising reading sequences (ms) Reg_N total count of backward-regressions Reg_px_tot total length of regressions (px) Knowledge Change Measures per task (I.V.) We used word-ranks of approximately 1/3 million most frequent English words, taken from Google’s Web Trillion Word Corpus ( 𝑟𝑒𝑙_𝑐ℎ𝑎𝑛𝑔𝑒_𝑖𝑛_𝑖𝑡𝑒𝑚𝑠 = 𝑖𝑡𝑒𝑚𝑠 𝑝𝑜𝑠𝑡 − 𝑖𝑡𝑒𝑚𝑠 𝑝𝑟𝑒 𝑖𝑡𝑒𝑚𝑠 𝑝𝑜𝑠𝑡 𝑚𝑒𝑎𝑛_𝑟𝑎𝑛𝑘_𝑃𝑂𝑆𝑇_𝑛𝑜𝑢𝑛𝑠 = σ𝑖=1 𝑛 𝑟𝑎𝑛𝑘𝑖 𝑛 Range of difference of the two knowledge-change measures, across three tasks: A1, A2, S Relating Eye Tracking with Knowledge Change measures A1 A2 S Lo 0 – 0.78 0 – 0.79 0 – 0.6 Hi 0.78 – 0.96 0.8 – 0.94 0.6 – 0.94 A1 A2 S Lo 4 k – 8.8 k 4.9 k – 10.9 k 1.4 k – 9.2 k Hi 9.4 k – 17.6 k 11.3 k – 22.1 k 9.6 k – 24 k Any Funded, in part, by IMLS Career Development Grant #RE-04-11-0062-11 to Jacek Gwizdka Post-task knowledge measurement Fixation heatmap on a SERP page. The task-prompt is in top-left. Fixation heatmap while taking notes. Pre-task knowledge measurement A simple line-oriented classifier labelled fixations as reading or scanning. Reading fixations along a line make a reading sequence. Relative change in the no. of phrases entered before and after a task Mean rank of nouns entered after a task