This study examines how individual differences, user perceptions, and eye gaze relate to the use of different biomedical search interfaces. It finds that searchers look most at abstracts and that interfaces and perceptions of task difficulty affect what elements are viewed. There are also interaction effects between cognitive style/search experience and interfaces for Medical Subject Headings area of interest. The findings suggest design of search engine results pages should consider these factors and allow detection of task difficulty. Individual differences should also inform search interface design.

1. Individual Differences, User Perceptions and Eye
Gaze in Biomedical Search Interfaces
7 September 2015
Ying-Hsang Liu 1,2
1
School of Information Studies
Charles Sturt University
2
Research School of Computer Science
The Australian National University

2. 1Outline
Introduction
Research Questions
Interfaces
Research Design
Results
Summary and Discussion

3. 2Introduction
Interactive Information
Retrieval (IIR)
▶ Current IR systems designed for
specified search (Belkin, 2008)
▶ Natural search user
interfaces (Hearst, 2011)
▶ Usefulness of controlled indexing
languages (Salton, 1972; Svenonius, 1986)
Medical Subject Headings
(MeSH) terms

4. 3Research Questions
Research questions
▶ What elements of search
interfaces do searchers look at
when searching for documents
to answer complex questions?
▶ What is the relation between
user perceptions of an
interface and the interface
elements they look at?
▶ What is the relation between
individual differences and the
interface elements which are
looked at?
User experiment in a
laboratory setting

5. 4Interface A: Google style

6. 5Interface B: Per query, ProQuest

7. 6Interface C: Per query, ProQuest+EBSCOhost

8. 7Interface D: Per document, EBSCOhost

9. 8Test Collection
Selection of search topics
▶ Document test collection from
OHSUMED (Hersh, Buckley, Leone, & Hickam, 1994)
▶ MEDLINE from 1987 to 1991;
348,566 records
▶ Randomly select 8 topics based
on proportion of judged relevant
documents
▶ 2 topics from each of the
quartiles (4 search topic pairs)
Sample search topic
▶ ID: 78
▶ Imagine that you are 42-year-old
black man with hypertension.
▶ You would like to find
information about beta blockers
and blacks with hypertension,
utility.

10. 9Experimental Design
Factorial design
▶ 4 × 2 × 2 Factorial design; 4
interfaces, controlled search
topic pairs and cognitive styles
▶ 4 × 4 Graeco-Latin Square to
arrange experimental conditions
▶ Power Analysis for ANOVA
Design; medium effect size of
.25, α < .05 and N = 256,
statistical power of .93 (Cohen, 1988;
Faul, Erdfelder, Lang, & Buchner, 2007)
4 × 4 Graeco-Latin Square

11. 10Software and Hardware
Experimental system setup
▶ Experimental search system
based on Solr
▶ Gaze tracking uses FaceLab
software and hardware
▶ EyeWorks for data recording
and analysis
▶ Emotiv headset for EEG data
▶ Search logs and mouse clicks
recorded
Gaze tracking by FaceLab

12. 11Experimental Procedure
Experimental procedure Data collection
▶ User characteristics (background
questionnaire and cognitive style
test)
▶ User perceptions (exit
questionnaire)
▶ Search behaviours (search logs,
mouse clicks and documents
saved)
▶ Physiological signals (eye gaze
and EEG)

13. 12Searcher Characteristics
▶ 32 subjects; male (50.0%), female
(50.0%)
▶ Student: postgraduate (46.9%),
undergraudate (40.6%)
▶ Age: 18–25 (59.4%), 25–35
(28.1%)
▶ Online database experience: < 5
years (62.5%), 5–10 years
(21.9%)
▶ Search engine: every day
(50.0%), several times a day or
more (37.5%)
▶ Pilot study (Liu, Thomas, Schmakeit, & Gedeon, 2012)
Biology background

14. 13Searcher Characteristics (cont’d)
▶ Cognitive style: Individual’s
preference or tendency to
process information
▶ E-CSA-WA (Extended Cognitive
Style Analysis–Wholistic
Analytic) test (Peterson, Deary, & Austin, 2003)
▶ Wholistic Analytic Ratio
▶ WA ratio (M = 1.31, SD = .24);
cut-off = 1.32 (Clewley, Chen, & Liu, 2010; Chen,
Magoulas, & Macredie, 2004; Yuan, Zhang, Chen, & Avery, 2011)
E-CSA-WA Test

15. 14Data Analysis
▶ Where do people look? Area of
interest (AOI)
▶ Logarithmic cross ratio analysis
between individual
differences/user perceptions and
AOI (Fleiss, Levin, & Paik, 2003; Saracevic, Kantor, Chamis, &
Trivison, 1988)
▶ ANOVA between interface and
searcher characteristics, such as
cognitive style and search
experience
Heat map and AOI

16. 15Search Interfaces and AOI
Title Author Abstract MeSH
q
q
qq
q
q
q
q
qq q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
0
25
50
75
A B C D A B C D A B C D A B C D
Types of Interface
ProportionoffixationsinAOI

17. 16User Perceptions and AOI
Table: Summary of the relation between user perceptions and AOI
Difficulty Usefulness Notice of Keywords Use of Keywords
B C D B C D
Title H H G G G G G —
Author H — H H H H H H
Abstract H G — — — — — G
MeSH H — — — — — — —
Note. The relation is not statistically significant (—), positively significant (G), or
negatively significant (H) at 95%).

18. 17Individual Differences and AOI
Table: Summary of the relation between individual differences and AOI
Domain Knowledge Search Experience Cognitive Style
UG PG Search Engine Online
Database
Title H H — — —
Author — — G — —
Abstract — — H — —
MeSH — — G — —
Note. The relation is not statistically significant (—), positively significant (G), or
negatively significant (H) at 95%).

19. 18Interface and Search Experience Interaction

20. 19Interface and Cognitive Style Interaction

21. 20Summary and Discussion
Research findings
▶ Searchers look at abstract more
often than other interface
elements
▶ Interfaces and user perception of
search task difficulty significantly
affects elements look at
▶ Significant interaction effect
between cognitive style/search
experience and interface for
MeSH AOI
Discussion
▶ Design of Search Engine Results
Page (SERP)
▶ Detection of search task
difficulty
▶ Individual differences for search
user interface design

22. 21
Thank You!
Questions or
Comments?
This study is partially funded by 2014 ALIA Research Grant
Award, led by Dr Ying-Hsang Liu with Marijana Bacic (Monash
Health), Dr Paul Thomas (CSIRO) and Professor Tom
Gedeon (ANU).