Individual Differences, User Perceptions and Eye Gaze in Biomedical Search Interfaces

1. Individual Differences, User Perceptions and Eye
Gaze in Biomedical Search Interfaces
7 September 2015
Ying-Hsang Liu 1,2
1
School of Information Studies
Charles Sturt University
2
Research School of Computer Science
The Australian National University

2. 1Outline
Introduction
Research Questions
Interfaces
Research Design
Results
Summary and Discussion

3. 2Introduction
Interactive Information
Retrieval (IIR)
▶ Current IR systems designed for
specified search (Belkin, 2008)
▶ Natural search user
interfaces (Hearst, 2011)
▶ Usefulness of controlled indexing
languages (Salton, 1972; Svenonius, 1986)
Medical Subject Headings
(MeSH) terms

4. 3Research Questions
Research questions
▶ What elements of search
interfaces do searchers look at
when searching for documents
to answer complex questions?
▶ What is the relation between
user perceptions of an
interface and the interface
elements they look at?
▶ What is the relation between
individual differences and the
interface elements which are
looked at?
User experiment in a
laboratory setting

5. 4Interface A: Google style

6. 5Interface B: Per query, ProQuest

7. 6Interface C: Per query, ProQuest+EBSCOhost

8. 7Interface D: Per document, EBSCOhost

9. 8Test Collection
Selection of search topics
▶ Document test collection from
OHSUMED (Hersh, Buckley, Leone, & Hickam, 1994)
▶ MEDLINE from 1987 to 1991;
348,566 records
▶ Randomly select 8 topics based
on proportion of judged relevant
documents
▶ 2 topics from each of the
quartiles (4 search topic pairs)
Sample search topic
▶ ID: 78
▶ Imagine that you are 42-year-old
black man with hypertension.
▶ You would like to find
information about beta blockers
and blacks with hypertension,
utility.

10. 9Experimental Design
Factorial design
▶ 4 × 2 × 2 Factorial design; 4
interfaces, controlled search
topic pairs and cognitive styles
▶ 4 × 4 Graeco-Latin Square to
arrange experimental conditions
▶ Power Analysis for ANOVA
Design; medium effect size of
.25, α < .05 and N = 256,
statistical power of .93 (Cohen, 1988;
Faul, Erdfelder, Lang, & Buchner, 2007)
4 × 4 Graeco-Latin Square

11. 10Software and Hardware
Experimental system setup
▶ Experimental search system
based on Solr
▶ Gaze tracking uses FaceLab
software and hardware
▶ EyeWorks for data recording
and analysis
▶ Emotiv headset for EEG data
▶ Search logs and mouse clicks
recorded
Gaze tracking by FaceLab

12. 11Experimental Procedure
Experimental procedure Data collection
▶ User characteristics (background
questionnaire and cognitive style
test)
▶ User perceptions (exit
questionnaire)
▶ Search behaviours (search logs,
mouse clicks and documents
saved)
▶ Physiological signals (eye gaze
and EEG)

13. 12Searcher Characteristics
▶ 32 subjects; male (50.0%), female
(50.0%)
▶ Student: postgraduate (46.9%),
undergraudate (40.6%)
▶ Age: 18–25 (59.4%), 25–35
(28.1%)
▶ Online database experience: < 5
years (62.5%), 5–10 years
(21.9%)
▶ Search engine: every day
(50.0%), several times a day or
more (37.5%)
▶ Pilot study (Liu, Thomas, Schmakeit, & Gedeon, 2012)
Biology background

14. 13Searcher Characteristics (cont’d)
▶ Cognitive style: Individual’s
preference or tendency to
process information
▶ E-CSA-WA (Extended Cognitive
Style Analysis–Wholistic
Analytic) test (Peterson, Deary, & Austin, 2003)
▶ Wholistic Analytic Ratio
▶ WA ratio (M = 1.31, SD = .24);
cut-off = 1.32 (Clewley, Chen, & Liu, 2010; Chen,
Magoulas, & Macredie, 2004; Yuan, Zhang, Chen, & Avery, 2011)
E-CSA-WA Test

15. 14Data Analysis
▶ Where do people look? Area of
interest (AOI)
▶ Logarithmic cross ratio analysis
between individual
differences/user perceptions and
AOI (Fleiss, Levin, & Paik, 2003; Saracevic, Kantor, Chamis, &
Trivison, 1988)
▶ ANOVA between interface and
searcher characteristics, such as
cognitive style and search
experience
Heat map and AOI

16. 15Search Interfaces and AOI
Title Author Abstract MeSH
q
q
qq
q
q
q
q
qq q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
0
25
50
75
A B C D A B C D A B C D A B C D
Types of Interface
ProportionoffixationsinAOI

17. 16User Perceptions and AOI
Table: Summary of the relation between user perceptions and AOI
Difficulty Usefulness Notice of Keywords Use of Keywords
B C D B C D
Title H H G G G G G —
Author H — H H H H H H
Abstract H G — — — — — G
MeSH H — — — — — — —
Note. The relation is not statistically significant (—), positively significant (G), or
negatively significant (H) at 95%).

18. 17Individual Differences and AOI
Table: Summary of the relation between individual differences and AOI
Domain Knowledge Search Experience Cognitive Style
UG PG Search Engine Online
Database
Title H H — — —
Author — — G — —
Abstract — — H — —
MeSH — — G — —
Note. The relation is not statistically significant (—), positively significant (G), or
negatively significant (H) at 95%).

19. 18Interface and Search Experience Interaction

20. 19Interface and Cognitive Style Interaction

21. 20Summary and Discussion
Research findings
▶ Searchers look at abstract more
often than other interface
elements
▶ Interfaces and user perception of
search task difficulty significantly
affects elements look at
▶ Significant interaction effect
between cognitive style/search
experience and interface for
MeSH AOI
Discussion
▶ Design of Search Engine Results
Page (SERP)
▶ Detection of search task
difficulty
▶ Individual differences for search
user interface design

22. 21
Thank You!
Questions or
Comments?
This study is partially funded by 2014 ALIA Research Grant
Award, led by Dr Ying-Hsang Liu with Marijana Bacic (Monash
Health), Dr Paul Thomas (CSIRO) and Professor Tom
Gedeon (ANU).