This document discusses using neuroscience techniques like eye tracking and fMRI to study information science constructs. It outlines two current projects - one using eye tracking to model reading patterns and measure cognitive effort during information searching, and another using fMRI and eye tracking to study neural correlates of information relevance judgments. The eye tracking project analyzes fixation patterns to infer mental states and predicts how measures like reading speed and regressions relate to task characteristics. The fMRI study aims to identify different brain regions activated during relevant vs. non-relevant information searches and low-level word matching. Preliminary analysis of one participant shows differences in activation for an information search vs. word search task.

1. Towards
Neuro–Information Science
Jacek Gwizdka & Michael Cole
iSchool @ Rutgers University, NJ, USA
jacek@neuroinfoscience.org
http://jsg.tel
June 5, 2012

2. Information Science
 Another IS 
 Information Science is about :
◦ understanding information seeking behavior (why/how/where/…)
◦ helping people find information they need
2

3. Information Systems vs. iScience
 A lot of common concerns and constructs:
◦ information is digital  accessed via information systems
◦ technology – task – individual
◦ IT usefulness, user interface design, usability …
◦ trust …
◦ decision making …
◦ affective and cognitive factors
◦ information search (e.g., stopping behavior) …
3

4. Information Systems vs. iScience
 Also new opportunities:
◦ neural-correlates of constructs specific to Information Science
◦ Information Relevance : most commonly refers to topical relevance
or aboutness, that is: to what extent the content of a search result
matches the topic of the query or a person’s information need (e.g.,
Saracevic, 2007)
 relevance judgment  decision making
  information stopping
4

5. Opportunities for Neuroscience to Inform IS
Seven opportunities for cognitive neuroscience to inform IS research:
1. localize the neural correlates of IS constructs to better understand
their nature and dimensionality;
2. complement existing sources of IS data with neuroscientific data;
3. capture hidden (automatic) processes that are difficult to measure
with existing measurement methods;
4. identify antecedents of IS constructs by exploring the specifics of
how IT stimuli (e.g., the design of graphical user interfaces) are
processed by the brain;
5. test the outcomes of IS constructs by showing how brain activation
predicts behavior (e.g., decisions);
6. infer causality among IS constructs by examining the timing of brain
activations due to a common stimulus;
7. challenge existing IS assumptions and enhance IS theories that do
not correspond to the brain’s functionality
(Dimoka et al. 2010)
5

6. Neuroscience and Information Science?
 Eye-tracking +++
 Galvanic skin response (GSR) ++
 Heart –rate variability (HRV) +
 EEG
 fNIRS
 fMRI

7. Recent and Current Projects
1. eye-tracking: modeling reading + cognitive effort
2. fMRI + eye-tracking: information relevance
7

8. Part I: Eye-tracking
 General research goal: infer and predict mental states and
context of a person engaged in interactive information
searching
 Influence system design  adaptive systems
Macro
user
task
characteris�cs,
cogni�ve
effort,
domain
knowledge
Meso
reading
pa�erns
Micro
eye-­‐gaze
posi�ons
+
�ming
8

9. Eye-movement Presentation
9

10. Eye-tracking Data à Patterns
State1
State2
State3
10

11. Eye-movement Patterns
 New methodology to analyze eye-movement patterns
◦ Model reading and Measure cognitive effort
◦ Correlate with higher-level constructs
user task characteristics,
user knowledge, etc.
11

12. Reading Model Origins
 Based on E-Z Reader model
Rayner , Pollatsek, Reichle
◦ Serial reading
◦ Words can be identified in parafovial region
◦ Early lexical access (word familiarity) + Complete lexical processing (word identification)
2o (70px) foveal region parafoveal region
MORE… 12

13. Two-State Reading Model
q
isolated
fixation fixations
sequences
p
Read
Scan
1-p 1-q
◦ Filter fixations < 150ms (min time required for lexical processing)
◦ Model states characterized by:
 probability of transitions; number of lexical fixations; duration
 length of eye-movement trajectory, amount of text covered
MORE… 13

14. Example Reading Sequence
Reading sequence:
Fixation model states: (F F F) S (F R F) S S S S (F F FR F F) F
R F F FFFF F S
Reading state – R | Scanning state – S
14

15. Cognitive Effort Measures of Reading
 Reading Speed
foveal region
regression
 Fixation Regression
a b c d
 Perceptual Span
Perceptual span = Mean(a,b,c,d)
 Fixation Duration excess
(“lexical processing excess”)
15

16. User Study 1: Cognitive Effort and Tasks
Journalists’
Information Search
OBI:
advanced
obituary
INT:
interview
prepara�on
CPE:
copy
edi�ng
BIC:
background
informa�on
N = 32
 Do the cognitive effort measures correlate with:
task difficulty (by design), observable search effort,
user’s subjective perception of task difficulty
 Can
we detect differences between task characteristics from
eye-movement patterns?
MORE…
16

17. Eye-data and Cognitive Effort Measures
Subjective Task
Cognitive effort measures Difficulty
Task difficulty derived from eye-tracking
by design reading speed
Copy Editing (CPE) mean fixation duration
Advance Obituary (OBI) perceptual span
total fixation regressions
CPE
INT
BIC
OBI
As expected:
Copy Editing CPE easiest
Search effort Advance Obituary OBI most difficult
task time Sig: Kruskal-Wallis χ2 =46.1, p<.0001
pages visited
queries entered
17

18. Eye-data and Task Characteristics
q
p
Read
Scan
1-p 1-q
Interview
prepara�on Copy
Edi�ng
Measure
Related
Task
Characteris�cs
Frequency
Advanced
obituary
and
Interview
prepara�on
tasks:
SR bias
to
read
search
for
document;
task
goal
not
specific
of
reading
state
Copy
Edi�ng
task:
search
for
segment
and
task
goal
transi�ons
RS bias
to
scan
specific
MORE…
18

19. Summary: Eye-tracking Methodology
 Domain independent
◦ Document content is not involved
 Culturally*and individually independent
 Real-time modeling of user and tasks is possible
 Adaptive systems feasible
 Eye-tracking is coming to us!
Tobii
19

20. Part II: Current fMRI+eye-tracking Study
 Information Relevance : refers to topical relevance
or aboutness, that is: to what extent the content of a document
(webpage) matches the topic of the query or a person’s
information need (e.g., Saracevic, 2007)
◦ Relevance multi-dimensional: topical, meaningful, useful, trust, affective…
 Neuralcorrelates of topical relevance judgments
 Hypothesis
◦ Brain regions that are activated when relevant information is found are
different from regions activated when no relevant info is found and when
person does “low-level” visual word search (orthographic matching)
 but no hypothesis in a sense where the brain activity is located
 Exploratory research
 (also: a similar experiment with eye-tracking, EEG, GSR)
20

21. fMRI + eye-tracking lab
 Lab Equipment:
◦ fMRI: 3T Siemens TRIO
◦ eye-tracker: Eyelink-1000
 non-ferromagnetic optimized design; up to 2000 Hz sampling rate
21

22. fMRI + eye-tracking
22

23. fMRI + eye-tracking
projected
screen
mirror
eye-tracker
Eye-tracking imposes additional constraints on projection (geometry)
23

24. Current Experimental Design
 Two blocks (types of tasks, balanced)
◦ WS – word search: find target word in a short news story – press yes/no
◦ IS – information search: find information that answers given question –
press yes/no. Three types of trials: relevant (R), topical (T), irrelevant (I)
◦ TR cycle: 2s
21 x
30s 4s 6s 4s 20s max
xmx ssms nsns snsns
4s
WS task
target:
jsdjsd djdjd djdj dkke ekek
+ + +
kdkddk dkdkdk dkdkdkd
instruc- kkdkd d d dd d djdj djdjdj
word
rjrjr rjr jweje ejejej ejej
tions kekekek ekeke wej e eej
eje j
21 x
30s 4s 8s + 20s max + 4s + 20s max
xmx ssms nsns snsns
+ 4s + 20s max +
IS task
xmx ssms nsns snsns
target: target: target:
jsdjsd ke ekek dkdkdkkd jsdjsd djdjd djdj dkke ekek xmx ssms nsns snsns
+
dkdkdkkd jsdjsd ke ekek dkdkdkk
kdkddk dkdkdk dkdkdkd kdkddk dkdkdkdkdkdkd
instruc- kkdkd d rjr jweje ejejej kdkddk dkdkdk dkdkdkd kkdkd d rjr jweje ejeje
info info info
ejej kkdkd d d dd d djdj djdjdj
tions
rjrjr rjjweje ejejej ejej ekeke wej e ejej fjfjf fjfjfjfjf
kekekek ekeke wee ejej fjfjrjr rreje j
fjfjf fjfjfjfjf fjfjrjr rreje j kek ekeke wej e ejej eje j
24

25. Planned Analysis
 Two blocks (types of tasks, balanced)
◦ WS – word search: find target word in a short news story
◦ IS – information search: find information that answers given question –
Three types of trials: relevant (R), topical (T), irrelevant (I)
 The main contrasts of interests are:
◦ IS-R - WS
◦ IS-R - IS-T
◦ IS-R - IS-I
25

26. A Very, Very Preliminary Analysis
 For
one participant, aggregated for all trials in each of two
blocks (tasks)
 Word search (WS)
 Information Search – Relevant (IS-R)
26

27. Stay Tuned for Results…

28. Neuro – Information Science
Acknowledgements: Funding: Google, HP, IMLS (now funded by IMLS CAREER)
Collaborators: Drs. Nicholas Belkin, Art Chaovalitwongse (U Wash), Xiangmin Zhang,
Ralf Bierig (Post Doc); PhD students: Michael Cole (co-author), Chang Liu, Jingjing Liu, Irene Lopatovska
+ many Master and undergraduate students …
28

29. Fragen?
More info & contact http://jsg.tel
29