In a real world visual search is a common task depending from sensory, perceptual and cognitive processes. Different classes of eye movements are necessary to hold an image on the retina during head rotation or movement of the image, and to move the eye suddenly to a new point of interest in space. From a functional point of view, two major classes of eye movements are described in humans: those stabilizing gaze (optokinetic nystagmus, oculovestibular reflex) and those movinggaze (saccades, pursuits and vergence). Under natural conditions, however, a mix of all kinds of eye movements permit continuous scanning of the visual scene. The sequence of fixations and saccades during visual exploration isan expression of a number of cognitive processes; the use of standardized tasks with pre-defined spatial-temporal variables allows us to assess specific cognitive domains, such as perception, attention, memory, preference and motivation. Manipulating the search task can vary the demands on brain. In turn, brain modulates visual search by selecting and limiting the information available at various levels of processing. The EVA software is a complete system based on a set of stimulus and patient’s case able to stress brain functionalities in order to assess some cognitive functions.

1. EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI
AUTOMATIC CASE
BASE SYSTEM
EYE TRACKING & VISION APPLICATIONS LAB (EVA
LAB) – DEPARTMENT OF NEUROLOGICAL
NEUROSURGICAL AND BEHAVIORAL SCIENCE –
UNIVERSITY OF SIENA
Giacomo Veneri, Pamela Federighi, Elena Pretegiani, Francesca Rosini, Nicola R Polizzotto, Francesco Fragnoli,
Emiliano Santarnecchi, Giacomo DeMurtas, Antonio Federico & Alessandra Rufa
www.evalab.unisi.it – visionlab@unisi.it
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena

2. EVA Light System
2
EVA
EYE TRACKING -
STIMULUS
INTEGRATED SEMI
AUTOMATIC CASE
BASESYSTEM
To test the cognitive
system by multi-stimulus
Diagnosis by Case Base
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

3. Summary
3
Background (2m)  Analysis (5m)
 Testing  Saccade: Velocity,Amplitude
the cognitive
functionalities by gaze Duration, Latency, Error
 Fixation: dispersion
 Stimulus Architecture (5m)
 Task: sequencing
 Saccadic Behavior
 Decision Making
 Visual Search
 Results (5m)
 Gaze Contingent
 TMS Interference  Case Base (1m)
 Sound integration and
joystick
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

4. Background
4
“BY TESTING THE RESPONSE TO VISUAL STIMULI WE CAN TEST
MORE THAN 60% OF BRAIN ACTIVITY”
www.evalab.unisi.it – visionlab@unisi.it
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI
AUTOMATIC CASE BASE SYSTEM 01/07/13

5. Vision
5
“More 60% of our Barin
works on vision …”:
 Image Processing
 Spatial Attention
 Working memory
 Memory
 Visual-Spatial exploration
 Eye Movement Command
 Decision Making The focus of attention is jointly determined by stimulus
salience and top-down control signals. IOR is implemented by
a mechanism that reduces the salience of stimuli at recently
explored spatial locations.
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

6. Eye Movement & Attention
6
Eye movements
 Sequences of fixation and
saccade
 The sequences of saccade-
and-fixation enable humans
to receive a continuous flow
of information from the world. Peripheral attention
Saccadic eye movements
Attention
Overlapped region
 Attention is the process of
selecting a few relevant
perceptive stimuli by filtering
of negligible aspect of the
environment
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

7. Working Memory & Visual Search
7
Visual working memory :
 is a theoretical construct
that refers to the
structures and processes
used for temporarily
storing and manipulating
visual information.
information
Visual Search
 memory maintains the
information in order to
allow past experience to
guide future behaviors
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

8. Decision making
8
Decision making:
 can be regarded as an
outcome of mental
processes (cognitive
process) leading to the
selection of a course of
action among several
alternatives.
Basic connectivity framework. Dotted arrows represent external
inputs to the model. Abbreviations: 5-HT, dorsal raphe
serotonergic neurons; ACC, anterior cingulate cortex; AMYG,
amygdala; DA, midbrain dopaminergic neurons; DLPFC,
dorsolateral prefrontal cortex; OFC, orbitofrontal cortex; VS,
ventral striatum.
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

9. The Main Idea: 3 steps
9
1. To develop a multistimlus
system able to stress some
part and functionalities of
brain in order to make a low
cost pre-diagnosis of patient.
1. Interference: TMS, sound
2. Sensory stimulus: video, audio
3. Output: Eyetracking, joystick
2. To develop indicators from
specific task in order to test
the brain response.
3. To develop a database of
case which can suggest
dignosis avoiding any euristic
or expert system logic.
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

10. Preview: clinical case & research case
10
Clinical case
 Patients with movement disorder including cerebellar patients.
 Neo Cortex injury.
 Neglect patients (parietal abnormalities).
 Frontal prefrontal lesions
 Pre and post pharmacological assumption.
 Deficit on attention & dyslexic patients.
Research case
 Temporary interference by TMS
 Decision Making (NeuroEconomy – Dept. Economic)
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

11. Task
11
SACCADES, MEMORY GUIDED Saccade, TMT, SOUND GUIDED
visual exploration or saccades, GAZE CONTINGENT, FACE
ANALYSIS, GAME THEORY.
www.evalab.unisi.it – visionlab@unisi.it
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI
AUTOMATIC CASE BASE SYSTEM 01/07/13

12. Stimulus System
12
Saccade Visual Search: time line
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

13. Saccade / Anti saccade / Memory guided
13
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

14. Trail Making Task
14
Visual Search depends on
sensory, perceptual and
cognitive processes
TMTB is used to measure
the cognitive domains of
processing speed,
sequencing, visual search
abilities and visual-motor
skills.
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

15. Gaze Contingent
15
Reactive gaze-contingent
display (GCD), which
uses a simple hole
forcing only foveal vision
during exploration by
blinding peripheral vision,
in order to test and
stress overt mechanisms
of visual search in
normal subjects and
clinical contexts
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

16. Decision Making
16
Game Theory
The subject was asked
to choose between two
sign according to
previous chooses
 Overconfident
 Bayesian
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

17. Analysis
17
INDICATORS
www.evalab.unisi.it – visionlab@unisi.it
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

18. Features
18
Common New
Saccades  Saccades
 Velocity Peak
Going from/in ROI

 Mean Velocity
 Time
Fixations  Delta time among ROI
 N° fixations in ROI
(step to step time)
 Fixations Duration
 Errors
Time
 ROI re-exploration
 Task duration
 Sequencing 1 2
 Time in ROI
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

19. Algorithms
19
Signal processing
 Spurious data purging
 Saccade detection
 Fixations identification
Visual Search indicator
 Best Squence [BestSequenceAlgorithm]
 Fixations distance
 8PS Model
Decision Indicator
 Likelihood function
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

20. Best Sequence Algorithm
20
Is a tree exploration
algorithm that extract the
Sequencing Index as an
indicator of the ability in
gazing correctly a logic
sequence
 Subject 1:
O1A2B3C4BD5EE5O5A1
AB52
 Patient 1:
O13C4BD5E2ABE5OAE1
A12B
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

21. SCA2 (subject 1) vs CTRL (subject 2)
21
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

22. Fixation Distance From ROI
22
The distance from
nearest ROI
 It’s an indicator of fixation
dispersion during visual
exploration
fs (xf,yf))**minROI ((δ((xf,yf), (xr,yr)))
fs (xf,yf minROI δ((xf,yf), (xr,yr)))
The distance from target
 It’s an indicator of goal
directed visual exploration
fs (xf,yf))**((δ((xf,yf), (xr r(t+1),yr r(t+1))))
fs (xf,yf δ((xf,yf), (x (t+1),y (t+1))))
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

23. 8Pointed Star -Model
23
Direction Error (ε*) = (direction observed (ο*) – expected (η*) )
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

24. Ongoing Search abilities
24
Direction error may
provide a quantitative
assessment of the
visual search strategy
provides an
estimation of the
ongoing visual search
capabilities.
Δεε*= εεt*t-- εε*t-1
Δ* = * *
t-1
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
Using Simplified Trial Making Test B to Evaluate Ongoing Learning Capabilities on Visual Search
01/07/13

25. Upcoming direction versus previous fixations
25
The forthcoming
direction minus the
minimum/maximum
distribution of fixation
(ο*)) – min(direction(fixations))
(ο* – min(direction(fixations))
It’s an expression to plan
the saccade far from
visited location (fixated)
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

26. Decision: likelihood function
26
Likelihood function was expressed as the probability
that a subject spends more time looking the object
which will be choose during the last 3 seconds
preceding the explicit decision.
Time on sign / /Time total elapsed
Time on sign Time total elapsed
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

27. Results
27
PATIENTS & SUBJECTS
www.evalab.unisi.it – visionlab@unisi.it
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI
AUTOMATIC CASE BASE SYSTEM 01/07/13

28. Movement Disorder Saccade Analysis
28
Saccade Cerebellar patients
 Three group were analyzed
600
 Control (healthy subject)
500
 SCA2 (Autosomal Dominant
400
Cerebellar Atrophiy)
 pCA (Autosomal Recessive
300 CTRL
pCA
Pure Cerebellar Atrophy)
SCA2
200
 Significant differences were
found on
100
 Peak Velocity, Duration. Amplitude
 Gain
0
10° 18°
Peak Velocity (°/s)
20° 10° 18°
Mean Velocity (°/s)
20° 10° 18°
Duration (s)
20°
 Error
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

29. Visual Search TMT Analysis
29
Fixation Cerebellar patients
 Three group were analyzed
 Control (healthy subject)
 SCA2 (Autosomal Dominant
Cerebellar Atrophiy)
Fixation dispersion
 pCA (Autosomal Recessive
Pure Cerebellar Atrophy)
 Significative differences
were found on
 Sequencing ability
 Fixation distances
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

30. Visual Search: gaze contingent analysis
30
TMT Fixation
 Two group were analyzed
 Control (healthy subject)
 FV (foveate vision inhibited by
gaze contingent)
 Significant differences were
found on
 Fixations dinstance
 Revisited ROI
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

31. Decision Making Analysis
31
20 subjects were asked to take a decision in a game
theory context
Overconfident subjects confirmed the theory that “gaze is
a renforcement of decision and vice-versa”
bayesiani overconfident
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

32. direction versus previous fixations
32
40 subjects
Distance from previous fixations increase on the last
800ms
Distance from unexplored region decreases on the last
800ms
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

33. Case Based System
33
www.evalab.unisi.it – visionlab@unisi.it
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI
AUTOMATIC CASE BASE SYSTEM 01/07/13

34. System Analysis
34
Ca ree
Ca ee
Tr
se
T
se
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

35. Analysis System
35
Time Saccade Analysis Main Sequence
Visual Search Analysis
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

36. Case Base
36
 The case base look for the N cases nearest to the current case according to the
previous indicators
 Euclidean distance
 Mahalanobis distance
 Case Base suggests diagnosis
 The case base actually works on 82 subjects
Case Base
New Patient
indicators
Diagnosis
Diagnosis
Diagnosis
Diagnosis
indicators Diagnosis
Diagnosis
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

37. 37
Case Base
Analysis
(Indicators)
Diagnosis
Vision Cognitive
Task
Rehabilitation
STIMULUS INTEGRATED SEMI
AUTOMATIC CASE BASE SYSTEM
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

38. Further Work
38
To extend the case base with:
 Degenerative brain diseases
 Movement disorders
 Attention deficit
Developing new indicators in order to find saccadic
intrusion and micro-saccade
Developing new indicators (markov chain) in order to
evaluate decision
Improve the search algorithm using Genetic
algorithm
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM
01/07/13

39. EVA Lab
ALESSANDRA RUFA (MD PhD),
GIACOMO VENERI (MSC),
PAMELA FEDERIGHI (MSC),
ELENA PRETEGIANI(MD),
FRANCESCA ROSINI (MD),
NICOLA R. POLIZZOTTO (MD),
EMILIANO SANTARNECCHI (PhD),
FRANCESCO FRAGNOLI (MD)
GIACOMO DE MURTAS (MSC),
ANTONIO FEDERICO (MD)
www.evalab.unisi.it – visionlab@unisi.it
Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena