EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM

870 views

Published on

In a real world visual search is a common task depending from
sensory, perceptual and cognitive processes. Different classes
of eye movements are necessary to hold an image on the retina
during head rotation or movement of the image, and to move
the eye suddenly to a new point of interest in space. From a
functional point of view, two major classes of eye movements
are described in humans: those stabilizing gaze (optokinetic
nystagmus, oculovestibular reflex) and those movinggaze
(saccades, pursuits and vergence). Under natural conditions,
however, a mix of all kinds of eye movements permit
continuous scanning of the visual scene. The sequence of
fixations and saccades during visual exploration isan
expression of a number of cognitive processes; the use of
standardized tasks with pre-defined spatial-temporal variables
allows us to assess specific cognitive domains, such as
perception, attention, memory, preference and motivation.
Manipulating the search task can vary the demands on brain. In
turn, brain modulates visual search by selecting and limiting
the information available at various levels of processing.
The EVA software is a complete system based on a set of
stimulus and patient’s case able to stress brain functionalities in
order to assess some cognitive functions.

Published in: Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
870
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM

  1. 1. EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM EYE TRACKING & VISION APPLICATIONS LAB (EVA LAB) – DEPARTMENT OF NEUROLOGICAL NEUROSURGICAL AND BEHAVIORAL SCIENCE – UNIVERSITY OF SIENAGiacomo Veneri, Pamela Federighi, Elena Pretegiani, Francesca Rosini, Nicola R Polizzotto, Francesco Fragnoli, Emiliano Santarnecchi, Giacomo DeMurtas, Antonio Federico & Alessandra Rufa www.evalab.unisi.it – visionlab@unisi.it Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena
  2. 2. EVA Light System 2 EVA EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASESYSTEMTo test the cognitive system by multi-stimulusDiagnosis by Case Base Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  3. 3. Summary 3Background (2m)  Analysis (5m)  Testing  Saccade: Velocity,Amplitude the cognitive functionalities by gaze Duration, Latency, Error  Fixation: dispersion Stimulus Architecture (5m)  Task: sequencing  Saccadic Behavior  Decision Making  Visual Search  Results (5m)  Gaze Contingent  TMS Interference  Case Base (1m)  Sound integration and joystick Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  4. 4. Background 4 “BY TESTING THE RESPONSE TO VISUAL STIMULI WE CAN TEST MORE THAN 60% OF BRAIN ACTIVITY” www.evalab.unisi.it – visionlab@unisi.itEye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  5. 5. Vision 5“More 60% of our Barin works on vision …”:  Image Processing  Spatial Attention  Working memory  Memory  Visual-Spatial exploration  Eye Movement Command  Decision Making The focus of attention is jointly determined by stimulus salience and top-down control signals. IOR is implemented by a mechanism that reduces the salience of stimuli at recently explored spatial locations. Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  6. 6. Eye Movement & Attention 6Eye movements  Sequences of fixation and saccade  The sequences of saccade- and-fixation enable humans to receive a continuous flow of information from the world. Peripheral attention Saccadic eye movementsAttention Overlapped region  Attention is the process of selecting a few relevant perceptive stimuli by filtering of negligible aspect of the environment Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  7. 7. Working Memory & Visual Search 7Visual working memory :  is a theoretical construct that refers to the structures and processes used for temporarily storing and manipulating visual information. informationVisual Search  memory maintains the information in order to allow past experience to guide future behaviors Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  8. 8. Decision making 8Decision making:  can be regarded as an outcome of mental processes (cognitive process) leading to the selection of a course of action among several alternatives. Basic connectivity framework. Dotted arrows represent external inputs to the model. Abbreviations: 5-HT, dorsal raphe serotonergic neurons; ACC, anterior cingulate cortex; AMYG, amygdala; DA, midbrain dopaminergic neurons; DLPFC, dorsolateral prefrontal cortex; OFC, orbitofrontal cortex; VS, ventral striatum. Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  9. 9. The Main Idea: 3 steps 91. To develop a multistimlus system able to stress some part and functionalities of brain in order to make a low cost pre-diagnosis of patient. 1. Interference: TMS, sound 2. Sensory stimulus: video, audio 3. Output: Eyetracking, joystick2. To develop indicators from specific task in order to test the brain response.3. To develop a database of case which can suggest dignosis avoiding any euristic or expert system logic. Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  10. 10. Preview: clinical case & research case 10Clinical case  Patients with movement disorder including cerebellar patients.  Neo Cortex injury.  Neglect patients (parietal abnormalities).  Frontal prefrontal lesions  Pre and post pharmacological assumption.  Deficit on attention & dyslexic patients.Research case  Temporary interference by TMS  Decision Making (NeuroEconomy – Dept. Economic) Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  11. 11. Task 11 SACCADES, MEMORY GUIDED Saccade, TMT, SOUND GUIDED visual exploration or saccades, GAZE CONTINGENT, FACE ANALYSIS, GAME THEORY. www.evalab.unisi.it – visionlab@unisi.itEye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  12. 12. Stimulus System 12Saccade Visual Search: time line Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  13. 13. Saccade / Anti saccade / Memory guided 13 Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  14. 14. Trail Making Task 14 Visual Search depends on sensory, perceptual and cognitive processes TMTB is used to measure the cognitive domains of processing speed, sequencing, visual search abilities and visual-motor skills. Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  15. 15. Gaze Contingent 15Reactive gaze-contingent display (GCD), which uses a simple hole forcing only foveal vision during exploration by blinding peripheral vision, in order to test and stress overt mechanisms of visual search in normal subjects and clinical contexts Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  16. 16. Decision Making 16Game TheoryThe subject was asked to choose between two sign according to previous chooses  Overconfident  Bayesian Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  17. 17. Analysis 17 INDICATORS www.evalab.unisi.it – visionlab@unisi.it Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  18. 18. Features 18Common NewSaccades  Saccades  Velocity Peak Going from/in ROI   Mean Velocity  TimeFixations  Delta time among ROI  N° fixations in ROI (step to step time)  Fixations Duration  ErrorsTime  ROI re-exploration  Task duration  Sequencing 1 2  Time in ROI Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  19. 19. Algorithms 19Signal processing  Spurious data purging  Saccade detection  Fixations identificationVisual Search indicator  Best Squence [BestSequenceAlgorithm]  Fixations distance  8PS ModelDecision Indicator  Likelihood function Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  20. 20. Best Sequence Algorithm 20Is a tree exploration algorithm that extract the Sequencing Index as an indicator of the ability in gazing correctly a logic sequence  Subject 1: O1A2B3C4BD5EE5O5A1 AB52  Patient 1: O13C4BD5E2ABE5OAE1 A12B Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  21. 21. SCA2 (subject 1) vs CTRL (subject 2) 21 Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  22. 22. Fixation Distance From ROI 22The distance from nearest ROI  It’s an indicator of fixation dispersion during visual exploration fs (xf,yf))**minROI ((δ((xf,yf), (xr,yr))) fs (xf,yf minROI δ((xf,yf), (xr,yr)))The distance from target  It’s an indicator of goal directed visual exploration fs (xf,yf))**((δ((xf,yf), (xr r(t+1),yr r(t+1)))) fs (xf,yf δ((xf,yf), (x (t+1),y (t+1)))) Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  23. 23. 8Pointed Star -Model 23 Direction Error (ε*) = (direction observed (ο*) – expected (η*) ) Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  24. 24. Ongoing Search abilities 24Direction error may provide a quantitative assessment of the visual search strategyprovides an estimation of the ongoing visual search capabilities. Δεε*= εεt*t-- εε*t-1 Δ* = * * t-1 Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaUsing Simplified Trial Making Test B to Evaluate Ongoing Learning Capabilities on Visual Search 01/07/13
  25. 25. Upcoming direction versus previous fixations 25The forthcoming direction minus the minimum/maximum distribution of fixation (ο*)) – min(direction(fixations)) (ο* – min(direction(fixations))It’s an expression to plan the saccade far from visited location (fixated) Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  26. 26. Decision: likelihood function 26Likelihood function was expressed as the probability that a subject spends more time looking the object which will be choose during the last 3 seconds preceding the explicit decision. Time on sign / /Time total elapsed Time on sign Time total elapsed Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  27. 27. Results 27 PATIENTS & SUBJECTS www.evalab.unisi.it – visionlab@unisi.itEye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  28. 28. Movement Disorder Saccade Analysis 28Saccade Cerebellar patients  Three group were analyzed600  Control (healthy subject)500  SCA2 (Autosomal Dominant400 Cerebellar Atrophiy)  pCA (Autosomal Recessive300 CTRL pCA Pure Cerebellar Atrophy) SCA2200  Significant differences were found on100  Peak Velocity, Duration. Amplitude  Gain 0 10° 18° Peak Velocity (°/s) 20° 10° 18° Mean Velocity (°/s) 20° 10° 18° Duration (s) 20°  Error Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  29. 29. Visual Search TMT Analysis 29Fixation Cerebellar patients  Three group were analyzed  Control (healthy subject)  SCA2 (Autosomal Dominant Cerebellar Atrophiy) Fixation dispersion  pCA (Autosomal Recessive Pure Cerebellar Atrophy)  Significative differences were found on  Sequencing ability  Fixation distances Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  30. 30. Visual Search: gaze contingent analysis 30TMT Fixation Two group were analyzed  Control (healthy subject)  FV (foveate vision inhibited by gaze contingent) Significant differences were found on  Fixations dinstance  Revisited ROI Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  31. 31. Decision Making Analysis 3120 subjects were asked to take a decision in a game theory contextOverconfident subjects confirmed the theory that “gaze is a renforcement of decision and vice-versa” bayesiani overconfident Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  32. 32. direction versus previous fixations 3240 subjectsDistance from previous fixations increase on the last 800msDistance from unexplored region decreases on the last 800ms Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  33. 33. Case Based System 33 www.evalab.unisi.it – visionlab@unisi.itEye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  34. 34. System Analysis 34 Ca ree Ca ee Tr se T se Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  35. 35. Analysis System 35Time Saccade Analysis Main SequenceVisual Search Analysis Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  36. 36. Case Base 36 The case base look for the N cases nearest to the current case according to the previous indicators  Euclidean distance  Mahalanobis distance Case Base suggests diagnosis The case base actually works on 82 subjects Case Base New Patient indicators Diagnosis Diagnosis Diagnosis Diagnosis indicators Diagnosis Diagnosis Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  37. 37. 37 Case Base Analysis (Indicators) Diagnosis Vision Cognitive Task Rehabilitation STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  38. 38. Further Work 38To extend the case base with:  Degenerative brain diseases  Movement disorders  Attention deficitDeveloping new indicators in order to find saccadic intrusion and micro-saccadeDeveloping new indicators (markov chain) in order to evaluate decisionImprove the search algorithm using Genetic algorithm Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of SienaEVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM 01/07/13
  39. 39. EVA Lab ALESSANDRA RUFA (MD PhD), GIACOMO VENERI (MSC), PAMELA FEDERIGHI (MSC), ELENA PRETEGIANI(MD), FRANCESCA ROSINI (MD), NICOLA R. POLIZZOTTO (MD), EMILIANO SANTARNECCHI (PhD), FRANCESCO FRAGNOLI (MD) GIACOMO DE MURTAS (MSC), ANTONIO FEDERICO (MD) www.evalab.unisi.it – visionlab@unisi.itEye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena

×