Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Investigating EEG for Saliency and
Segmentation Applications in Image
Processing
Eva Mohedano

1
CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Proce...
CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Proce...
1- PROBLEM STATEMENT
Design a system based on a Brain Computer Interface (BCI) wich measure
Electroencephalography (EEG) s...
1- PROBLEM STATEMENT
Design a system based on a Brain Computer Interface (BCI) wich measure
Electroencephalography (EEG) s...
1- PROBLEM STATEMENT
Design a system based on a Brain Computer Interface (BCI) wich measure
Electroencephalography (EEG) s...
1- PROBLEM STATEMENT
1 - Are the EEG signals to compute Saliency Maps?
Motivation:
- New way to compute maps of the atenti...
1- PROBLEM STATEMENT
2 - Are the EEG signals useful to images segmentation?
Motivation:
- Reduce the user interaction to t...
CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Proce...
2- RELATED WORK
2.1 – BCI in image processing applications

The oddball paradigm

10
2- RELATED WORK
2.1 – BCI in image processing applications

The oddball paradigm

P300

11
2- RELATED WORK
2.1 – BCI in image processing applications

The oddball paradigm

P300

• Speed rate around 10Hz
• Usually...
2- RELATED WORK

Event-Related Potential

•8 electrodes placed mainly in the
posterior points on the scalp.
• Which is con...
2- RELATED WORK

Event-Related Potential

•8 electrodes placed mainly in
How to present the scalp. to the
image
posterior ...
CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Proce...
3- LOCAL EXPLORATION OF THE IMAGE
First Design: Sliding Window

http://www.youtube.com/watch?v=bKTGKVx58Ps

16
3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 1
• Eyes movement affect to the EEG signals – Introduce Artifacts to the signa...
3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 2
• Progressive inspection may not generate a useful reaction in the EEG waves...
3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 3
•Syncronitzation Problem

19
3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 4
•Size of the object / window
Grabcut Dataset – Objects of different size

Su...
3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 5
•What am I seeing?

21
3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 5
•What am I seeing?

22
3- LOCAL EXPLORATION OF THE IMAGE

CHALLENGES

SOLUTIONS

1 - Eyes movement

Display a fixed window on the screen

2 - Pro...
CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Proce...
http://www.youtube.com/watch?v=KsgtvQkOE
lQ&feature=youtu.be

4- EXPERIMENTAL SET-UP
Second Design: Starting from the easi...
4- EXPERIMENTAL SET-UP
a) Device Calibration

?
Is the device well
connected?

Is the syncronitzation
method correct ?

Am...
4- EXPERIMENTAL SET-UP
a) Device Calibration

SIGNAL EXPECTED - Closed eyes – Alpha waves (8-12 Hz)

Closed-eye EEG alpha ...
4- EXPERIMENTAL SET-UP
a) Device Calibration

SIGNAL OBTAINED

5 seconds Closed Eyes

Amplitude (uV)

40
20
0
-20
-40

0

...
4- EXPERIMENTAL SET-UP
a) Device Calibration

Finding ERPS response after a white flash
SIGNAL EXPECTED: After the flash P...
4- EXPERIMENTAL SET-UP
a) Device Calibration

Finding ERPS response after a white flash
SIGNAL OBTAINED: 60 Flashes to get...
CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Proce...
5- SIGNAL PROCESSING OF EEG SIGNALS
Data adquisition

100 windows per Image

1 Target
99 Distractors
32
5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Single trial
One Image

1

2

3

4

5

6

7

8
33
5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Single trial - PROBLEM

- Signal very noisy
- Single Targets and Single...
5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Feature
96 Distractors
Mean Absolute Amplitude
looks different

96 Targ...
5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Averaged trials

1 averaged target
99 averaged
distractors

Energy from...
5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Averaged trials
SINGLE

AVERAGED

1 x 32 single repeats

1 averaged tar...
5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Averaged trials
1 averaged target

Problem:
Too few (target) samples
fo...
5- SIGNAL PROCESSING OF EEG SIGNALS
Bootstrapping
SINGLE

AVERAGED

1 x 32 single repeats

1 x 32 averaged target

99 x 32...
5- SIGNAL PROCESSING OF EEG SIGNALS
Classification

Problem:
Unbalanced dataset for
binary classification

AVERAGED
1 x 32...
5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
AVERAGED

AVERAGED

1 x 32 averaged targets

99 x 1
averaged distractor...
5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
AVERAGED

HISTOGRAM

1 x 32 averaged targets

32 x 1 avgd distractors

...
5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
AVERAGED
1 x 32 averaged targets

32 x 1 avgd distractors

1 x 32 avera...
5- SIGNAL PROCESSING OF EEG SIGNALS
Classification

100 x 32 single repeats

Average
by 32

100 avgd windows
44
5- SIGNAL PROCESSING OF EEG SIGNALS
Classification

100 x 32 single repeats

Average
by 32

100 avgd samples

Classifier

...
5- SIGNAL PROCESSING OF EEG SIGNALS
8 samples feature vectors
Cross validation approach 3 train + 1 test
CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Proce...
6- CONCLUSIONS
- Results from sythetic images provide and evidence that BCI devices could be
used to located an object int...
Upcoming SlideShare
Loading in …5
×

Eva Mohedano, "Investigating EEG for Saliency and Segmentation Applications in Image Processing"

822 views

Published on

Master thesis at Dublin City University, September 2013.
Co-advised by Xavier Giró-i-Nieto, Kevin McGuinness and Noel O'Connor.

More details:
https://imatge.upc.edu/web/publications/investigating-eeg-saliency-and-segmentation-applications-image-processing

Published in: Technology, Education
  • Best survey site online! $1,500 a month thanks to you guys! Without a doubt the best paid surveys site online!I have made money from other survey sites but made double or triple with GoldOpinions.com for the same time and effort. The variety and number of daily paid surveys I get from them is unmatched. A must for anyone looking for extra cash or a full time income. ★★★ https://bit.ly/2Ruzr8s
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Eva Mohedano, "Investigating EEG for Saliency and Segmentation Applications in Image Processing"

  1. 1. Investigating EEG for Saliency and Segmentation Applications in Image Processing Eva Mohedano 1
  2. 2. CONTENT 1 - Problem statement 2 – Related Work 3 – Local exploration of the image 4 – Experimental set-up 5 – Signal Processing of EEG Signals 6 – Conclusions 2
  3. 3. CONTENT 1 - Problem statement 2 – Related Work 3 – Local exploration of the image 4 – Experimental set-up 5 – Signal Processing of EEG Signals 6 – Conclusions 3
  4. 4. 1- PROBLEM STATEMENT Design a system based on a Brain Computer Interface (BCI) wich measure Electroencephalography (EEG) signals to answer the following questions: 1 - Are the EEG signals to compute Saliency Maps? 2 - Are the EEG signals useful to images segmentation? EEG Signals Visual stimulus BCI Data Processing 4
  5. 5. 1- PROBLEM STATEMENT Design a system based on a Brain Computer Interface (BCI) wich measure Electroencephalography (EEG) signals to answer the following questions: 1 - Are the EEG signals to compute Saliency Maps? 2 - Are the EEG signals useful to images segmentation? Visual stimulus BCI Data Processing 5
  6. 6. 1- PROBLEM STATEMENT Design a system based on a Brain Computer Interface (BCI) wich measure Electroencephalography (EEG) signals to answer the following questions: 1 - Are the EEG signals to compute Saliency Maps? 2 - Are the EEG signals useful to images segmentation? Visual stimulus BCI Data Processing 6
  7. 7. 1- PROBLEM STATEMENT 1 - Are the EEG signals to compute Saliency Maps? Motivation: - New way to compute maps of the atention of the image based directly in the reaction of the brain and not in the features of the images (Niebur and Koch (1996) algorithm). Visual stimulus BCI Data Processing 7
  8. 8. 1- PROBLEM STATEMENT 2 - Are the EEG signals useful to images segmentation? Motivation: - Reduce the user interaction to the minimun expression. - Measure the brain reaction at local scale of the image. Visual stimulus BCI Data Processing 8
  9. 9. CONTENT 1 - Problem statement 2 – Related Work 3 – Local exploration of the image 4 – Experimental set-up 5 – Signal Processing of EEG Signals 6 – Conclusions 9
  10. 10. 2- RELATED WORK 2.1 – BCI in image processing applications The oddball paradigm 10
  11. 11. 2- RELATED WORK 2.1 – BCI in image processing applications The oddball paradigm P300 11
  12. 12. 2- RELATED WORK 2.1 – BCI in image processing applications The oddball paradigm P300 • Speed rate around 10Hz • Usually experiements centered to find target images not target regions 12
  13. 13. 2- RELATED WORK Event-Related Potential •8 electrodes placed mainly in the posterior points on the scalp. • Which is consistent with the discriminating activity typically produced by a P300 ERP. [Optimising the Number of Channels in EEG-Augmented Image Search. Graham Healy] 13
  14. 14. 2- RELATED WORK Event-Related Potential •8 electrodes placed mainly in How to present the scalp. to the image posterior points on the generate and is consistent signal? • Which detect this with the discriminating activity produced by a P300 ERP. typically [Optimising the Number of Channels in EEG-Augmented Image Search. Graham Healy] 14
  15. 15. CONTENT 1 - Problem statement 2 – Related Work 3 – Local exploration of the image 4 – Experimental set-up 5 – Signal Processing of EEG Signals 6 – Conclusions 15
  16. 16. 3- LOCAL EXPLORATION OF THE IMAGE First Design: Sliding Window http://www.youtube.com/watch?v=bKTGKVx58Ps 16
  17. 17. 3- LOCAL EXPLORATION OF THE IMAGE CHALLENGE 1 • Eyes movement affect to the EEG signals – Introduce Artifacts to the signal Opened eyes / Closed eyes. Image from the slides Dr. Ranjith Polusani 17
  18. 18. 3- LOCAL EXPLORATION OF THE IMAGE CHALLENGE 2 • Progressive inspection may not generate a useful reaction in the EEG waves. Suggestions meeting Thomas Ward and Nima Bidgely Shamlo : - Follow Oddball Paradigm and perform RSVP od the windows P300 SNAP - Simulation and Neuroscience Application Platform 18
  19. 19. 3- LOCAL EXPLORATION OF THE IMAGE CHALLENGE 3 •Syncronitzation Problem 19
  20. 20. 3- LOCAL EXPLORATION OF THE IMAGE CHALLENGE 4 •Size of the object / window Grabcut Dataset – Objects of different size Suggestions meeting Thomas Ward and Nima Bidgely Shamlo : To use images with an homogeneus background with a salient object. The number of distractors (windows with background) must be higher than the number of targets (windows with object). 20
  21. 21. 3- LOCAL EXPLORATION OF THE IMAGE CHALLENGE 5 •What am I seeing? 21
  22. 22. 3- LOCAL EXPLORATION OF THE IMAGE CHALLENGE 5 •What am I seeing? 22
  23. 23. 3- LOCAL EXPLORATION OF THE IMAGE CHALLENGES SOLUTIONS 1 - Eyes movement Display a fixed window on the screen 2 - Progressive inspection SNAP to perform a random RSVP 3 - Syncronitzation Problem First test with flashes to find ERPS 4 - Size of the object / window Generate my own dataset 5 - What am I seeing? • Is it just noise? • Am I able to detect something? Real time visualitzation of the signal 23
  24. 24. CONTENT 1 - Problem statement 2 – Related Work 3 – Local exploration of the image 4 – Experimental set-up 5 – Signal Processing of EEG Signals 6 – Conclusions 24
  25. 25. http://www.youtube.com/watch?v=KsgtvQkOE lQ&feature=youtu.be 4- EXPERIMENTAL SET-UP Second Design: Starting from the easiest case a) Device Calibration I. Real time visualization of Alpha waves II. Detecting ERPS b) Synthetic Images I. RSVP synthetic images fitted in the window. c) RSVP real images 25
  26. 26. 4- EXPERIMENTAL SET-UP a) Device Calibration ? Is the device well connected? Is the syncronitzation method correct ? Am I able to detect something? 26
  27. 27. 4- EXPERIMENTAL SET-UP a) Device Calibration SIGNAL EXPECTED - Closed eyes – Alpha waves (8-12 Hz) Closed-eye EEG alpha waves (10-20 channels Pz-Top, Fz-Bottom) extracted from http://blog.grahamhealy.com/ 27
  28. 28. 4- EXPERIMENTAL SET-UP a) Device Calibration SIGNAL OBTAINED 5 seconds Closed Eyes Amplitude (uV) 40 20 0 -20 -40 0 0.5 1 1.5 0 0.5 1 1.5 2 2.5 3 3.5 Time (sec) 5 seconds Opened Eyes 4 4.5 5 4 4.5 5 Amplitude (uV) 40 20 0 -20 -40 2 2.5 3 Time (sec) 3.5 28
  29. 29. 4- EXPERIMENTAL SET-UP a) Device Calibration Finding ERPS response after a white flash SIGNAL EXPECTED: After the flash P100 and a negative peak between 150-200ms Presenting a serie of white flashes (2 seconds between the flashes) 29
  30. 30. 4- EXPERIMENTAL SET-UP a) Device Calibration Finding ERPS response after a white flash SIGNAL OBTAINED: 60 Flashes to get the response. Averaged ERP waveform per channel Channel P100 (ms) 1 130 320 2 90 210 3 90 210 4 10 220 5 110 220 6 90 210 7 10 220 8 100 22 Mean 10 N1 (ms) 80 23 Amplitude (uV) 5 0 -5 -10 -15 0 100 200 300 400 500 600 Time (ms) 700 800 900 1000 30
  31. 31. CONTENT 1 - Problem statement 2 – Related Work 3 – Local exploration of the image 4 – Experimental set-up 5 – Signal Processing of EEG Signals 6 – Conclusions 31
  32. 32. 5- SIGNAL PROCESSING OF EEG SIGNALS Data adquisition 100 windows per Image 1 Target 99 Distractors 32
  33. 33. 5- SIGNAL PROCESSING OF EEG SIGNALS Preprocessing: Single trial One Image 1 2 3 4 5 6 7 8 33
  34. 34. 5- SIGNAL PROCESSING OF EEG SIGNALS Preprocessing: Single trial - PROBLEM - Signal very noisy - Single Targets and Single Distractors very similar 34
  35. 35. 5- SIGNAL PROCESSING OF EEG SIGNALS Preprocessing: Feature 96 Distractors Mean Absolute Amplitude looks different 96 Targets Energy from 0 to 600ms Feature for the window presented 35
  36. 36. 5- SIGNAL PROCESSING OF EEG SIGNALS Preprocessing: Averaged trials 1 averaged target 99 averaged distractors Energy from 0-600 ms 36
  37. 37. 5- SIGNAL PROCESSING OF EEG SIGNALS Preprocessing: Averaged trials SINGLE AVERAGED 1 x 32 single repeats 1 averaged target 99 x 32 single repeats 99 averaged distractors 1 x 32 window repeats 1 averaged target 99 x 32 single repeats Average by 32 99 averaged distractors 1 x 32 window repeats 1 averaged target 99 x 32 single repeats 99 averaged distractors 37
  38. 38. 5- SIGNAL PROCESSING OF EEG SIGNALS Preprocessing: Averaged trials 1 averaged target Problem: Too few (target) samples for training 99 averaged distractors 1 averaged target 99 averaged distractors 1 averaged target 99 averaged distractors 38
  39. 39. 5- SIGNAL PROCESSING OF EEG SIGNALS Bootstrapping SINGLE AVERAGED 1 x 32 single repeats 1 x 32 averaged target 99 x 32 single repeats 99 x 1 averaged distractors 1 x 32 window repeats 99 x 32 single repeats 1 x 32 window repeats 99 x 32 single repeats 1 x 32 averaged target Bootstrapping 99 x 1 averaged distractors 1 x 32 averaged target 99 x 1 averaged distractors39
  40. 40. 5- SIGNAL PROCESSING OF EEG SIGNALS Classification Problem: Unbalanced dataset for binary classification AVERAGED 1 x 32 averaged target 99 x 1 averaged distractors 1 x 32 averaged target 99 x 1 averaged distractors 1 x 32 averaged target 99 x 1 averaged distractors40
  41. 41. 5- SIGNAL PROCESSING OF EEG SIGNALS Classification AVERAGED AVERAGED 1 x 32 averaged targets 99 x 1 averaged distractors 1 x 32 averaged targets Subsample 1 x 32 averaged targets 99 x 1 averaged distractors 1 x 32 averaged targets Subsample 1 x 32 averaged targets 99 x 1 averaged distractors 32 x 1 avgd distractors 32 x 1 avgd distractors 1 x 32 averaged targets Subsample 32 x 1 avgd distractors 41
  42. 42. 5- SIGNAL PROCESSING OF EEG SIGNALS Classification AVERAGED HISTOGRAM 1 x 32 averaged targets 32 x 1 avgd distractors 1 x 32 averaged targets 32 x 1 avgd distractors 1 x 32 averaged targets 32 x 1 avgd distractors 42
  43. 43. 5- SIGNAL PROCESSING OF EEG SIGNALS Classification AVERAGED 1 x 32 averaged targets 32 x 1 avgd distractors 1 x 32 averaged targets 32 x 1 avgd distractors SVMTRAIN (linear kernel) Classifier 1 x 32 averaged targets 32 x 1 avgd distractors 43
  44. 44. 5- SIGNAL PROCESSING OF EEG SIGNALS Classification 100 x 32 single repeats Average by 32 100 avgd windows 44
  45. 45. 5- SIGNAL PROCESSING OF EEG SIGNALS Classification 100 x 32 single repeats Average by 32 100 avgd samples Classifier SVM PREDICT 45
  46. 46. 5- SIGNAL PROCESSING OF EEG SIGNALS 8 samples feature vectors Cross validation approach 3 train + 1 test
  47. 47. CONTENT 1 - Problem statement 2 – Related Work 3 – Local exploration of the image 4 – Experimental set-up 5 – Signal Processing of EEG Signals 6 – Conclusions 47
  48. 48. 6- CONCLUSIONS - Results from sythetic images provide and evidence that BCI devices could be used to located an object into an image. - Simplicity of the system: Energy value from 8 channels to train SVM with lineal kernel. Future work -Study the impact of the number of repetitions. - Extract better features. -Analize data from real images. -Tool to evaluate and compare the EEG mask (ROC, Jaccard index) 48

×