This document presents research on using EEG signals to perform interactive object segmentation in images. The researchers propose a system where an image is partitioned into windows that are rapidly displayed to a user wearing an EEG headset. The user's EEG data is analyzed to identify their response to "target" windows containing objects versus other windows. This brain response data is then used to guide an image segmentation algorithm to segment the objects from the image. Initial results show the approach can achieve reasonable segmentation accuracy, though further improvements are still needed before it would be practical compared to using a mouse. Future work may explore changing parameters and combining the BCI with eye tracking.

1. Object Segmentation in Images
using EEG Signals
Eva Mohedano, Graham Healy, Kevin McGuinness, !
Xavier Giró-i-Nieto, Noel E. O’Connor and Alan F. Smeaton!
!!
November 6, 2014

2. Outline
•Interactive Object Segmentation!
•ACM MultiMedia High Risk High Reward 2014!
•Related Work!
•System Proposal!
•Results!
•Conclusions
E. Mohedano
2

3. Interactive Object Segmentation
•Object Segmentation
3
E. Mohedano

4. Interactive Object Segmentation
4
E. Mohedano

5. Interactive Object Segmentation
5
1) P. Arbelaez and L. Cohen. Constrained image segmentation from hierarchical boundaries. In
CVPR'08, 2008.!
2) McGuinness, K., & O’Connor, N. E. (2010). A comparative evaluation of interactive segmentation
algorithms. Pattern Recognition, 43(2), 434-444.
E. Mohedano

6. Outline
•Interactive Object Segmentation!
•ACM MultiMedia High Risk High Reward 2014!
•Related Work!
•System Proposal!
•Results!
•Conclusions
E. Mohedano
02 May 2014
6

7. Brain-Computer Interface (BCI)
7
E. Mohedano

8. Brain-Computer Interface (BCI)
8
E. Mohedano

9. Brain-Computer Interface (BCI)
9
Electroencephalography (EEG) signals
E. Mohedano

10. Brain-Computer Interface (BCI)
• Non invasive!
• Well established tool within
clinical practice
10
Strengths
E. Mohedano

11. Brain-Computer Interface (BCI)
• Mostly BCI applications
remain prototypes not
used outside laboratories!
• Users need to be trained!
• Poor BCI performances!
• Low signal-to-noise ratio!
• High dimensional data
11
Challenges HIGH RISK
E. Mohedano

12. Potentially High Reward
• Medical applications!
• Locked in Syndrome (LIS)!
• Prosthetics control, wheelchairs,
spellers
12
•Healthy Users!
• BCI with Virtual
Reality technologies!
• Augmenting gaze
control
E. Mohedano

13. Outline
•Interactive Object Segmentation!
•ACM MultiMedia High Risk High Reward 2014!
•Related Work!
•System Proposal!
•Results!
•Conclusions
02 May 2014
13
E. Mohedano

14. Related Work: RSVP
14
!
•A positive waveform occurring
approximately 300-550ms after
an infrequent task-relevant
stimulus
E. Mohedano

15. Related Work: RSVP
15
E. Mohedano

16. RSVP: Demo
16
E. Mohedano

17. Related Work
•Image Retrieval
17
E. Mohedano

18. Related Work
•Object Detection
18
E. Mohedano

19. Related Work
•BCI speller
19
Ref: D. Fernández-Cañellas, “Modeling temporal dependency of brain responses to rapidly
stimuli in ERP based BCIs” (2013)
E. Mohedano

20. Index
•Interactive Object Segmentation!
•ACM MultiMedia High Risk High Reward 2014!
•Related Work!
•System Proposal!
•Results!
•Conclusions
02 May 2014
20
E. Mohedano

21. System Proposal
•Local RSVP (5Hz visualisation windows)
21
E. Mohedano

22. System Proposal
•Different Reaction after seeing a target
22
Targets Distractors
E. Mohedano

23. System Proposal
23
E. Mohedano

24. System Proposal
Data Acquisition!
Set of 22 images with an associated ground truth mask
24
E. Mohedano

25. System Proposal
Data Acquisition!
Images were partitioned into 192 non overlapped windows!
! ! ! ! !
25
• 15% Target windows!
• RSVP windows at 5Hz!
• User asked to count the
target windows
visualised
E. Mohedano

26. System Proposal
26
EEG processing
E. Mohedano

27. 27
!
1) Down sample from 1000Hz to 250Hz!
2) Bandpass filter 0.1-70 Hz!
3) Cut EEG activity related to each visual event!
4) Down sample from 250Hz to 20Hz!
5) Concatene 31 channels (434D)
!
Support Vector Machine Model (SVM)
System Proposal
EEG processing
!
EEG feature vectors
E. Mohedano

28. 28
E. Mohedano

29. 29
System Proposal
Segmentation
• GrabCut: Interactive Foreground Extraction
OpenCV’s GrabCut Tutorial:!
http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_grabcut/py_grabcut.html !
E. Mohedano

30. 30
System Proposal
Evaluation Metric: Jaccard Index
Measure of similarity between the segmentation results and the ground truth mask
E. Mohedano

31. Outline
•Interactive Object Segmentation!
•ACM MultiMedia High Risk High Reward 2014!
•Related Work!
•System Proposal!
•Results!
•Conclusions
02 May 2014
31
E. Mohedano

32. Results
•Single User
32
Jaccard Index = 0.47
E. Mohedano

33. Results
•Averaged Users
33
Jaccard Index = 0.72
E. Mohedano

34. Index
•Interactive Object Segmentation!
•ACM MultiMedia High Risk High Reward 2014!
•Related Work!
•System Proposal!
•Results!
•Conclusions
02 May 2014
34
E. Mohedano

35. 35
Conclusions
The approach is feasible: it is possible to use BCI as an interactive
segmentation method based on simple EEG processing.
E. Mohedano

36. Conclusions
36
BCI Interaction for segmentation Mouse Interaction for segmentation
E. Mohedano
BCI is time consuming
Mouse interaction provides better results

37. Future work
37
• Improvements in EEG processing!
• Change resolution of windows!
• Use object candidates instead of a grid!
• Active search!
• Combine local EEG with eye tracker
E. Mohedano

38. Thank you!
38
Questions?
!
This publication resulted from research conducted with the financial
support of Science Foundation Ireland (SFI) under grant number SFI/12/
RC/2289 and partially funded by the Project TEC2013-43935-R BigGraph
of the Spanish Government.
E. Mohedano