1. Date: 12 July 2016
BCI and IoT for smart living
environment
Realised by Abdel Rahman IAALY
FINAL YEAR PROJECT - FACULTY OF ENGINEERING I - ELECTRICAL & ELECTRONICS DEPARTMENT
Presented to Dr. Rola NAJA, Dr. Mohamad KHALIL, Dr. Salah EL-FALOU
Supervised by Dr. Wassim EL-FALOU, Dr. Nisrine JRAD
2. OUTLINE
✤ Introduction
✤ 1. Brain Computer Interface (BCI)
✤ Definition
✤ Electroencephalogram (EEG) and system used
✤ NeuroPhysiological Brain Signals
✤ Steady State Visual-Evoked Potentials
✤ 2. Internet of Things (IoT)
✤ Definition
✤ Raspberry PI
✤ Web-Based Home Automation
✤ 3. State of the art
✤ Processing pipeline
2
5. Introduction: Project description
headlines
✤ Control home devices using the human brain
✤ No muscular activity needed
✤ Specifically, we want to detect where the user is
looking… and then perform a command
5
6. Introduction: Project goal
✤ 10% the total world's population, or roughly 650
million people, live with a disability.
✤ This work could help totally
disabled people to live
normally
✤ Or just make life easier
6
90%
10%
7. Introduction: Project modules
1) Brain Computer Interface (BCI) module
2) Internet of Things (IoT) module
Developed independently and joined together finally to
form a promising prototype
7
9. BCI: Definition
✤ Brain-computer interface (BCI) is a collaboration
between a brain and a device that enables signals from
the brain to direct some external activity
9
10. Electroencephalogram (EEG)
✤ It is a test that detects electrical activity in your brain
using small electrodes attached to your scalp.
✤ Your brain cells communicate via electrical impulses
and are active all the time, even when you're asleep.
✤ This activity shows up as wavy lines on an EEG
recording.
10
12. TMSi’s water based EEG electrodes
EEG headset Water-based electrodes ground wrist band The Porti amplifier
12
13. EEG acquisition configuration
Preparations before the acquisition
Electrode placements are usually
according to the 10-20 international
system shown in figure below
Electrodes placements 13
14. NeuroPhysiological Brain Signals
✤ The aim of BCI is to identify the brain activity
✤ For this, various signals can be identified easily
14
15. NeuroPhysiological Brain Signals
✤ These signals are divided in two parts:
1.Spontaneous signals: generated by the user, without external
stimulation, due to an internal cognitive process.
2.Evoked Potentials (EP): generated unconsciously by the
participant, when he perceives a specific external stimulus.
15
16. NeuroPhysiological Brain Signals
✤ These signals are divided in two parts:
1.Spontaneous signals: generated by the user, without external
stimulation, due to an internal cognitive process.
2.Evoked Potentials (EP): generated unconsciously by the
participant, when he perceives a specific external stimulus.
16
17. Evoked Potentials Brain Signals
✤ The most important signals
that falls under this category
are:
1.P300: Occurs if the subject is actively
engaged in the task of detecting the
targets. Its amplitude varies with the
improbability of the targets.
2.SSVEP: Natural responses to visual
stimulation at specific frequencies.
17
SSVEP
P300
18. Evoked Potentials Brain Signals
✤ The most important signals
that falls under this category
are:
1.P300: Occurs if the subject is actively
engaged in the task of detecting the
targets. Its amplitude varies with the
improbability of the targets.
2.SSVEP: Natural responses to visual
stimulation at specific frequencies.
18
SSVEP
P300
19. Steady State Evoked Potentials
(SSVEP) based-BCIs
✤ High information transfer rates (bitrate)
✤ Minimal training required
✤ Require few EEG channels
19
20. SSVEP wave
✤ Appears in response to a stimulus modulated at a
certain frequency
✤ The frequency of the SSVEP matches that of the
stimulus or its harmonics
20
21. SSVEP acquisition system
✤ The main part of the brain responsible for visual
processing is the occipital lobe.
21
Occipital regionBrain parts
22. SSVEP acquisition system
✤ We used 8 electrodes placed on the occipital region
22
The ampli connected to the 8 electrodes
24. IoT: Definition
✤ The internet of things (IoT) is the network of physical
devices, embedded with electronics, software, sensors,
and network connectivity that enable these objects to
collect and exchange data.
24
25. Raspberry PI
✤ Mini computer
✤ Low cost
✤ Easily accessible
✤ Simple to use
25
Raspberry PI 2 model B
26. Raspberry PI hardware
✤ 32 bit 900 MHZ quad-core ARM A7
✤ 256 KB shared L2 cache
✤ 4 USB ports
✤ 1 Ethernet port
✤ 1 HDMI port
✤ 3.5 mm phono jack for audio
✤ 40-pin pinout
26
Raspberry PI 2 model B pins
27. Web-based home automation
✤ Home Automation (HA):
A home that have a life on its own
✤ Web-based HA: devices communicate with each other
via the internet
27
28. Web-based home automation
✤ One main task is needed: Deploying a web server to a “mini
server”, which is the Raspberry PI.
28
Communication between a client (web browser) and raspberry PI (web server)
29. Webserver on Raspberry PI
29
✤ Developed with VB.NET as a Universal Windows
Platform (UWP) application, using Visual Studio 2015.
✤ OS: Windows 10 IoT core, a version of windows 10
optimised for smaller devices
✤ We used the library named MetroAir to install the web
server
30. Web server commands
✤ From client side, request of type GET:
Sent to the URL of the web server running on the Raspberry PI
With data appended to the URL as string:
http://IPorDomain/web?param1=val1¶m2=val2&…
✤ From the server side (Raspberry PI), receiving GET
request:
Query for param1 to get its value val1, and eventually use this value
to access pins and controlling devices
30
32. Processing pipeline
✤ Reduce artefacts from the signal
✤ Determine various descriptors of the signal
✤ Reduce the dimensionality of the feature space and removing redundancy —>
increase the classification accuracy
✤ Assigning the EEG signals into one of several predetermined categories/classes
32
block diagram showing the several processing steps
33. Preprocessing: Filtering
✤ Goal: reshape the spectrum of the signal in our
advantage
✤ Band pass filter with range 4-48Hz, defined by the
stimuli frequencies and their harmonics
✤ There’s two main group of filters: Finite Impulse
Response (FIR) and Infinite Impulse Response (IIR)
33
34. Preprocessing: Filtering (cont’d)
34
FIR IIR
Stable,
characteristics of
linear phase
Not stable, non-
linear
characteristics
Require a lot of
memory
Economist in
memory
[1] [2] [3]
35. Preprocessing:Artefacts removal
✤ After filtering, some noises still persists
✤ Artefacts are unwanted physiological signals that
come from the body.
Such as eye blinks, muscular movements,
Electrooculography (EOG)…
35
36. Techniques for removing Artefacts
✤ AMUSE: most widely used for artefact removal, relies on Blind
Source Separation (BSS) [3], used successfully with SSVEP in [4].
✤ Independent Component Analysis (ICA): uses technique
from linear algebra [5].
36
37. Feature extraction
✤ signal feature = alternative representation of the signal
✤ Two categories of extracting features:
1) Non-transformed features: moments, power,
energy
2)Transformed features: frequency, amplitude spectra
37
38. Feature extraction methods for SSVEP
✤ Periodogram [1][2]: simply the Discrete Fourier Transform (DFT) of the signal
Computational advantage = Uses the Fast Fourier Transform (FFT)
Spectral leakage, frequency resolution
✤ Welch Spectrum:
Based on the periodogram, and modify it to solve its problems
✤ Short Time Fourier Transform [6]
✤ Discrete Wavelet Transform [6]
38
39. Feature selection
Dimensionality reduction, Removing redundancy
✤ Principal Component Analysis (PCA): Converts a set of
observations of possibly correlated variables into a set of values of linearly
uncorrelated variables called principal components [7].
✤ Singular Value Decomposition (SVD): is a factorization of a
real or complex matrix [8].
✤ Independent Component Analysis (ICA): separates a
multivariate signal into additive subcomponents [9].
39
40. Classification
✤ Decision tree classifiers: uses a decision tree as a predictive model
which maps observations about an item to conclusions about the item's target
value [10].
✤ Rule-based classifiers: represented as a set of IF-THEN rules.
✤ Support Vector Machines (SVM): supervised learning models with
associated learning algorithms that analyze data used for classification and
regression analysis [11].
✤ Naive Bayes: simple probabilistic classification based on applying Bayes'
theorem.
40
42. General goal
✤ Acquire signals from a subject
✤ Understand these signals (processing)
✤ Generate a classifier, with high accuracy, representing the subject
✤ Use this classifier to classify live signals (Online)
✤ Predict his thoughts
42
43. Acquisition scenario (1)
✤ Goal: Collect data for three frequencies to be able to
predict them later
✤ Performed using OpenVibe software
✤ 5 sessions/subject
✤ Frequencies: 7.5 Hz, 10 Hz, 12 Hz
✤ Represented by red squares on a black background
43
44. Acquisition scenario (2)
✤ 4 squares are shown, one black
(still), and three red (flickering)
✤ Each frequency is stimulated 8
times /session
✤ Sessions were separated by
breaks of 30 seconds
44
acquisition scenario
45. Acquisition scenario (3)
✤ The subject was asked to focus on each of the targets
in a predefined order
✤ Each stimulation lasts for 5 seconds, and repeated 8
times
✤ 5 seconds break between each stimulation
✤ 32 stimulations/sessions
45
46. Experimental EEG recordings
✤ 1 stimulation = 1 trial{subject, frequency, signal
chunk} —> object in Matlab ready for later processing
✤ 8 subjects participated
✤ 5 sessions/subject * 32 trials/session = 160 trials/
subject
✤ 160 trials/subject * 8 subjects = 1280 total trials
46
47. Adopted processing methods
✤ Signal filtering: IIR elliptic filter (best output filtered signal)
✤ Artefact removal: AMUSE (because of its popularity with SSVEP)
✤ Feature extraction: PWelch (good results and simple to use)
✤ Feature selection: SVD (gave the most successful results)
✤ Classification: SVM (most popular)
47
49. Experimental setup
To evaluate the work offline, we
proposed three methods to train a
classification model :
✤ Method 1: Leave one sample out
✤ Method 2: Leave one sample out,
given a subject
✤ Method 3: Leave one subject out
49
method 1
method 2
method 3
50. Results: Electrode selection
✤ Based on features quality,
we selected the most
accurate electrode for
each subject
50
Subject Channel
S1 POz
S2 POz
S3 O2
S4 POz
S5 Oz
S6 O2
S7 Oz
S8 POz
51. Results: Feature extraction
51PSD when S2 was looking at the 7.5Hz box
PSD when S2 was looking at the 10Hz box PSD when S2 was looking at the 12Hz box
54. Experimental setup
54
Online test architecture
Goal: Develop a software
solution to be able to control
home devices
Materials:
•Classifier object
•Stimulator (.Net app)
•Signal Acquisition and
Processing server (local
desktop)
•Web-based HA
55. .Net app UI
55
A trick to use the same frequency for multiple tasks
56. Openvibe-Matlab communication
✤ Real-time signal acquisition using OpenVibe
✤ Forwarded to Matlab using Lab Streaming Layer
(LSL)
✤ Received in Matlab as chunks of signals
✤ Each chunk is processed and classified using our
algorithm and a previously created classifier object
56
57. Matlab-Web communication
✤ Every classification —> prediction —> GET request —> command
✤ Sending request in matlab is easy:
URL = 'X.X.X.X/command';
str = urlread(URL,'Get',{'bulb','on'});
57
58. Matlab-.Net communication
✤ MATLAB app:
predicts —> writes the frequency to text file
✤ .Net app:
reads the frequency from the file —> update UI
58
59. Results: online accuracy
✤ The subject was asked to do a set of actions in order
✤ The prediction accuracy was the same as the offline
accuracy (using method 2) of the classifier object we
used.
✤ We performed the online test using a simple LED
✤ Resulting online test accuracy = 90%.
59
60. 6. Discussion and conclusion
✤ In this work, we’ve projected the SSVEP power and
capabilities into the IoT world.
✤ Possible future applications:
Self-driving car/robot using human eyes.
SSVEP based mind speller
60
61. References
✤ [1] A. V. Oppenheim, R. Schafer, and J. Buck, Discrete-time Signal Processing (2Nd Ed.). Prentice-Hall, Inc., 1999.
✤ [2] J. Proakis and D. Manolakis, Digital Signal Processing (3rd Ed.): Principles, Algorithms, and Applications. Prentice-Hall, Inc., 1996.
✤ [3] S. White, Digital Signal Processing: A Filtering Approach. Delmar Cengage Learning, 2000.
✤ [4] P. Martinez, H. Bakardjian, and A. Cichocki, “Fully online multicommand brain-computer interface with visual neurofeedback
using ssvep paradigm,” Computational intelligence and neuroscience, vol. 2007, pp. 13–13, 2007.
✤ [5] A. Hyvarinen and E. Oja, “Independent component analysis: Algorithms and applications,” Neural Networks, vol. 13, pp. 411–
430, 2000.
✤ [6] S. Mallat, A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way. Academic Press, 3rd ed., 2008.
✤ [7] Wikipedia contributors. "Principal component analysis." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 10
Jul. 2016. Web. 11 Jul. 2016.
✤ [8] Wikipedia contributors. "Singular value decomposition." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 4
Jul. 2016. Web. 11 Jul. 2016.
✤ [9] Wikipedia contributors. "Independent component analysis." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia,
6 Jul. 2016. Web. 11 Jul. 2016.
✤ [10] Wikipedia contributors. "Decision tree learning." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 14 May.
2016. Web. 11 Jul. 2016.
✤ [11] Wikipedia contributors. "Support vector machine." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 7 Jul.
2016. Web. 11 Jul. 2016.
61
