This document presents a motor imagery-based brain-computer interface speller for mobile devices. It introduces a motor imagery structure model consisting of pre-processing, feature extraction, dimensionality reduction, and classification blocks. It develops an autoencoder-based dimensionality reduction method and compares it to PCA. It also develops a Hex-O-Spell mobile application using motor imagery to spell words. Results show the autoencoder approach achieves better performance than PCA. Testing on three subjects demonstrates the utility of the Hex-O-Spell mobile application. Future work involves enhancing the methods and application.

1. A Brain-Computer Interface Speller for
Smart Devices
Mahmoud A. Helal
Supervised by : Dr. Mohamed Taher and Dr. Seif Eldawlatly
Computer and Systems Engineering Department, Faculty of Engineering
Ain Shams University
Cairo, Egypt

2. Agenda
2
INTRODUCTION
▰Brain-Computer Interfaces (BCI)
▰Electroencephalography (EEG)
▰BCI Applications
▰BCI Mobile Applications
▰10-20 System
▰Event Related Desynchronization (ERD)
CONTRIBUTIONS
▰Motor Imagery Structure Model
▰Hex-O-Spell Mobile Application
CONCLUSION AND FUTURE WORK

3. INTRODUCTION
3
1

4. Brain-Computer Interface (BCI)
A communication system that facilitates controlling external devices by recording,
processing and analyzing signals detected from the brain neural activity
4
INVASIVE NON-INVASIVE

5. Electroencephalography (EEG)
Record electrical activity along the scalp produced by the firing of neurons over a
short period of time
5
EEG Signal
Command
Feedback

6. BCI Applications
6
Wheelchair Gaming Spellers

7. BCI Mobile Applications
7
NeuroPhone System
Campbell, A., T. Choudhury, et al. (2010)

8. BCI Mobile Applications
8
BCI Messenger
Li, Y., J. Zhang, et al. (2009)

9. BCI Mobile Applications
9RunApp and ImgView
Elsawy, A. S. and S. Eldawlatly (2015)

10. 10 – 20 System
▰ Introduced by the International Federation of
Societies for Electroencephalography and Clinical
Neurophysiology
▰ Method used to describe electrodes position over
the scalp
10

11. Event Related Desynchronization (ERD)
▰ Mu [8–13 Hz] and Beta [13–30 Hz] bands
▰ Always correlated with ERS
11
(Neuper et al ,2006)

12. Event Related Desynchronization (ERD)
▰ Mu [8–13 Hz] and Beta [13–30 Hz] bands
▰ Always correlated with ERS
12
(Neuper et al ,2006)

13. Raw EEG data for right and left hand imaginary
13
(μV)(μV)
(μV)(μV)

14. CONTRIBUTIONS
14
2

15. Motor Imagery Structure Model
▰ Consists of 4 main blocks:
▻ Pre-Processing removing artifacts
▻ Feature Extraction extract information from raw signals
▻ Dimensionality Reduction reduce data dimensionality to speedup classification
▻ Classification classify the reduced dimension data
15
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
ClassificationEEG Data Command

16. Pre-Processing
▰ Common Average Reference (CAR) to eliminate external noise
▰ Z-score to adjust values measured from different channels on different scales to a
notionally common scale
▰ Band-pass filter [4-41] Hz to remove eye artifacts and get domain that contains
valuable ERD/RDS information
▰ Down-sample to speedup subsequent processing.
16
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
Classification
EEG Data Command

17. 17
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
Classification
EEG Data Command
NormalizedSignal(μV)

18. Feature Extraction - Band power
18
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
Classification
EEG Data Command
▰Apply 5th order Butterworth band-pass filter using the frequency bands
▰Square every sample in the output signal to obtain power
𝑝 𝑡 = 𝑥2
(𝑡)
▰Average power
𝑝 𝑛 =
1
𝑤
𝑘=0
𝑤
𝑝(𝑛 − 𝑘)
▰Log
𝑙𝑜𝑔10 𝑝(𝑛)

19. ▰Principal Component Analysis (PCA) is a classical
technique in dimensionality reduction
▰The goal is to project data on the minimum number of
principal components that represent the maximum amount
of variance in the data
𝒛 = 𝒙 . 𝒘
Dimensionality Reduction – Principal Component
Analysis (PCA)
19
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
Classification
EEG Data Command
Feature 1
Feature2

20. ▰An Autoencoder (AE) is a neural network with three
or more layers, an input layer and an output layer
▰The main target of AE is to reconstruct its input on
its output nodes
▰Number of nodes of the hidden layer is less than the
number for the input layer
Dimensionality Reduction - AutoEncoder
20
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
Classification
EEG Data Command
x zh
Input layer
Hidden layer
Output layer
Encoder Decoder
Feature
Data
Compressed
Features

21. Classification – Linear Discriminant Analysis (LDA)
21
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
Classification
EEG Data Command
▰ LDA is most commonly used as a data classification
technique in BCI
▰ One-versus-all method is used to classify 4 classes
𝐽 𝑤 =
𝑤 𝑡
. 𝑆 𝑏 . 𝑤
𝑤 𝑡 . 𝑆 𝑤 . 𝑤
▰ 𝑆 𝑏 is the between class scatter matrix
𝑆 𝑏 =
𝑘=1
𝑚
𝑛 𝑘(µ 𝑘 − µ)(µ 𝑘 − µ) 𝑇
▰ 𝑆 𝑤 is the within-class scatter matrix
Feature 1
Feature2

22. ▰Based on Bayes’ Theorem
▰Naive Bayes classifier assumes independence between features
𝑃 𝑐| 𝑓 =
𝑃 𝑓| 𝑐 . 𝑃(𝑐)
𝑃(𝑓)
𝑃 𝑐| 𝑓 is the posterior probability of class c given feature f, . 𝑃(𝑐) is the prior probability of class c, 𝑃 𝑓| 𝑐 is the likelihood which is the
probability of feature f given class c
Classification – Naive Bayes Classifier (NBC)
22
Pre-
Processing
Feature
Extraction
Dimensionality
Reduction
Classification
EEG Data Command

23. RESULTS
DataSet , Evaluation and Performance of Methods
23

24. ▰BCI Competition IV (2008) - Dataset 2a
▰9 subjects. 2 sessions were recorded on 2 days
▰Single session consisted of 288 trials over 22 EEG channels
▰The subject to perform one of the following motor imagery tasks: left hand, right hand, both
feet, or tongue
BCI Competition Dataset
24

25. Kappa Evaluation
25
Predicted
Right
Hand
Left
Hand
Feet Tongue
Actual
Right Hand
𝑫 𝟏𝟏 𝑫 𝟏𝟐 𝑫 𝟏𝟑 𝑫 𝟏𝟒
Left Hand
𝑫 𝟐𝟏 𝑫 𝟐𝟐 𝑫 𝟐𝟑 𝑫 𝟐𝟒
Feet
𝑫 𝟑𝟏 𝑫 𝟑𝟐 𝑫 𝟑𝟑 𝑫 𝟑𝟒
Tongue
𝑫 𝟒𝟏 𝑫 𝟒𝟐 𝑫 𝟒𝟑 𝑫 𝟒𝟒
▰ kappa is calculated by equation
𝑘 =
𝑝0 − 𝑝 𝑒
1 − 𝑝 𝑒
▰ Observed proportionate agreement between raters
𝑝0 =
1
𝑛
𝑖=1
𝑐
𝐷𝑖𝑖
▰ Expected agreement on the same data
𝑝 𝑒 =
1
𝑛2
𝑖=1
𝑐
𝐷𝑖+ . 𝐷+𝑖

26. ▰BCI Competition IV (2008) - Dataset 2a
▰9 subjects. 2 sessions were recorded on 2 days
▰Single session consisted of 288 trials over 22 EEG channels
▰The subject to perform one of the following motor imagery tasks: left hand, right hand, both
feet, or tongue
BCI Competition Dataset
26

27. ▰ Technique used to estimate the optimal number of reduced dimensions in dimensionality
reduction methods or optimal classifiers used.
▰ It is commonly used to prevent overfitting the classifier
Cross-Validation
27

28. ▰Examine different frequency bands ranges in order to determine the most effective frequency
bands in classification process
Results: Feature Extraction
28
#
Bands
(Hz)
#
features
kappa
1 8,13 22 0.449074
2 8-15 22 0.472737
3 10-13 22 0.431070
4 5-10 22 0.316358
5 9-12 22 0.415123
#
Bands
(Hz)
#
features
kappa
1 8-15, 13-20 44 0.518519
2 8-15, 18-25 44 0.517490
3 8-15, 12-21 44 0.534979
4 8-15, 20-28 44 0.513889
5 8-15, 20-25 44 0.503086
# Bands (Hz)
#
features
Kappa
1 8-15, 12-21, 20-25 66 0.509259
2 8-15, 12-21, 22-29 66 0.529835
3 8-15, 12-21, 24-27 66 0.524177
4 8-15, 12-21, 20-30 66 0.538580

29. ▰Cross-validation is used by dividing the training data to 80% training and 20% validation
selected randomly, then repeat this process 10-times
Results: Classification
29
Classifier Kappa
1 NBC 0.45
2 LDA 0.51

30. ▰Compare the performance of AE and PCA for the same number of reduced dimensions. range
of 45 to 65 with a step of 5 averaged across subjects
Results: Performance over Number of Dimensions
30
Dimensionality
Reduction
Kappa
1 Without 0.51
2 PCA 0.52
3 AutoEncoder 0.56

31. ▰Subject-dependent optimal parameters used for each method (number of hidden neurons in
AE and number of principal components in PCA)
Results: AutoEncoder versus PCA
31
Dimensionality
Reduction
Kappa
1 Without 0.51
2 PCA 0.52
3 AutoEncoder 0.55

32. ▰Examine the performance when sigmoid activation function is used compared to linear
activation function averaged across subjects.
Results: AE Activation Function
32
AE Activation Kappa
1 Linear 0.53
2 Sigmoid 0.55

33. Results: AutoEncoder versus Competition Results
33
# contributor
Subjects
Mean
1 2 3 4 5 6 7 8 9
1 Kai Keng Ang 0.68 0.42 0.75 0.48 0.40 0.27 0.77 0.75 0.61 0.57
2 Mahmoud Helal 0.76 0.33 0.74 0.53 0.18 0.32 0.78 0.71 0.64 0.55
3 Liu Guangquan 0.69 0.34 0.71 0.44 0.16 0.21 0.66 0.73 0.69 0.52
4 Wei Song 0.38 0.18 0.48 0.33 0.07 0.14 0.29 0.49 0.44 0.31
5 Damien Coyle 0.46 0.25 0.65 0.31 0.12 0.07 0.00 0.46 0.42 0.30
6 Jin Wu 0.41 0.17 0.39 0.25 0.06 0.16 0.34 0.45 0.37 0.29

34. ▰Results in channels 9, 10 and 11 have the highest value of the weights
▰Channels correspond to C1, Cz and C2 which cover the left, central and right areas of the
motor cortex
Analysis of AE Weights
34

35. 35
Command
AE 157
BP 1
8-15
BP 2
12-21
BP 3
20-30
LDA 157 Evaluation 157
..
1
2
.
.
157
AE 2 LDA 2 Evaluation 2
AE 1 LDA 1 Evaluation 1
Pre-Processing
EEG Data
Dimensionality ReductionFeature Extraction Classification Evaluation
..
..
Best
Sample
Feature
EEG Signal Flow in Motor Imagery System
[672528x22]
[168132 x 22]
[22 x 228] [228x 45] [228]

36. HEX-O-SPELL
Algorithm and Mobile Application
36

37. ▰Five letters in each six Hexs
▰Right Feet : select
▰Right Hand : Return
Hex-O-Spell : Algorithm
37

38. Hex-O-Spell : Mobile Application
38
SPELLING PHASE
User try to write word
TRAINING PHASE
Application asks user
to imagne right hand /
right feet movement

39. Hex-O-Spell : Mobile Application
39

40. ▰Test on three subject train 50 trial (25 right hand, 25 right feet) 80% training data 20% testing
data
Hex-O-Spell : Results
40
Subject Gender
Best
PCA
Training
Accuracy
Cross-
Validation
Accuracy
1 Subject A Male 22 96.66% 90%
2 Subject B Female 28 100% 70%
3 Subject C Male 26 96.66% 80%

41. Hex-O-Spell : Experiment
41

42. CONCLUSION AND
FUTURE WORK
42
3

43. MOTOR IMAGERY SYSTEM
▻ Different methods implemented and tested
▻ Autoencoder dimension reduction method is introduced, and compared with
traditional and classical method PCA
▻ Results suggest using autoencoders with sigmoid activation functions achieve
better performance compared to using linear PCA
Conclusion
HEX-O-SPELL
▻ BCI Hex-O-Spell application is developed.
▻ Results for three different subjects demonstrate the utility of the
application 43

44. MOTOR IMAGERY SYSTEM
▻ Enhance Methods automated detection for best frequency bands in band power,
Employing deep autoencoder architectures (stacked and denoising autoencoders)
▻ Other Methods wavelet packet decomposition (WPD) in feature extraction.
▻ Add new methods common spatial patterns (CSP).
Future Work
44

45. Future Work
45
HEX-O-SPELL
▻ Enhancing algorithm : allow the arrow to rotate clock wise and counter clock wise
to speedup character selection process)
▻ Add Features autocomplete word, type integers (0-9), or symbols (&, # …) or
operate in multiple languages

46. PUBLICATIONS
▰ M. A. Helal, S. Eldawlatly and M. Taher, "Using Autoencoders for Feature
Enhancement in Motor Imagery Brain-Computer Interfaces," 2017 13th IASTED
International Conference on Biomedical Engineering (BioMed), Innsbruck, Austria, pp. 89-93,
2017.
▰ M. A. Helal, S. Eldawlatly and M. Taher, "A Brain-Computer Interface Hex-O-spell
Application for Mobile Devices,” in preparation.
46

47. 47
THANK YOU