2. (ICCE). Dr. Flynn, founder and chief executive officer of Blue
Marble Game Company, is a world-renowned expert with a
unique perspective on therapeutic neurogaming products.
Based on her experience and the feedback from the audience, it
behooves me to elaborate on the state of the art of BCI technol-
ogy and products.
BACKGROUND AND ROAD MAP
In 1929, neuroscientists started to observe the primary cur-
rents generated by synchronous firing of large populations of
neurons in the brain. The secondary currents induced as the
extracellular return currents are measurable as extracranial
electric potentials by electroencephalography (EEG), reflect-
ing human cognizance. These research activities intensified
after the 1970s. The Wadsworth Center in New York [15] cre-
ated a BCI system that incorporated electronic signals from
the brain into a novel communication-and-control device in
1991, using as many as 75 sensors. However, this device
required institutional electronic expertise, and it was very
costly. It took the next 20 or so years for a consumer-grade
product to be introduced. Since 2012, BCI headsets using one
Brain–Computer
Interface Technology
and Development
By Narisa N.Y. Chu
The emergence of imprecise
brainwave headsets
in the commercial world.
Digital Object Identifier 10.1109/MCE.2015.2421551
3. Date of publication: 15 July 2015
july 2015 ^ IEEE ConsumEr ElECtronICs magazInE 35
Im
ag
e lIc
en
sed
by In
g
r
am
pu
blIsh
In
g
to nine sensors have entered the market, with some examples
demonstrated in Figures 1 and 2, where sensor placements
have also been indicated for various settings based on standard
P300 EEG detection locations on the scalp. A road map that
highlights the BCI technology development is shown in Figure
3,
contrasting with the original research focus on brain signal
processing algorithms with the advent of many products’
introduction of headsets.
4. Due to the small signal induced by neurons, event-revoked
potentials (ERPs) of the brain activities, measured from the
outside of the scalp via sensors, require intricate interpreta-
tion. More than 50 digital processing algorithms have been
developed since 1981 to dwarf the noise, overcome attenua-
tion, and discriminate from physiological interferences due to
tissue thickness and wetness; chemical change; and stimulus
of visual, audio, and muscle movements from eye to toe. The
major schools in developing these algorithms are represented
by the five blue arrows in Figure 3. The reconstructed brain
features reflect many assumptions, sometimes taking into
consideration a priori knowledge, however, leaving plenty of
room to guess about the real intention of the brain. These so-
called “inverse problem solutions,” similar to “reverse engi-
neering” effort, demonstrated accuracy between 60% to 90%
in various controlled environments. The performance so far
36 IEEE ConsumEr ElECtronICs magazInE ^ july 2015
has significantly limited BCI usage in real life. Regardless of
the range of uncertainty, innovative applications have been tri-
aled recently, and BCI headsets (a sample product is shown at
the bottom center in Figure 3, with no commercial endorse-
ment) designed for mass consumption have made inroads for
rehabilitation and learning fun. This accelerated product intro-
duction has been quite strategic in building around an impre-
cise BCI measurement while creating special effects for
entertainment such as in games and videos. However, before
more headsets and applications become popular, it is helpful
to establish a standardized brainwave databank for identifying
and sharing brain signals from many denominations with the
aim to facilitate an intelligent search down the road. This stan-
5. dardization effort would represent a broader collaboration
between software and neuroscience leading to treatments for
brain-related sickness, rehabilitation, and wellness, and proba-
bly furthering brain-triggered marketing and privacy protec-
tion. The new approach to standardization is illustrated in the
gold stream in Figure 3.
The conventional digital signal processing algorithms
based on statistics, complex mathematical transformation,
and estimation tend to be agnostic, reaching a performance
limit. Entrepreneurs have not been shy with this technology
Nz
Fp1
Fpz Fp2
AF7
F9
FT9
A1
TP9
P9
TP7
P7
PO7
PO3 PO4 PO8
8. • Meditation
• Engagement
• Relaxation
• Interest
• Focus
TP9, TP10, Fp1, Fp2
Fp1, Fp2, F7, F8, C3, C4, O1, O2
eMotiv
� EPOC (2014) � INSIGHT (2015)
Interaxon NeuroSky
� Muse (2014)
� MindWave Mobile (2012)
FIGURE 1. BCI sensor placement to detect EEG based on the
international P300 standard.
FIGURE 2. BCI products entering the market.
july 2015 ^ IEEE ConsumEr ElECtronICs magazInE 37
imprecision and have developed products with promising
benefits to eHealth, testing out the brain signals of the young
and the sick. Although growth has been proliferating where
selective parameters were chosen to represent brainwaves
within private groups, it is extremely difficult for reliable pat-
tern matching and self-learning algorithms to be incorporated
9. universally. With standardization, brainwave data from all
sources and reactions can be intelligently accumulated and
used. This new approach is facilitated by the understanding
of the threshold (TH) between the brain’s actual and inter-
preted meaning, as denoted in Figure 1. Applications can be
triggered by this threshold and go on to another proven activ-
ity such as games.
The databank should include not only the brainwave diagram
but the processing and search algorithms associated with the
brainwave, plus all prior knowledge. This databank will be well
positioned to blend fuzzy logic inference and pattern
recognition
related to big data evolution.
The sections that follow will illustrate the paths taken in
achieving the level of interpretability of brain signals to date.
The emergence of imprecise brainwave headsets in the com-
mercial world is illustrated. The current tools for research and
future development are discussed, with a recommendation to
standardize the brain-signal databank, anticipating its reach to
big data and, perhaps, cloud computing.
BRAIN SIGNAL FREQUENCY BANDS
Brain waves can be represented by six typical bands based on
the frequency range between 1 and 100 Hz, designated as the
, , , , ,Ta b c n and i bands. These band frequency ranges are
shown with a collective interpretation as follows [3]–[5].
▼ Frequency 1–4 Hz: T band, symbolizing high emotional
conditions or in a sleep stage.
▼ Frequency 4–7 Hz: i band, similar to the T band, also
symbolizing a calm and relaxed mood.
10. ▼ Frequency 8–12 Hz: a band, symbolizing smooth pat-
terns: awaken, calm, and eyes closed in a relaxed mood.
▼ Frequency 8–13 Hz: n band, desires from the sensorimotor
cortex.
▼ Frequency 12–30 Hz: b band, for desynchronized—nor-
mal awaken, open eyes, busy, churning, and concentrating.
▼ Frequency 25–100 Hz: c band, desires from somatosenso-
ry cortex for touch—busy, churning, and concentrating.
At these bands, typically a very low signal is collected by the
noninvasive sensor (in the range of 5–10 nV), while interfering
noise of 10–20 times stronger than the brain signal is measured
on the scalp. The correlation of these band signals with respect
Common
Spatial Patterns
Blind Source
Separation
Band Power
Decomposition
Assumptions and
Prior Knowledge
Stimulants: Eye,
Audio, Motion, Etc.
Brain-Wave
Databank
Architecture and
11. Intelligent Search
Conventional Approach
A New Approach
Threshhold Standardized User Brain Data
Input and Retrieval
Sensor
Interpreted
Headset
≠
FIGURE 3. A BCI technology road map—two approaches.
BCI headsets, injecting new
break points in games and
entertainment, deliver desirable
special effects that can blend
in our pursuit of wellness
and rehabilitation.
38 IEEE ConsumEr ElECtronICs magazInE ^ july 2015
to a person’s brain condition varies from group to group. In
gen-
eral, the interpretation states whether a person is in attention or
mediation. Exactly what attention and mediation means is left
for one to judge.
Various digital processing algorithms were attempted for
12. the purpose of making the brain signal interpretable via an
increase of the signal-to-noise ratio, manipulation of sensor
spatial and time domain parameters (TDPs), and fitting of a
priori knowledge and various assumptions to yield some per-
formance improvement. These digital processing algorithms
are profiled as follows.
DIGITAL BRAIN SIGNAL PROCESSING ALGORITHMS
Major research efforts have been spent developing digital
processing algorithms to identify brain signals at various fre-
quency bands. A plethora of inverse problem solving and pat-
tern-matching analyses have been applied to signals
measured from many sensors placed on a cap noninvasively
covering a person’s head. Stable reconstructions of brain-sig-
nal features could be achieved through the use of many tech-
niques. Since the measurements and modeling techniques
both contain noise and assumptions, the true solution could
hardly be totally derived from the algorithms or totally deter-
mined from the measurements. A comprehensive strategy for
dealing with noisy data could also include data filtering and
an optimal selection of geometric parameters, such as sensor
positioning. One particularly notable technique suggested
was regularization [13], which attempted to achieve a com-
promise between a close fit to the data and stability of the
algorithmic solution. By removing the high-frequency com-
ponent from a derived solution, it believed that it effectively
filtered a portion of the noise [13].
Various algorithms have demonstrated performances up to
90% accuracy in a controlled environment. They could be
highly
computationally intensive. They were primarily carried out
offline in batch processing, with unproven real-time applica-
tions. They also required stringent calibration and testing to
facilitate performance dedicated to an individual. To appreciate
the level of effort spent in the development of these brain signal
13. processing algorithms, three major categories are elaborated:
1) band-power feature extraction, 2) common spatial patterns
(CSP) analysis, and 3) statistical source separation [6]. The year
cited, spanning from 1981 to 2014, refers to approximately the
first introduction of the algorithm.
Band Power Feature extraction
1) band-pass filtering and power estimation taking tem-
poral average
2) periodogram (Fourier decomposition)
3) power spect ra l density f rom autoregressive
coefficients
4) wavelet scalogram (time-scale representation)
5) spectrogram (time-frequency decomposition with
averaged spectrums over time) [2008].
cSP analySiS
1) spatial filtering (SF)
i) bipolar
ii) Laplacian
2) physical forward modeling: inverse solution methods
i) Minimum current estimate [2008, 2009]
– Focal underdetermined system solver [1995]
ii) Weighted minimum norm estimate
– LORETA, eLORETA, and sLORETA—all
assuming smoothness [1987, 1994]
iii) Mixed norm estimate, combining sparsity and
smoothness [2008]
15. StatiStical Backward Modeling:
Blind Source SeParation
1) linear classifiers—linear discriminant analysis (LDA),
support vector machine (SVM), and infinite impulse
response [2010]
2) linear regression—OLS, ridge regression, LASSO
[2005]
3) principal component analysis (PCA) [2005]
A comprehensive strategy for
dealing with noisy data could
also include data filtering and
an optimal selection of
geometric parameters, such
as sensor positioning.
july 2015 ^ IEEE ConsumEr ElECtronICs magazInE 39
4) canonical correlation analysis (CCA)—hyperscanning
ERP studies [2011]
5) independent component analysis (ICA)
6) Granger-causally interacting—SCSA, MVARICA—
brain connectivity studies [2008]
7) dimensionality reduction—stationary subspace analy-
sis (SSA) [2009].
Performance data ranging from 60.7%—with rudimentary
band-pass filtering—to 92.8%, with DFCSP, have been dem-
16. onstrated in carefully calibrated and tested setup.
It should be noted that every method performs well if its
specific assumptions are met. Unfortunately, no method can
perform well in all real cases. It is anticipated that multiple
methods may be combined to lead to better solutions. It is also
possible that multiple methods exist for the same solution.
With a common goal to characterize brain activity of interest,
there is still no assurance that one method is better than others
in all circumstances.
Assumptions play a major role in the derivative and con-
vergence of these methods:
1) Assumptions often made with inverse analysis and blind
source separation:
Brain activity is assumed to be:
– correlated with behavior or stimulus variables [OLS,
ridge regression, LASSO]
– reflected in the strongest components of EEG [PCA]
– correlated across subjects/stimulus repetitions [CCA]
– stationary, major signaling stays local, unaffected by
neurons away from the measuring point [SSA]
– different from experimental conditions [LDA, SVM].
Brain components are assumed to be:
– mutually independent [ICA]
– Granger-causally interacting [SCSA, MVARICA].
2) Not all EEG phenomena are phase-locked to certain
events. There are rhythms depending on the mental state.
Most rhythms are idle, attenuated during activation (e.g.,
eyes open/close, arm at rest/moves).
17. 3) Sensor-spatial analysis assumes smoothness and sparsity
where neighboring voxels (discrete volume elements)
show similar activity, and only a small part of the brain is
active for a single task.
A limited evaluation has provided some insight as to how
to improve the BCI performance. The performance of the
digital brain signal algorithmic processing has reached nearly
93% under a controlled environment. For example, the basic
band-power approach demonstrated 60.7% average accuracy
in a BCI competition [6]. Applying the Laplacian SF tech-
nique, the average accuracy reached 68%. Using the bipolar
SF technique, the average accuracy was improved to 70.5%,
slightly better than Laplacian. Combining supervised regulat-
ed spatial filters on EEG with CSP and TDP, the average
accuracy was demonstrated between 78.7 and 88.9%, a range
too large to lend enough confidence. Another combination of
TDP, SF, and CSP provided 80.1% accuracy. Using filter
bank CSP, the average accuracy achieved was between 81.1
and 90.9%. The best accuracy of 92.8% was accomplished by
adopting DFCSP [6]. Thus, the performance has not demon-
strated enough robustness for all known algorithms, to say
the least.
One reaches a diminishing return if one continues searching
for better algorithmic processing of brain activities. Training
and testing input have been employed to augment these algo-
rithms. Recently, another notable approach by means of virtual
reality and gaming has opened up new frontiers for BCI, in lieu
of the imprecise algorithms.
BCI TOOLS FOR DEVELOPMENT ACCELERATION
Two major research tools have been developed extensively:
BCI2000 [15] from the Wadsworth Center in New York and
BCILAB [8] from the University of California, San Diego, the
18. Swartz Center of Computational Neuroscience (SCCN). These
pioneering platforms of the BCI were initiated for the disabled
to operate wheelchairs/computers. Later, they were extended to
support more efforts for rehabilitation, education, and enter-
tainment purposes. Recently, commercial tools offered by Neu-
rosky [5], Interaxon [4], and eMotiv [3], bundled with
products, have also been made available with various degrees
of open utility and readiness. Most of the commercial software
development kits (SDKs) present various degrees of stability
and maturity. There is enough momentum from the develop-
ment community to trial and inject further “kickstarter” initia-
tives. Benefiting from more than two decades of neuroscience
research and development, these tools have become widely
available within the last several years. Table 1 summarizes the
status of these tools.
It is clear that both a vigorous research tool and a product
realization kit can be chosen for development. It should also be
noted that the number of sensors gathering brain signals has
purposefully decreased significantly in commercial product
realization. Some of the earlier algorithms of broader spatial
coverage might lose their effectiveness as sensor placements
are minimized for user comfort.
With all due diligence exploring the intricate brain, it
becomes evident that collective efforts among engineering,
medical, and user disciplines have to come together for
optimal use of the brain signal data. Thus, forming a stan-
dardized databank should be an obvious next step. The
naming preference, “databank” instead of “data base,” is to
acknowledge the amount and the dynamics of the data (big
data) associated with various algorithms, training, testing,
and feedback that can eventually realize brain activities. It
Collective efforts among
engineering, medical, and user
19. disciplines have to come together
for optimal use of the brain
signal data.
40 IEEE ConsumEr ElECtronICs magazInE ^ july 2015
is necessary to recognize user brain reactions eventually
across the board: gender, age, environment, stimulus, intent,
processing algorithm, intelligent search, and pattern associ-
ation, not to omit involvement from big data, cloud comput-
ing, and sensor networking.
This databank, built upon a standard brain signal profile
format plus intelligent linking factors can be destined to make
the retrieval and triggering function much more timely and
meaningful. Once standardized, new applications and benefits
can be then accelerated beyond imagination. Attributes that
Rehabilitation
Computer and
Wheelchair
Operation
Attention Oblivion Defocused Neglect Random Hyped
Marketing Applications
Like Dislike Maybe Trial Committed Etc.
Signal Characteristics + Algorithmic Processing
Feature
Vector
20. Gamma
25~100 Hz
Beta
12~30 Hz
Alpha
8~12 Hz
Mu
8~13 Hz
Theta
4~7 Hz
Delta
1~4 Hz
Empirical
Composite
Learning Factor from
Training and Testing
Band
Power
Spacial Filter
Temporal Average
Periodogram
Fourier
Decomposition
Power Spectral
21. Density + AR
Coefficients
Wavelet Scalogram
Time-Scale
Representation
Spectrogram Time-
Frequency
Decomposition
Analysis
Type
ERP: Inverse
Solution
and CSP
BSS, Statistic
Modeling
CCA ICA LDA PCA SPoC SSA
Entertainment Directives
22. Happy Sad Excited Scary Disgusted Don’t Care Etc.
Database Classifier
Gender Age Eye
Blinking
Audio
Effect
Facial Muscle Movement: Chewing,
Clenching Teeth, Grinding Jaw, Etc.
Hand/Foot
Imagery Motion
Chemical, Light,
Color, Etc. Stimulant
FIGURE 4. Attributes in a brain signal databank.
Table 1. The BCI SDK/platform [3]–[5], [7], [14]–[16].
Company/
university Product/Platform Year
23. sensor/
Channel tools/Platforms apps
NeuroSky MindWave Mobile 2013 1 MWM SDK Rehab for
ADD, stroke; education,
entertainment
Interaxon MUSE 2014 7/4 Basic SDK for con-
nection
Entertainment
device control
eMotiv EPOC 2014 9/14 SDK Lite Entertainment neurotherapy
Insight 2015 9/5
SCCN/
UCSD
BCILAB/
LSL
2012 Many Open/
24. MATLAB
Focus: Comparative evaluation of
BCI methods
Wadsworth Center, New York BCI 2000 2010 Many Open
Rehab + general purpose
Various Developers Pyff in Python 2010 – Open and free
Standardization of feedback and
stimulus
july 2015 ^ IEEE ConsumEr ElECtronICs magazInE 41
require consideration in such a standardization effort are intro-
duced in Figure 4 as a starting point. These attributes are not
meant to be exclusive.
CHALLENGES IN TIME
One would wonder about the time frame for the BCI to become
fully developed from the first patent/prototype to mass produc-
tion for general consumption. Some historical timelines of simi-
lar technology can be referenced. Considering the following
25. track records:
▼ Wired telephone: patent established in 1876 to mass pro-
duction in 1970s
▼ Mobile phone: 1946–1994 (iPhone was introduced in 2007)
▼ Brain-wired cap: from 1981 onward
▼ Brain headset: since 2012, limited products have been
introduced.
One can thus anticipate the acceleration point on BCI
production
in the next two decades or sooner. A few crucial questions are
still
waiting to be tackled by the research community [6], [11], [12].
▼ What are the fundamental accuracy limits imposed by the
current EEG sensors?
▼ What assumptions are widely agreeable, and what empiri-
cal data are required to improve the accuracy of the avail-
able mathematical models?
▼ How can hierarchical models be constructed to include data
from multiple people, environments, and applications?
26. ▼ As the brain-signal performance improves, how will sensor
convergence be handled?
▼ How are auxiliary data included (e.g., muscle movement,
eye contact, and chemical change)?
▼ How can designing methods directly target real-world
applications with robustness?
▼ Will there be standardization of the brain databank?
▼ What are the privacy issues?
SUMMARY
Major thrusts combining neuroscience, sensor chip design, and
software development have already shown remarkable advance-
ment, regardless of the many uncertainties and challenges.
Entrepreneurs have started to capture the low-hanging fruit
from
the BCI technology evolutionary “branches.” The BCI headsets,
injecting new break points in games and entertainment, deliver
desirable special effects that can blend in our pursuit of
wellness
and rehabilitation. To foster these promises, brainwave
databank
27. standardization can play a major role in converging the collec-
tion and utilization of users’ essential, private brain
information,
following the example of DNA and fingerprints.
ABOUT THE AUTHOR
Narisa N.Y. Chu ([email protected]) is the cofounder of
CWLab International and is currently focusing on BCI
research. She is also a member of the IEEE Consumer Elec-
tronics Society Board of Directors. It is with the latter role
that she contributed to the writing of this article.
REFERENCES
[1] P. Herman, G. Prasad, T. M. McGinnity, and D. Coyle,
“Comparative
analysis of spectral approaches to feature extraction for EEG-
based
motor imagery classification,” IEEE Trans. Neural Syst. Rehab.
Eng.,
vol. 16, no. 4, pp. 317–326, Aug. 2008.
[2] N. Brodu, F. Lotte, and A. Lécuyer, “Exploring two novel
features for
EEG-based brain-computer interfaces: Multifractal cumulants
and pre-
dictive complexity,” Neurocomputing, vol. 79, no. 1, pp. 87–94,
28. 2012.
[3] eMotiv. (2015). EPOC & INSIGHT. [Online]. Available:
http://
emotiv.com/store/compare/
[4] Interaxon. (2015). MUSE. [Online]. Available:
https://sites.google.
com/a/interaxon.ca/muse-developer-site/data-files
[5] NeuroSky. NeuroSky MindWave Mobile [Online].
Available: http://
www.braincorner.com.tw/en/?product=mindwave_mobile_mynd
play
[6] S. Haufe. (2014). Berlin brain-computer interface, Winter
School
Lecture. [Online]. Available:
bbci2014_haufe_statistical_models_01.
pdf
[7] C. Kothe. (2012). Lectures, SCCN/UCSD. [Online].
Available: http://sccn.
ucsd.edu/wiki/Introduction_To_Modern_Brain-
Computer_Interface_Design
[8] C. Kothe. (2013). BCILAB, SCCN/UCSD. [Online].
Available: ftp://
sccn.ucsd.edu/pub/bcilab
[9] S. Haufe, V. V. Nikulin, A. Ziehe, K.-R. Muller, and G.
Nolte, “Com-
29. bining sparsity and rotational invariance in EEG/MEG source
reconstruc-
tion,” NeuroImage, vol. 42, no. 2, pp. 726–738, June 2008.
[10] S. Haufe, V. V. Nikulin, A. Ziehe, K.-R. Muller, and G.
Nolte, “Esti-
mating vector fields using sparse basis field expansions,” in
Advances in
Neural Information Processing Systems, vol. 21, Cambridge,
MA: MIT Press,
2008, pp. 617–624.
[11] Y. Zhang and Y. Ding, “CTL model update for system
modifica-
tions,” J. Artificial Intell. Res., vol. 31, pp. 113–155, 2008.
[12] K.-R. Muller, S. Fazli, P. von Bunau, F. Meinecke, W.
Samek, G.
Curio, and B. Blankertz. (2014). BCI and nonstationarity. BBCI
Winter
School, Berlin. [Online]. Available:
bbci2014_muller_non_stationarity_01.pdf
[13] F. Lotte. (2014). Oscillatory EEG-based BCI design: Signal
process-
ing and more. InriaBordeaux Sud-Ouest, France, Potiocteam.
[Online].
Available: bbci2014_lotte_signal_processomg_01.pdf
[14] I. F. Gorodnitsky, J. S. George, and B. D. Rao,
30. “Neuromagnetic source
imaging with FOCUSS: A recursive weighted minimum norm
algorithm,”
Electroencephalogr. Clin. Neurophysiol., vol. 95, no. 4, pp.
231–251, Oct.
1995.
[15] C. Brunner, G. Andreoni, L. Bianchi, B. Blankertz, C.
Breitwieser,
S. Kanoh, C. A. Kothe, A. Lecuyer, S. Makeig, J. Mellinger, P.
Perego, Y.
Renard, G. Schalk, I. P. Susila, B. Venthur, and G. R. Muller-
Putz.
(2012). BCI software platforms. [Online]. Available:
http://people.
rennes.inria.fr/Anatole.Lecuyer/BOOKBCI_bciplatforms12.pdf
[16] T. M. Vaughan, D. J. McFarland, G. Schalk, W. A.
Sarnacki,
D. J. Krusienski, E. W. Sellers, and J. R. Wolpaw, “The
Wadsworth BCI
research and development program: At home with BCI,” IEEE
Trans.
Neural Syst. Rehab. Eng., vol. 14, no. 2, pp. 229–233, June
2006.
[17] B. Venthur, S. Scholler, J. Williamson, S. Dahne, M. S.
Treder,
31. M. T. Kramarek, K. R. Muller, and B. Blankertz, “Pyff—A
pythonic
framework for feedback applications and stimulus presentation
in neuro-
science,” Front. Neurosci., vol. 4:179, Dec. 2010.
[18] EEG Resources. (2014, Jan. 5). Biomed research, Middle
East Med-
ical Information Center and Directory. Epilepsy Awareness
Program.
[Online]. Available:
http://www.biomedresearches.com/root/pages/
researches/epilepsy/eeg_resources.html
Some of the earlier algorithms
of broader spatial coverage might
lose their effectiveness as sensor
placements are minimized
for user comfort.