fNIRS and Brain Computer Interface for Communication

fNIRS and Brain Computer
Interface for Communication
Dr. Ujwal Chaudhary and Dr. Bettina Sorger present groundbreaking
research in the field of fNIRS-based BCI for communication with
healthy subjects and patients in completely locked-in states

InsideScientific is an online educational environment
designed for life science researchers. Our goal is to aid
in the sharing and distribution of scientific information
regarding innovative technologies, protocols, research
tools and laboratory services.
JOIN FOR FREE AT WWW.INSIDESCIENTIFIC.COM

Bettina Sorger, PhD
Assistant Professor
Leader of Hemodynamic BCI group
Cognitive Neuroscience Department
Faculty of Psychology and Neuroscience
Maastricht University
Functional Near-Infrared Spectroscopy (fNIRS):
A promising BCI technique
ψ

1. What is a brain-computer interface (BCI)?
• definition, methodology, application
2. Functional-neuroimaging methods for BCIs
• neuroelectric vs. hemodynamic methods
• BCI-relevant characteristics of hemodynamic brain signals
3. Functional near-infrared spectroscopy (fNIRS)
• working principles, practical/procedural aspects, information encoding,
advantages and limitations
4. A multiple-choice fNIRS-based communication BCI
5. Summary and conclusion
Overview of Presentation

1. What is a brain-computer interface (BCI)?

1. What is a BCI?
Definition and application possibilities
A brain-computer interface (BCI) is system
which connects a (human or animal) brain with
a computer.
It allows for controlling a external devices
through brain activity, i.e., without having to
rely on motor output.
BCIs can be used for researching, enhancing,
assisting, repairing or replacing cognitive or
sensory-motor functions.

1. What is a BCI?
General BCI methodology
Wolpaw et al., 2002

Electrophysiological vs. hemodynamic methods
electrophysiological (direct)
methods
• Electroencephalography (EEG)
• Magnetoencephalography (MEG)
• Electrocorticography (ECoG)
• Intracortical recordings (ICoR)
• Functional near-infrared spectroscopy (fNIRS)
• Functional magnetic resonance imaging (fMRI)
Figure obtained from here

• Electroencephalography (EEG)
• Magnetoencephalography (MEG)
• Electrocorticography (ECoG)
• Intracortical recordings (ICoR)
• Functional near-infrared spectroscopy (fNIRS)
• Functional magnetic resonance imaging (fMRI)
Figure obtained from here
Electrophysiological vs. hemodynamic methods
hemodynamic (indirect)
methods

Ratings ranging from highly advantageous (+ + +) to extremely disadvantageous (- - -). Sorger (PhD thesis, 2010)
Advantages and disadvantages of functional-neuroimaging methods for BCI purposes

0 3 10 time [s]
neuroelectric response hemodynamic response
100ms
BCI-relevant characteristics of hemodynamic signals (1): delay

BCI-relevant characteristics of hemodynamic signals (2): high single-trial reliability
Resting periods (20s)
Motor-imagery periods (10s)
fMRIresponse
fMRIresponse
Time
0 10 Time [s]

BCI-relevant characteristics of hemodynamic signals (3a): sufficient spatial resolution
Hemodynamic (fMRI)
brain-activation patterns
related to:
• Motor imagery
• Mental calculation
• Inner speech

BCI-relevant characteristics of hemodynamic signals (3b): sufficient spatial resolution
Motor imagery
Mental calculation
Inner speech
fNIRS-coverage mask

Working principles of fNIRS
Red light penetrating in biological tissue
Figure obtained from here adapted from Naseer (2013)
Head surface
Banana-shaped
pathway of photonsAbsorption spectra of Hb
Figure adapted from here
Hb HbO2

Practical/procedural aspects of fNIRS (1)
NIRScout (NIRx)

Source and detector optodes placed on the
head surface using an elastic cap
Example optodes montage (covering prefrontal and
occipital cortex) following the 10-20 EEG system
CZ
sources
detectors

Spatial coverage depending on
optode montagefNIRS-coverage
mask

Information encoding in fNIRS-BCIs
How can we enable patients to encode different
information or intentions into differentiable fNIRS
signal?

“yes” “no”
“yes” “no”
thinking “yes” thinking “no”

“yes” “no”
Inner speechMotor imagery

• exploitable fNIRS-signal features (spatial, temporal and magnitudinal)
Motor imagery
Mental calculation
Inner speech

20s10s0s
fNIRSresponse
Start of encoding time window

fNIRSresponse

• two general forms information-encoding strategies:
- intentionally controlled mental activities
1. higher-order cognitive tasks (e.g., mental calculation)
2. covert language-related tasks (e.g., mental speech, mental singing)
3. mental-imagery tasks (motor, visual, auditory, tactile and emotion imagery)
4. selective-attention tasks (visual, auditory, and tactile attention)
- automatic cognitive processing

• relatively low purchase/operating costs [fMRI]
• medium temporal resolution (sample rate: up to 20Hz, but cave hemodynamic delay) [fMRI]
• sufficient spatial resolution (1cm³ range)
• no vulnerability to electromagnetic environment [EEG, fMRI]
• relatively low sensitivity to head motion artifacts [EEG, fMRI]
• no inverse problem [EEG]
• ease of application [EEG, fMRI]
• portability/mobility [fMRI]
• harmlessness, non-invasiveness
• noiselessness [fMRI]
• compatibility with paramagnetic (medical) equipment [fMRI]
• measuring both oxygenated and deoxygenated Hb changes ( two dependent variables) [EEG, fMRI]
• compatibility with EEG, fMRI, TMS/tCS ( simultaneous measurements possible)
Advantages of fNIRS as BCI technique

NIRSport –
a mobile fNIRS system

Limitations of fNIRS as BCI technique
• light scattering/absorption by extracerebral tissue and skin/hair pigmentation
• limited depth pervasion (reaching only surface layers of cortex)
• still-existing practical limitations
Figure adapted from here

Overview of existing hemodynamic communication BCIs
×
Naito et al. (2007)
Gallegos-Ayala et al. (2015)
Chaudhary et al. (2017)

Option “D”
Option “C”
Option “B”
Option “A”
Answer encoding by exploiting temporal fNIRS signal features
Figure obtained from here A B C D

Instruction:
“Please imagine drawing simple
geometric figures (such as circles,
triangles, cubes, etc.) or small contour
drawings (e.g., a butterfly, star, car,
tree, boat, or house) with your right
hand in a comfortable but consistent
speed. Try to imagine using a pen.
This might support your imagination”.
Mental drawing

Answer encoding with auditory guidance
Question: “What is your favorite color?”
time
@20s
“Red” “Green”
@30s
“Black”
@40s
“Other”
@50s
“Stop”
@60s
@0s
→ 5 repetitions (within 6:05 min) to increase accuracy

Option “Other”
Option “Black”
Option “Green”
Option “Red”
Expected (ideal) fNIRS responses

General procedure

Equidistant optode positioning on left sensorimotor cortex

Online Data analysis using Turbo-Satori
developed by Rainer Goebel
(Brain Innovation B.V., Maastricht, the Netherlands)
Time-course inspection of localizer data for best-channel selection
(Goebel & Luehrs, submitted)

*
Best-channel selection based on general linear model (GLM) analysis
(channel with maximal mean of oxy- and deoxy-Hb t-values)
GLMoutput[y-value]
Channel

fMRI data of same
participant
Time course inspection of answer-encoding data (oxy-Hb) within best-channel
(subject encoded answer option “A”)

for
“Red”
for
“Green”
for
“Black”
for
“Other”
Answer decoding: GLM analysis of the best-channel’s time course
(using response functions according to the four different answer periods)

Answer decoding: Plotting GLM results and deriving encoded answer
(based on maximal mean of oxy- and deoxy-Hb t-values within selected channel)
Example question: “What is your favorite color?”
*
GLMoutput[y-value]
Answer option
S8-D8

Answer encoding results
25.0%-97.5% (mean: 62.8%)
*
*
*
*
*
*
*
*
*significant at p<.05
(8 out of 9 participants, ca. 89%)
(theoretical) chance level (25%)
37.5%-100% (mean: 76.4%)
1st decoder option correct
4th decoder option correct
3rd decoder option correct
2nd decoder option correct

✓ fNIRS constitutes a promising BCI method
• non-invasiveness, harmlessness, (repeated/frequent usability)
• Portability/mobility, ease of use (applicability)
• high-single trial reliability (robustness)
• high flexibility (individualization potential )
• relatively low costs
✓ high potential for further improvement of fNIRS methodology
(hard- and software)
 fNIRS highly suited for clinical and daily-life BCI applications
Summary & Conclusions

Acknowledgements
Colleagues
Rainer Goebel
Rico Möckel
Lars Riecke
PhD students
Amaia Benitez-Andonegui
Hannah Boeijkens
Michael Luehrs
Laurien Nagels-Coune
Internship students
Denizhan Kurban
Louisa Gossé

Dedicated Near-Infrared
Spectroscopy
LEARN MORE
Thank you to our event sponsor

Dr. Ujwal Chaudhary
Research Group Leader,
Institute of Medical Psychology,
University of Tübingen
Research Fellow
WYSS Center for Bio and
Neuroengineering, Genèva
fNIRS and Brain Computer
Interface for Communication

1. What is Brain Computer Interface (BCI)?
2. BCI for Communication
3. Functional Near Infrared Spectroscopy (fNIRS)
• Signal processing
4. fNIRS based BCI
• Lifecycle of BCI
• Feature selection
5. fNIRS based BCI for Communication
• Results from patient population
Presentation Agenda

The human being lives according to its communication capacity: loosing the
capacity for communication means loosing life.
Der Mensch lebt in dem Maße, wie er kommunikationsfähig ist: ist die
Kommunikationsfähigkeit vorbei, so ist auch das Leben vorbei.
L’uomo vive in base alla sua capacita di communicazione, quando la
communicazione e finita finisce anche la vita
Ludwig Hohl
Communication

• Amyotrophic Lateral
Sclerosis
• Stroke
• Brain Inujry
• Muscular dystrophy
Loss of
Communication
Augmentative
and Alternative
Communication

What is Brain Computer Interface?
Revolves around the idea to convert thoughts
directly into movement without the detour via
the motor system
Coined for the first time by French
neurophysiologist Jacques Vidal in 1973

Chaudhary et al., Nature Reviews Neurology 2016.

Neuroimaging Modalities
1. Electroencephalography (EEG)
2. Magnetic Encephalography (MEG)
3. Functional Magnetic Resonance Imaging (fMRI)
4. Near Infrared Spectroscopy (NIRS)

1. Electroencephalography (EEG)

2. Magnetic Encephalography (MEG)

3. Functional Magnetic Resonance Imaging (fMRI)

4. Near Infrared Spectroscopy (NIRS)
Source Detector

BCI for Communication
1. EEG-BCI
• Slow Cortical Potential (SCP)- BCI
• Sensorimotor rhythm (SMR)-BCI
• P300-BCI
2. fNIRS-BCI
Rockstroh et al., 1989; Birbaumer, 1997

Near Infrared Spectroscopy (NIRS) of Brain : Principle
Minimally absorbed and Preferentially scattered.
Source Detector distance 2.5 – 3.5 cm
Optical WindowLight Propagation in Brain
Strangman et al. Biol Psychiatry. 52 (2002) 679-693.
Source Detector
Ting Li et al Journal of Biomedical Optics 16(4),
045001(April 2011)

Three
different
analysis
Analysis of NIRS Signal
Activation analysis
Connectivity analysis
Lateralization analysis

Activation Analysis
• The raw optical data is filtered using a band pass filter to
remove the signals arising from systemic physiology.
• Modified Beer Lambert law
• Change in relative concentration of HbO, HbR and HbT
(sum of HbO and HbR) as a function of time.
• Statistical Analysis (Statistical significance)

Brain Connectivity Analysis
• Brain connectivity refers to the pattern of connection and
interaction between units of brain.
• Forms a distributed network wherein the different regions
of the brain communicate with each other.
• Three types of brain connectivity.
Friston, K.J., 1994. Hum. Brain Mapp. 2, 56–78.

1) Anatomical/Structural connectivity which
describe the anatomical links between
different units in the brain.

1) Anatomical/Structural connectivity which
describe the anatomical links between
different units in the brain.
2) Functional connectivity which describes the
temporal correlation among the activity of
different neurons.
✓ It measures simultaneous coupling
between two time series.
✓ Agnostic to the directional or causal
relationship between two time series

3) Effective connectivity which describes the
causal interaction between distinct units
within a nervous system.

Lateralization Analysis
Hemispherical dominance of the one cortical region over other during the
task or even during the resting state of brain.
+ve laterality index = left-side dominance
-ve laterality index = right-side dominance
≈ 0 laterality index = no dominance, also termed as bilateral activation.
)(HbT)(HbT
)(HbT)(HbT
L(t)
rightleft
rightleft
tt
tt



L (t) - Laterality index
HbT (t) – Total hemoglobin

Applications of NIRS
Activation Study
• Auditory
• Audio Visual
• Cognition
• Motor
• Language
• Resting state
• Somatosensory
• Visual
Healthy:
Adults, Children
and Infants.
Neurological Disorder:
Addiction, Alzheimer,
Brain Ischemia,
Depression, Epilepsy,
Hematoma, Seizure,
Schizophrenia, Sleep
apnea, Stroke
rehabilitation.
Subject Groups

BCI for Communication in
Locked-in State Patient
If we are left alone everything becomes unbearable.
Wenn wir allein gelassen werden, ist alles zu viel.
Si nos dejan solos, todo esta de mas.
Antonio Porchia

Amyotrophic Lateral Sclerosis
• ALS: A progressive
motor disease
• No Treatment
• Artificial Respiration
• Locked in State (LIS)
• Completely Locked in State
(CLIS)
• Only affecting sensory and
cognitive functions to minor
degree.
• Communication

• Birbaumer et al.
Nature (1999)
ALS Patient using SCP-BCI
• Kübler & Birbaumer
(2008)
Clin.Neurophysiol.
• Birbaumer et al.
Current Opinion in
Neurology, 2008

• Several techniques for communication in LIS.
• Previously trained patient did not transition.
• No motor channels left for communication.
• Patient in CLIS without previous BCI training
• No techniques work.
• Extinction of goal directed thinking proposed.
• Circadian rhythm disrupted.
Transition from LIS to CLIS

BCI for Communication in
Completely Locked-in State Patient
We conclude that we would remain in a state of anesthesia
(numbness) if our perception would not have a particular
preference for this or that.
Man sieht daraus, dass wir stets in einem Zustande der Betäubung
verharren würden, wenn unsere Perception nicht sozusagen eine
hervorstechende Eigentümlichkeit und eine bestimmte Vorliebe für
dieses oder jenes besässen.
Gottfried Wilhelm Leibniz (1646-1716)

No Means of
Communication
With permission of
parents (caretakers)

ALS patient attending to
auditory stimuli
NIRS optodes + EEG electrodes placed
on the motor region of the patient
*Patient’s face revealed with
permission from primary caretaker
Chaudhary et al., PLoS Biology 2017.Chaudhary et al., Nature Reviews Neurology 2016

Semantic Classical Conditioning
Thinking
YES
„Berlin is the Capital of Germany“
Change in fNIRS
EEG
(EKG)
(EOG)
Thinking
NO
„Berlin is the Capital of Spain“
Change in fNIRS
EEG
(EKG)
(EOG)
Furdea et al. (2012) J.Neurosc.MethodsDeMassari et al, Brain (2013)

auditory stimuli
Patients are presented 20 personal
questions, 10 true and 10 true false in
random order in each session
Patient thinking “ja,ja,…” (yes) and
“Nein,Nein,…” (no) for true and false
sentence, respectively for 15 sec.

NIRS Signal for
Yes and No
EEG Signal for
Yes and No
NIRS Signal Acquisition System EEG Signal Acquisition System
auditory stimuli

NIRS Signal for
Yes and No
EEG Signal for
Yes and No
auditory stimuli
Sleep Monitoring and
offline signal classification

NIRS Signal for
Yes and No
EEG Signal for
Yes and No
Audio Feedback ALS patient attending to
auditory stimuli
Feature Extraction
Changes in oxy or de-oxy
hemoglobin for Yes and No
Train the NIRS SVM classifer
Sleep Monitoring and
offline signal classification

Hemodynamic Change of “yes” and “no” Response

Feature Extraction & Machine Learning
Linear and non linear classifier:
Support vector machine,
Linear Discriminant Analysis,
and others.
Features:
Mean
Slope
Variance
Moving average
Root mean square
Skewness

Classification Accuracy: Patient F
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14
30
40
50
60
70
80
90
100
Mean classification accuracy (CA)
Training session chance level threshold
Feedback session CA
Open question session CA
Feedback session chance level threshold
+
Classificationaccuracy(%)
30
40
50
60
70
80
90
100
B. EEG classification accuracy
Number of days
A. fNIRS classification accuracy
1 2 3 4 5 6 7 8 9 10 11 12 13 14
C. EOG classification accuracy

0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
A. Training Sessions’ ROC Curve
False Positive Rate
TruePositiveRate
TruePositiveRate
Random
Above
Chance
Random
Below
Chance
Above
Chance
Below
Chance
B. Feedback Sessions’ ROC Curve
(0.71,0.33) (0.68,0.28)
False Positive Rate
Receiver Operating Characteristic Curve: Patient F
Total Sentences Predicted True Predicted False
True Sentence True Positive False Negative
False Sentence False Positive True Negative
True positive rate =
Ʃ 𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
Ʃ 𝑇𝑟𝑢𝑒 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒𝑠
False positive rate =
Ʃ 𝐹𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
Ʃ 𝐹𝑎𝑙𝑠𝑒 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒𝑠
Where…
and

Feedback
Sessions
With permission of

With permission of
Open Question
Sessions

• Numerous studies on able bodied individual
• Limited controlled, systemic clinical trial on patient population.
• fNIRS based BCI for Communication in CLIS
• BCI for rehabilitation in stroke
• Controlled clinical study in patient population
• BCI for motor control in stroke, tetraplegia and Spinal cord Injury.
Conclusion

Niels Birbaumer
Aygul Rakihmkulova
Azim Maleksahi
Alessandro Tonin
Alberto Lucas
Stefano Silvoni
WYSS Center
John Donoghue
Ali Zaidi
Thank you!

Thank You
Dr. Bettina Sorger,
Assistant Professor,
Dept. of Cognitive Neuroscience
Maastricht University
b.sorger@maastrichtuniversity.nl
Dr. Ujwal Chaudhary,
Research Group Leader,
Institute of Medical Psychology,
University of Tübingen
uchau001@fiu.edu
For additional information on the products and applications
presented during this webinar please visit www.nirx.net

fNIRS and Brain Computer Interface for Communication

More Related Content

What's hot

Similar to fNIRS and Brain Computer Interface for Communication

More from InsideScientific

Recently uploaded

fNIRS and Brain Computer Interface for Communication