Search

A predictive speller for a brain-computer interface based on motor-imagery

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

4 comments

Comments 1 - 4 of 4 previous next Post a comment

  • + tizyweb Tiziano D'Albis 2 months ago
    you can download it for free from this same website: there’s a 'get file' link on top ;)
  • + akshut akshut 2 months ago
    hi can you please give me this presentation. I need it for my college seminar.
    My email is akshutjoshi2005@yahoo.co.in
    Thanks in advance.
  • + tizyweb Tiziano D'Albis 4 months ago
    Yes, it coul’d be just a visual imagination or something like movement planning and preparation.
  • + guest59d24c Saikat Ray 4 months ago
    motor imagery refers to only imagining that you are moving the limbs?
Post a comment
Embed Video
Edit your comment Cancel

Notes on slide 1

The target users of this application are people with severe motor impairments and especially people affected by the so-called locked-in syndrome. These people are conscius and can think and reason but are unable to speak or move. The impossibility to communicate clearly compromises the quality of life of these people. A primising means for giving back basic communication capabilities to these people are brain-computer interfaces.

The objective of a brain-computer interface is indeed to enstablish a direct communication channel between the brain of a person and an external device.This communication is performed in several stages.

In the first stage a signal related to neural brain activity needs to be recorded from the brain of the person.

Than the acquired signal needs to be processed for reducing noise and extract the required information to produce a control signal.

A BCI may have also a stimulation or a feedback module depending on the paradigm employed by the system.

The source signal used in our application is the EEG signal. The EEG records the eletric potentials induced by the extracellular currents that characterize the synaptic communication among neurons. The signal is acquired by means of 19 electrodes positioned on the scalp of the patient in standard locations. The signal is then amplified by means of an EEG amplifier. The EEG signal is very noisy and is easly affected by artifacts.

The paradigm adopted to extract information from the EEG signal is ‘Motor Imagery’. This paradigm is based on the fact that the sensory-motor cortex is interested by particular brain rhythms called sensory-motor rhythms. These rhythms are visible on the EEG spectrum in specific frequency bands the mu band (around 10Hz) and the beta band (around 20Hz). These rhythms are synchronyzed when there is no movement or motor imagery.

When instead we move a part of the body or simply we imagine to do that we observe an attenuation or desynchronization of these rhythms, mainly controlaterally to the movement.

These plots shows for example that when we imagine to move the left hand there is a power attenuation on the characteristics frequency bands over the right side of the scalp, while the opposite happens when we imagine to move the right hand. In our application we considered four motor imagery tasks: the motor imagery of the left hand, right hand, both hands and both feet.

The general scheme of the application developed is the following. The system is composed of three main modules. The BCI module receives as input the raw EEG signal and produces as output a control signal plus a feedback signal. The feedback is shown to the user so that he can learn to modulate its brain ryhthms. The speller module receives as input the control signal and produces as output text in natural language. Text composition is aided by the prediction module that provides predictions at the natural language level.

The BCI module translates the raw EEG signal in a control signal plus a feedback signal. This translation is performed by a processing pipeline divided in five stages: spatial filtering, spectral estimation, feature extraction, feature projection and classification.

The objective of spatial filtering is to enhance the signal localization around interesting points. In our system we used three large laplacians filters centered on electrodes C3, Cz and C4, that re-estimate the potential at these electrodes subtracting the mean of the four electrodes in the neighbourhood.

At the second stage we estimate the spectrum of the EEG signal. Indeed all the features are defined in the frequency domain and also the feedback signal is computed starting from the EEG spectrum. The spectrum needs to be estimated in real-time from a limited set of samples. For this reason we adopted a parametric approach to spectral estimation and specifically we used the Burg’s algorithm for Maximum Entropy spectral estimation. This algorithm estimates from data the coefficients of an AR process with maximum entropy. With this method we can expresses maximum uncertainty with respect to the unknown information, while being consistent with the observed data.

At the following stages we perform feature extraction, feature projection and classification. These three operations are performed at the end of a signal acquisition which lasts 6 seconds. We defined two types of features. Some feature are defined on the average spectrum estimated with the 6 seconds of the signal, other features are defined considering 6 spectra each estimated after one second during the acquisition. In total we defined 255 features which may be useful to discriminate among the four motor imagery tasks.

Anyway not all features are used for online classification. Indeed we implemented also a feature selection algorithm which selects the subset of feature that maximize the classification performance estimated offline with a cross-validation algorithm. The problem of feature selection is NP-complete and we used a genetic algorithm to solve this optimization problem. The genetic algorithm encodes a solution in a binary format assigning a 1 to a selected feature and a 0 to a discarded one.

The selected features are then projected in a space with N-1 dimensions where N is the number of classes considered. For this purpose we used the Fisher Discriminant Analysis whose goal is to find the projection which maximize the distance among class means, while minimizing the variance inside classes.

The projected features are then classified using a linear classifier. The classifier estimates for each class the parameters of an hyperplane separating the class from the remaining feature space. For classification we used the Linear Discriminant Analysis algorithm which assumes that all classes can be modeled as gaussians with the same covariance matrix. Linear classifiers are normally used in the BCI field since they are robust to outliers and they can be parametrize with a limited training dataset.

The control signal produced by the BCI module is used to send commands to the spelling application. In designing the interface of the speller we accounted for three main requirements: the usability of the system, the formalization of the structure and the behaviour of the interface, and the possibility to employ natural language processing techniques to speedup text composition.

The application provides the selection of 27 symbols the standard 26 letters of the english alphabet plus the spacing character. Moreover we considered also some auxiliary functions, to select a suggested word, to correct an error or to activate the function for vocal synthesis. Symbols and functions are arranged inside targets which are recursively expanded till the selection of a single element. Each target is associated to a class and thus to a specific motor-imagery task.

The predictive capabilities are provided at two levels: at word level, providing word suggestions that are selectable through appropriate interface menus, and at letter level disabling the most improbable symbols and thus sparing some targets’ expansions.

We considered also the errors that could be introduced while using the spelling application. We considered three categories of errors: errors introduced by the BCI (due to errors in signal classification), errors caused by a misinterpretation of the current interface state by the user and errors due to the fact that the user changes idea about what to write.

The structure and the behaviour of the interface are formalized by means of a push-down automaton, in which states are associated to interface screens and transitions determine the next screen visualized when selecting a target. The stack is instead used to keep in memory the hostory of visited states and thus modeling the error-correction functionalities in a simple way. We defined different transition types and some constraints that ensures the consistency of the interface. We defined different interface versions controlled by either a 3-class or a 4-class control signa. Moreover the formal interface model has been employed to implement a simulator program that was used to assess automatically the performance of different interface versions.

Language prediction capabilities are provided by the prediction module. This module queries a statistical language model which in turn is built starting from training corpus.

A copus is a collection of texts from which statistics are computed. We used a subset of the British National Corpus, selecting only transcribed speech acquired from informal conversations and educational contexts. Indeed these texts are characterized by a direct and simple language that is suitable for the kind of communication we want to enstablish with our application.

The statistical language model is based on N-grams. Thus the probability of a word is computed based on the N-1 preceding words. We used a Katz-backoff trigram model. The peculiarity of this model is that considers recursively orders of lower orders when there are not occurences of word sequences at higher levels. The probabilities are estimated with a maximum likelihood approach, based on counts performed on the training corpus.

The predictions are computed starting from a vocabulary containing the 20000 most frequent words in the training corpus. From that we extract a list of candidate words, that could be inserted from the current interface state. Thus we estimate the probability of each candidate word and we select the N most probable words as words’ suggestons. Starting from words’ probabilities we compute also letters’ probabilities that are used to enable or disable letters in the spelling interface.

The interaction with the application adopts a synchronous control which requires different phases. The preparation phase precedes the actual acquisition session. Than the selection of a target is performed in three steps.

In the first step the interface state is shown to the user who decides the target to select.

In the second phase the user performs the motor imagery task and a feedback signal is displayed to the user. The feedback is provided by the two green bars whose lenght is proportional to the power of the EEG signal in the mu frequency band, acquired from electrodes C3 and C4.

Finally the selected target is shown to the user before an interface transition occurs.

In this thesis we performed tests at three different levels. We tested the BCI alone (without the speller), than we performed simulations to access the performance of different interface configurations and finally we performed online tests with the whole spelling application. The tests involved three sane subjects that had not previous experiences with BCI systems. First we performed BCI acquisition sessions without feedback and the acquired data have been used to estimate the classification performance offline. Then we performed BCI session with feedback and the classification algorithm has been implemented online. The classification performances obtained both offline and online have been used to perform simulation on the speller interfaces in order to find the best solution for each subject. For subject P the interface with four classes resulted to be the best one, while for the other two subject the interface with just three classes resulted to perform better.

Performing simulations we evaluated also the relationship between the classification accuracy and the performances of the spelling application and we found that there is an explonential relationship between these two variables.

Performingsimualtions we evaluated also the impact of language predictions on the performance. We found that word predictions affect significantly the performances whie letter prediction is relevant only when there is not a suggestion at word level.

Finally we evaluated the performances of the whole system online with the three subject. They performed several repetitions of the same task, which is to write the phrase “what a wonderful day”. Subject P completed the task with an average of 7 minutes reaching a communication rate of about 3 char/min, while subject T and F reached respectively a speed of 2 and 2.7 char/min. These results are satisfactory and inline with the latest works about these systems reported in literature.

The average communcation rate with BCI systems, indeed, ranges from 0.5 to 6 characters per minute.

Favorites, Groups & Events

A predictive speller for a brain-computer interface based on motor-imagery - Presentation Transcript

  1. A predictivespeller for a
    Brain-Computer interface
    based on motor imagery
    Tiziano D’Albis
    Student ID nr. 707766
    tiziano.dalbis@gmail.com
    21, July 2009
    Airtifical Intelligence & Robotics Laboratory
    Politecnico di Milano
  2. Outline
    Introduction
    The Locked-in syndrome
    Brain-Computer Interfaces
    EEG recordings
    Motor Imagery
    The system
    System overview
    The BCI module
    The speller interface
    The prediction module
    Demo, results, conclusion
    Demo
    Experiments, tests and results
    Conclusion and future work
  3. Outline
    Introduction
    The Locked-in syndrome
    Brain-Computer Interfaces
    EEG recordings
    Motor Imagery
    The system
    System overview
    The BCI module
    The speller interface
    The prediction module
    Demo, results, conclusion
    Demo
    Experiments, tests and results
    Conclusion and future work
  4. Outline
    Introduction
    The Locked-in syndrome
    Brain-Computer Interfaces
    EEG recordings
    Motor Imagery
    The system
    System overview
    The BCI module
    The speller interface
    The prediction module
    Demo, results, conclusion
    Demo
    Experiments, tests and results
    Conclusion and future work
  5. The locked-in syndrome
    The target users for this application are people with severe motor impairments.
    The locked-in syndrome (LIS) is a neurological disorder characterized by complete paralysis of voluntary muscles.
    Individuals are conscious and can think and reason, but are unable to speak or move.
    Main causes are related to tumors, encephalitis, lesions of the brain stem, spinal-cord injuries, neuro-degenerative deseases (ALS)
    ALS patient using a BCI
    Brain stem
  6. Brain-Computer Interfaces
  7. Brain-Computer Interfaces
  8. Brain-Computer Interfaces
  9. Brain-Computer Interfaces
  10. EEG recordings
    EEG amplifier
    Electrode cap
    EEG signal
    EEG signal corrupted by ocular artifacts
    The 10-20 standard defining the electrode montage
  11. Motor Imagery
    Motor Imagery
    R^2 topography maps
    C3
    C4
    Right hand
    Left hand
    Channel C3 (left side)
    Channel C4 (right side)
    Frequency [Hz]
    Frequency [Hz]
  12. Motor Imagery
    R^2 topography maps
    C3
    C4
    Right hand
    Left hand
    Channel C3 (left side)
    Channel C4 (right side)
    Frequency [Hz]
    Frequency [Hz]
  13. Motor Imagery
    R^2 topography maps
    C3
    C4
    Right hand
    Left hand
    Channel C3 (left side)
    Channel C4 (right side)
    Frequency [Hz]
    Frequency [Hz]
  14. System overview
    feedback
    Interface state
    classes
    and feedback
    Speller
    interface
    BCI
    module
    EEG signal
    text
    trigger
    context
    predictions
    Prediction
    module
  15. BCI module
    projected
    features
    filtered EEG
    EEG spectrum
    features
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    class
    raw EEG
    BCI
    module
    classes and feedback
    EEG signal
    trigger
  16. Spatial filtering
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    Large Laplacian filtering
    No spatial filter
    Filtered with Large Laplacian
    Center
    Small Laplacian reference
    Large Laplacian reference
  17. Spatial filtering
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    Large Laplacian filtering
    No spatial filter
    Filtered with Large Laplacian
    Center
    Small Laplacian reference
    Large Laplacian reference
  18. Spatial filtering
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    Large Laplacian filtering
    No spatial filter
    Filtered with Large Laplacian
    Center
    Small Laplacian reference
    Large Laplacian reference
  19. Spectral estimation
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    • EEG signal is modeled as an autoregressive (AR) stochastic process with maximum entropy.
    • MEM chooses the spectrum corresponding to the most unpredictable time series whose autocorrelation sequence coincides with the observed values.
    • MEM expresses maximum uncertainty with respect to the unknown information, but is consistent with the known information
    Maximum entropy spectral analysis. Stanford University, Burg, J.P.
    MEM spectral estimation
    Find
    such that maximizes
    and for observed data holds
  20. Feature extraction (1/2)
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    Features computed on average trial spectrum
    Features capturing the intra-trial dynamics
    Left hand motor-imagery (C3 – C4)
    Left-hand motor-imagery (C3)
  21. Feature extraction (2/2)
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    Feature selection
    Training data
    • Feature selection has been performed using a wrapper approach
    • The objective function is proportional classification accuracy obtained offline with a k-fold cross-validation
    • The problem is NP-complete, as search method we used a genetic algorithm
    Complete feature set
    GA search
    Feature
    subset
    Classification
    accuracy
    Projection & Classification
    1
    0
    0
    0
    0
    1
    1
    0
    1
    0
    0
    0
    0
    1
    0
    1
    1
    1
    0
    1
    Chrosmosome for feature selection
    Final feature subset
    Projection & Classification
  22. Feature projection
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    Fisher Discriminant Analysis (FDA)
  23. Classification
    projected
    features
    filtered EEG
    EEG spectrum
    features
    control
    signal
    Spectral estimation
    Feature projection
    Spatial
    filtering
    Feature
    Extraction
    Classifi-cation
    raw EEG
    Linear Discriminant Analysis (LDA)
    LDA assumes that each class has a multivariate normal distribution with mean µi and that all classes have the same covariance matrix C
  24. Speller interface
    • Requirements
    • Usability
    • Formalization
    • Predictive capabilities
    • Selection strategy
    • 27 symbols, functions, numbers
    • Hierarchical target expansions
    • Predictive capabilities
    • Word suggestions
    • Disabilitation of improbable symbols
    • Handling errors
    • Introduced by the BCI
    • Caused by the interface
    • Due to a change of mind of the user
  25. Speller interface
    • Requirements
    • Usability
    • Formalization
    • Predictive capabilities
    • Selection strategy
    • 27 symbols, functions, numbers
    • Hierarchical target expansions
    • Predictive capabilities
    • Word suggestions
    • Disabilitation of improbable symbols
    • Handling errors
    • Introduced by the BCI
    • Caused by the interface
    • Due to a change of mind of the user
  26. Speller interface
    • Requirements
    • Usability
    • Formalization
    • Predictive capabilities
    • Selection strategy
    • 27 symbols, functions, numbers
    • Hierarchical target expansions
    • Predictive capabilities
    • Word suggestions
    • Disabilitation of improbable symbols
    • Handling errors
    • Introduced by the BCI
    • Caused by the interface
    • Due to a change of mind of the user
  27. Speller interface
    • Requirements
    • Usability
    • Formalization
    • Predictive capabilities
    • Selection strategy
    • 27 symbols, functions, numbers
    • Hierarchical target expansions
    • Predictive capabilities
    • Word suggestions
    • Disabilitation of improbable symbols
    • Handling errors
    • Introduced by the BCI
    • Caused by the interface
    • Due to a change of mind of the user
  28. Speller interface model
    Stack
    S(t-1)
    S(t-2)
    S(t-3)
    …
    Undo
    Menu
    Exit-menu
    Action
  29. Language predictions
    Prediction
    module
    Language
    model
    Suggested words
    Context
    Words’
    probabilities
    Letters’
    probabilities
    Composition
    context
    Training data
    Corpus
  30. Training corpus
    Predicted words
    Context
    Training
    data
    Language
    model
    Corpus
    Prediction
    module
    Letters’
    probabilities
    Words’
    probabilities
    Composition
    context
    • The training corpus is a subset of the British National Corpus (BNC)
    • We considered only transcribed speech acquired from informal conversations and educational contexts
    • All text have ben post-processed to convert all letters to lowercase and to retain only symbols considered by the spelling application
  31. Language model
    Predicted words
    Context
    Training
    data
    Language
    model
    Corpus
    Prediction
    module
    Letters’
    probabilities
    Words’
    probabilities
    Composition
    context
    Katz-Backoff trigram model
    • α is used to ensure that the probability mass from all the lower order N-grams sums up to exactly the amount saved by discounting the higher-order.
    • P∗is the MLE probability discounted to save some probability mass for the lower order N-grams
  32. Prediction module
    Predicted words
    Context
    Training
    data
    Language
    model
    Corpus
    Prediction
    module
    Letters’
    probabilities
    Words’
    probabilities
    Composition
    context
    Predicted words
    Extract a list of candidate
    words for the current
    context
    Evaluate probabilities of
    candidate wordsfor the current context
    Take the N words with highest probabilities
    Words’
    probabilities
    Vocabulary
    Candidate
    words
    Letters’
    probabilities
    Letters’ probabilities
    Evaluate letters’ probabilities
    based on words’ probabilities
  33. Application protocol
    repeat
    R
    T
    P
    S
    6s
    3s
    3s
    0.5s
    Preparation
    Thinking
    Recording
    Selection
  34. Application protocol
    repeat
    R
    T
    P
    S
    6s
    3s
    3s
    0.5s
    Preparation
    Thinking
    Recording
    Selection
  35. Application protocol
    repeat
    R
    T
    P
    S
    6s
    3s
    3s
    0.5s
    Preparation
    Thinking
    Recording
    Selection
  36. Application protocol
    repeat
    R
    T
    P
    S
    6s
    3s
    3s
    0.5s
    Preparation
    Thinking
    Recording
    Selection
  37. Demo
  38. Experiments, tests and results (1/2)
    Testing the BCI
    BCI sessions without feedback used for offline analyses and offline classification
    BCI sessions with feedback and online classification
    Testing the speller: Speller interface simulations using offline and online classification performance
    Testing the whole system
    BCI classification accuracies
  39. Experiments, tests and results (2/2)
    Task: “what a wonderful day”
    Impact of classification accuracy on performance
    Impact of language predictions on performance
    Online performances with the spelling application
    Performance comparison with other systems
  40. Experiments, tests and results (2/2)
    Task: “what a wonderful day”
    Impact of classification accuracy on performance
    Impact of language predictions on performance
    Online performances with the spelling application
    Performance comparison with other systems
  41. Experiments, tests and results (2/2)
    Task: “what a wonderful day”
    Impact of classification accuracy on performance
    Impact of language predictions on performance
    Online performances with the spelling application
    Performance comparison with other systems
  42. Experiments, tests and results (2/2)
    Task: “what a wonderful day”
    Impact of classification accuracy on performance
    Impact of language predictions on performance
    Online performances with the spelling application
    Performance comparison with other systems
  43. Conclusions & future work
    • BCI is noisy channel with low rates of information transfer and prone to errors
    • BCI spelling applications should minimize the number of selections required to the user and handle errors properly
    • Language predictions are useful to speedup text composition and enhance the overall communication rate
    • Increase classification accuracy with features in the spatial domain
    • Investigate alternative symbol arrangements to speedup the selection process
    • Test the system with impaired subjects
  44. Thanks for
    the attention

+ Tiziano D'AlbisTiziano D'Albis, 4 months ago

custom

587 views, 0 favs, 0 embeds more stats

A predictive speller for a brain-computer interface more

More info about this document

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Go to text version

  • Total Views 587
    • 587 on SlideShare
    • 0 from embeds
  • Comments 4
  • Favorites 0
  • Downloads 29
Most viewed embeds

more

All embeds

less

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

Cancel
File a copyright complaint
Having problems? Go to our helpdesk?

Categories

Tags

more
-->
Search
RSS Feed
What's new?

© 2009 SlideShare Inc. All Rights Reserved