Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Multimodal Analysis of Speech, Handwriting
and Gait for the Assessment of Patients with
Parkinson’s Disease
Student: Juan ...
Outline
Introduction
Methods
Speech Analysis
Gait Analysis
Handwriting Analysis
Multimodal Analysis
Results
Multi-view lea...
Introduction: Parkinson’s Disease
Second most prevalent
neurological disorder
worldwide.
Patients develop sev-
eral motor ...
Introduction: Parkinson’s Disease II
Motor impairments
Bradykinesia
Rigidity
Resting tremor
Micrographia
Dysartrhia
Non–Mo...
Introduction: Parkinson’s Disease II
Motor impairments
Bradykinesia
Rigidity
Resting tremor
Micrographia
Dysartrhia
Non–Mo...
Introduction: Research Problem
Motor evaluation is expensive and time–consuming.
Neurologists evaluate perceptually the mo...
Introduction: Research Problem
Motor evaluation is expensive and time–consuming.
Neurologists evaluate perceptually the mo...
Introduction: Research Problem
Motor evaluation is expensive and time–consuming.
Neurologists evaluate perceptually the mo...
Introduction: Justification
if the motor evaluation is performed with bio–signals such
as speech, handwriting and gait, the...
Introduction: Proposal
These three bio-signals constitute a reliable source of
information to describe several symptoms of...
Introduction: Proposal
These three bio-signals constitute a reliable source of
information to describe several symptoms of...
Introduction: Proposal
These three bio-signals constitute a reliable source of
information to describe several symptoms of...
Methods: Speech analysis
Speech impairments can be assessed using four dimensions
(J. R. Orozco-Arroyave 2016)
Phonation
A...
Methods: Speech analysis
https://github.com/jcvasquezc/NeuroSpeech1
1J. R. Orozco-Arroyave, J. C. V´asquez-Correa, et. al....
Methods: Gait analysis
Gait acquisition
15 / 33
Methods: Gait analysis
Heel
strike
Toe
off
Heel
strike
Stance time Swing time
Stride time & Stride length
Toe off / Heel s...
Methods: Gait analysis
Time–frequency analysis
0 5 10 15 20
Time (s)
0
10
20
30
40
50
Frequency(Hz)
Healthy Control
0 5 10...
Methods: Handwriting analysis
Handwriting acquisition
18 / 33
Methods: Handwriting analysis
Speed of the stroke
Acceleration
In–air movement
Pressure of the pen
Azimuth
(P. Drot´ar et ...
Methods: Multimodal analysis
GCCA
(J. C. V´asquez-Correa, et
al. 2017)
Early fusion
Weak learners
Multimodal convolutional...
Methods: Pattern analysis
Classical machine learning
Support vector
machines.
Support vector
regressors.
minimize
w,b,ξ
1
...
Methods: Pattern analysis
Deep learning
Convolutional neural networks.
Recurrent neural networks and LSTM
Variational deep...
Results: Multi-view learning using GCCA2
To obtain a new feature representation when multimodal
data is not available.
To ...
Results: Multi-view learning using GCCA3
Baseline GCCA
Classification PD vs. HC 77% 78%
Neurological state prediction 0.36 ...
Results: CNNs for speech analysis4
Convolution layer I Convolution layer IIMax-pool. layer 1 Max-pool layer 2 Fully conect...
Results: CNNs for speech analysis5
Voiced-Unvoiced transitions are modeled with CNNs and
TFRs.
The STFT and the continuous...
Results: CNNs for speech analysis6
Language Accuracy
Spanish 85.9%
German 75.0%
Czech 89.4%
Table: Classification of PD vs....
Ongoing work
Combination of convolutional and recurrent neural
networks for multimodal analysis of PD.
Convolutional layer...
Ongoing work
Combination of convolutional and recurrent neural
networks for multimodal analysis of PD.
𝒙(𝑡)
𝒉(𝑡−1)
+
𝑼 𝑓 𝑾...
Conclusion
Speech, handwriting and gait signals constitute a reliable
source of information to describe several symptoms o...
References I
J. Klucken et al. “Unbiased and mobile gait analysis detects motor impairment in Parkin-
son’s disease”. In: ...
References II
P. Drot´ar et al. “Evaluation of handwriting kinematics and pressure for differential diag-
nosis of Parkins...
Multimodal Analysis of Speech, Handwriting
and Gait for the Assessment of Patients with
Parkinson’s Disease
Student: Juan ...
Upcoming SlideShare
Loading in …5
×

Multimodal Analysis of Speech, Handwriting and Gait for the Assessment of Patients with Parkinson’s Disease

316 views

Published on

Multimodal Analysis of Speech, Handwriting
and Gait for the Assessment of Patients with
Parkinson’s Disease

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Multimodal Analysis of Speech, Handwriting and Gait for the Assessment of Patients with Parkinson’s Disease

  1. 1. Multimodal Analysis of Speech, Handwriting and Gait for the Assessment of Patients with Parkinson’s Disease Student: Juan Camilo V´asquez Correa Advisors: Prof. Juan Rafael Orozco Arroyave1, Prof. Elmar N¨oth2 1GITA research group, University of Antioquia UdeA. 2Pattern recognition Lab. Friedrich Alexander Universit¨at. Erlangen-N¨urnberg. jcamilo.vasquez@udea.edu.co 1 / 33
  2. 2. Outline Introduction Methods Speech Analysis Gait Analysis Handwriting Analysis Multimodal Analysis Results Multi-view learning using GCCA Convolutional neural networks for speech analysis Ongoing work Conclusion 2 / 33
  3. 3. Introduction: Parkinson’s Disease Second most prevalent neurological disorder worldwide. Patients develop sev- eral motor and non- motor impairments. (O. Hornykiewicz 1998). 3 / 33
  4. 4. Introduction: Parkinson’s Disease II Motor impairments Bradykinesia Rigidity Resting tremor Micrographia Dysartrhia Non–Motor impairments Depression Sleep disorders Cognitive impairments Sensory system deficits Evaluated by neurologist experts according to the MDS-UPDRS-III scale (C. G. Goetz et al. 2008). 4 / 33
  5. 5. Introduction: Parkinson’s Disease II Motor impairments Bradykinesia Rigidity Resting tremor Micrographia Dysartrhia Non–Motor impairments Depression Sleep disorders Cognitive impairments Sensory system deficits Evaluated by psychologist experts. 5 / 33
  6. 6. Introduction: Research Problem Motor evaluation is expensive and time–consuming. Neurologists evaluate perceptually the motor deficits of the patients. The assessment of the motor capabilities provides suitable information to update the treatment and the medication. 6 / 33
  7. 7. Introduction: Research Problem Motor evaluation is expensive and time–consuming. Neurologists evaluate perceptually the motor deficits of the patients. The assessment of the motor capabilities provides suitable information to update the treatment and the medication. 7 / 33
  8. 8. Introduction: Research Problem Motor evaluation is expensive and time–consuming. Neurologists evaluate perceptually the motor deficits of the patients. The assessment of the motor capabilities provides suitable information to update the treatment and the medication. 8 / 33
  9. 9. Introduction: Justification if the motor evaluation is performed with bio–signals such as speech, handwriting and gait, the treatment could be fol- lowed in a more objective way 9 / 33
  10. 10. Introduction: Proposal These three bio-signals constitute a reliable source of information to describe several symptoms of PD patients. The combination of such sources of information allows to perform an accurate quantification of the neurological state of the patients. The multimodal analysis that includes information from different kind of sensors for the analysis of PD has not been enough studied (Q. W. Oung, et al. 2015) 10 / 33
  11. 11. Introduction: Proposal These three bio-signals constitute a reliable source of information to describe several symptoms of PD patients. The combination of such sources of information allows to perform an accurate quantification of the neurological state of the patients. The multimodal analysis that includes information from different kind of sensors for the analysis of PD has not been enough studied (Q. W. Oung, et al. 2015) 11 / 33
  12. 12. Introduction: Proposal These three bio-signals constitute a reliable source of information to describe several symptoms of PD patients. The combination of such sources of information allows to perform an accurate quantification of the neurological state of the patients. The multimodal analysis that includes information from different kind of sensors for the analysis of PD has not been enough studied (Q. W. Oung, et al. 2015) 12 / 33
  13. 13. Methods: Speech analysis Speech impairments can be assessed using four dimensions (J. R. Orozco-Arroyave 2016) Phonation Articulation Prosody Intelligibility pataka pataka 13 / 33
  14. 14. Methods: Speech analysis https://github.com/jcvasquezc/NeuroSpeech1 1J. R. Orozco-Arroyave, J. C. V´asquez-Correa, et. al. “NeuroSpeech: an open-source software for Parkinson’s speech analysis”. In: Digital Signal Precessing and SoftwareX, (Under review) (2017). 14 / 33
  15. 15. Methods: Gait analysis Gait acquisition 15 / 33
  16. 16. Methods: Gait analysis Heel strike Toe off Heel strike Stance time Swing time Stride time & Stride length Toe off / Heel strike angles ( J. Klucken et al. 2013; P. A. P´erez-Toro 2016 ) 16 / 33
  17. 17. Methods: Gait analysis Time–frequency analysis 0 5 10 15 20 Time (s) 0 10 20 30 40 50 Frequency(Hz) Healthy Control 0 5 10 15 20 25 Time (s) Patient Short time Fourier Transform Wavelet transform Modulation spectra Wigner Ville distribution 17 / 33
  18. 18. Methods: Handwriting analysis Handwriting acquisition 18 / 33
  19. 19. Methods: Handwriting analysis Speed of the stroke Acceleration In–air movement Pressure of the pen Azimuth (P. Drot´ar et al. 2016) Static handwriting analysis (Z. Naiquian et. al 2017 ) 19 / 33
  20. 20. Methods: Multimodal analysis GCCA (J. C. V´asquez-Correa, et al. 2017) Early fusion Weak learners Multimodal convolutional neural networks Deep autoencoders. 20 / 33
  21. 21. Methods: Pattern analysis Classical machine learning Support vector machines. Support vector regressors. minimize w,b,ξ 1 2 ||w||2 + C N i=1 ξi subject to yi · (xT i w + b) ≥ 1 − ξi, ξi ≥ 0 21 / 33
  22. 22. Methods: Pattern analysis Deep learning Convolutional neural networks. Recurrent neural networks and LSTM Variational deep autoencoders Convolutional layer 1 Convolutional layer 2Pooling layer Pooling layer 2 RNN-LSTM Input y J hidden filters K hidden filters 22 / 33
  23. 23. Results: Multi-view learning using GCCA2 To obtain a new feature representation when multimodal data is not available. To predict missing information. Machine learning methods are trained with the new feature representation. arg min Uj J j=1 G − XjUj 2 F s.t. GT G = I 2J. C. V´asquez-Correa, et al. “Multi-view representation learning via GCCA for multimodal analysis of Parkinson’s disease”. In: 42nd International Conference on Acoustic, Speech, and Signal Processing (ICASSP). 2017. 23 / 33
  24. 24. Results: Multi-view learning using GCCA3 Baseline GCCA Classification PD vs. HC 77% 78% Neurological state prediction 0.36 0.40 Speech quality prediction 0.67 0.71 Table: Results GCCA The proposed approach is suitable to map the features from other modalities that are not always available. 3J. C. V´asquez-Correa, et al. “Multi-view representation learning via GCCA for multimodal analysis of Parkinson’s disease”. In: 42nd International Conference on Acoustic, Speech, and Signal Processing (ICASSP). 2017. 24 / 33
  25. 25. Results: CNNs for speech analysis4 Convolution layer I Convolution layer IIMax-pool. layer 1 Max-pool layer 2 Fully conected MLP Input layer PD vs. HC Feature maps 1 Feature maps 2 convolutional neural networks (CNN) learns high–level representations from the low–level raw data. CNN is formed with an array of convolutional filters and subsampling layers. HL(i, j, d) = (I ∗ Kd )(i, j) = m n I(i + m, j + n)Kd (m, n) 4J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural Network to Model Articulation Impairments in Patients with Parkinson’s Disease”. In: 18th International Conference of the Speech and Communication Association (INTERSPEECH). 2017. 25 / 33
  26. 26. Results: CNNs for speech analysis5 Voiced-Unvoiced transitions are modeled with CNNs and TFRs. The STFT and the continuous wavelet transform (CWT) are considered. Speech of PD patients in three languages: Spanish, German and Czech. 5J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural Network to Model Articulation Impairments in Patients with Parkinson’s Disease”. In: 18th International Conference of the Speech and Communication Association (INTERSPEECH). 2017. 26 / 33
  27. 27. Results: CNNs for speech analysis6 Language Accuracy Spanish 85.9% German 75.0% Czech 89.4% Table: Classification of PD vs. HC using CNNs 50 100 150 Time (ms) 0 1000 2000 3000 4000 Frequency(Hz) 50 100 150 Time (ms) Low Energy High Energy Figure: Output of the CNN after the last max–pool layer: PD patient (left) and a HC speaker (right) 6J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural Network to Model Articulation Impairments in Patients with Parkinson’s Disease”. In: 18th International Conference of the Speech and Communication Association (INTERSPEECH). 2017. 27 / 33
  28. 28. Ongoing work Combination of convolutional and recurrent neural networks for multimodal analysis of PD. Convolutional layer 1 Convolutional layer 2Pooling layer Pooling layer 2 RNN-LSTM Speech y J hidden filters K hidden filters Gait Handwriting 28 / 33
  29. 29. Ongoing work Combination of convolutional and recurrent neural networks for multimodal analysis of PD. 𝒙(𝑡) 𝒉(𝑡−1) + 𝑼 𝑓 𝑾 𝑓 𝑓: forget gate + 𝑼 𝑾 𝒙(𝑡) 𝒉(𝑡−1) 𝒙(𝑡) 𝒉(𝑡−1) + 𝑼 𝑔 𝑾 𝑔 𝑔: input gate ∗+∗ 𝒔(𝑡) 𝒔(𝑡−1) 𝑠: hidden state 𝑡𝑎𝑛ℎ 𝒉(𝑡) 𝒙(𝑡) 𝒉(𝑡−1) + 𝑼 𝑞 𝑾 𝑞 𝑞: output gate LSTM cell 29 / 33
  30. 30. Conclusion Speech, handwriting and gait signals constitute a reliable source of information to describe several symptoms of PD patients. The combination of such sources of information allows to perform an accurate quantification of the neurological state of the patients. Several features and pattern analysis approaches could be considered to improve the classification of the disease, and the monitoring of the neurological state of the patients. 30 / 33
  31. 31. References I J. Klucken et al. “Unbiased and mobile gait analysis detects motor impairment in Parkin- son’s disease”. In: PloS one 8.2 (2013), e56956. C. G. Goetz et al. “Movement Disorder Society-sponsored revision of the Unified Parkin- son’s Disease Rating Scale (MDS-UPDRS): Scale presentation and clinimetric test- ing results”. In: Movement disorders 23.15 (2008), pp. 2129–2170. J. C. V´asquez-Correa, et al. “Multi-view representation learning via GCCA for multimodal analysis of Parkinson’s disease”. In: 42nd International Conference on Acoustic, Speech, and Signal Processing (ICASSP). 2017. J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural Net- work to Model Articulation Impairments in Patients with Parkinson’s Disease”. In: 18th International Conference of the Speech and Communication Association (IN- TERSPEECH). 2017. J. R. Orozco-Arroyave. Analysis of Speech of People with Parkinson’s Disease. Ger- many: Logos Verlag Berlin, 2016. J. R. Orozco-Arroyave, J. C. V´asquez-Correa, et. al. “NeuroSpeech: an open-source software for Parkinson’s speech analysis”. In: Digital Signal Precessing and Soft- wareX, (Under review) (2017). O. Hornykiewicz. “Biochemical aspects of Parkinson’s disease”. In: Neurology 51.2 Suppl 2 (1998), S2–S9. P. A. P´erez-Toro J. C. V´asquez-Correa, et. al. “An´alisis motriz en las extremidades in- feriores para el monitoreo del estado neurol´ogico de pacientes con enfermedad de Parkinson”. In: XXI Symposium on Image, Signal Processing and Artificial Vision (STSIVA). 2016.
  32. 32. References II P. Drot´ar et al. “Evaluation of handwriting kinematics and pressure for differential diag- nosis of Parkinson’s disease”. In: Artificial intelligence in Medicine 67 (2016), pp. 39– 46. Q. W. Oung, et al. “Technologies for assessment of motor disorders in Parkinson’s dis- ease: a review”. In: Sensors 15.9 (2015), pp. 21710–21745. Z. Naiquian et. al. “Toward Monitoring Parkinson’s through Analysis of Static Handwriting Samples: A Quantitative Analytical Framework”. In: IEEE journal of biomedical and health informatics 21.2 (2017), pp. 488–495.
  33. 33. Multimodal Analysis of Speech, Handwriting and Gait for the Assessment of Patients with Parkinson’s Disease Student: Juan Camilo V´asquez Correa Advisors: Prof. Juan Rafael Orozco Arroyave1, Prof. Elmar N¨oth2 1GITA research group, University of Antioquia UdeA. 2Pattern recognition Lab. Friedrich Alexander Universit¨at. Erlangen-N¨urnberg. jcamilo.vasquez@udea.edu.co 33 / 33

×