Multimodal Assessment of Parkinson’s Disease: A Deep Learning Approach

Multimodal Assessment of Parkinson’s Disease:
A Deep Learning Approach
Camilo Vásquez
Pattern Recognition Lab, Friedrich-Alexander University of Erlangen-Nürnberg
July 18, 2018

Progress for summer PRS-2018
• Lab member since 01.05.2018
• First PRS
• Publications (subset)
• Journal papers
• J. C. Vasquez-Correa, J. R. Orozco-Arroyave, T. Bocklet, E. Nöth. «Towards an Automatic Eval-
uation of the Dysarthria Level of Patients with Parkinson’s Disease». Journal of Communication
Disorders (to appear), 2018.
• J. C. Vasquez-Correa, T. Arias-Vergara, J. R. Orozco-Arroyave, B. Eskoﬁer, J. Klucken, and E.
Nöth. « Multimodal assessment of Parkinson’s disease: a deep learning approach » IEEE Journal
of Biomedical and Health Informatics (to appear), 2018.
• T. Arias-Vergara, J. C. Vasquez-Correa, J. R. Orozco-Arroyave, and E. Nöth. «Speaker models
for monitoring Parkinson’s disease progression considering different communication channels and
acoustic conditions» Speech Communication, Vol 101, pp 11-25, 2018.
• Conference proceedings
• J. C. Vasquez-Correa, T. Arias-Vergara, J. R. Orozco-Arroyave, and E. Nöth. «A Multitask Learning
Approach to Assess the Dysarthria Severity in Patients with Parkinson’s Disease» Proceedings of
INTERSPEECH, 2018.
• N. Garcia-Ospina, J. C. Vásquez-Correa, J. R. Orozco-Arroyave, and E. Nöth. «Multimodal i-
vectors to Detect and Evaluate Parkinson’s Disease» Proceedings of INTERSPEECH, 2018.
• T. Arias-Vergara, J. C. Vásquez-Correa, J. R. Orozco-Arroyave, P. Klumpp, and E. Nöth.«Unontrusive
Monitoring of Speech Impairments of Parkinson’s Disease Patients Through Mobile Devices» Pro-
ceedings of ICASSP, 2018.
Camilo Vásquez | Multimodal Assessment of Parkinson’s disease July 18, 2018 0

Introduction: Parkinson’s Disease
• Second most prevalent neuro-
logical disorder worldwide.
• Patients develop several mo-
tor and non-motor impairments.
(Hornykiewicz 1998).

Introduction: Parkinson’s Disease
• Second most prevalent neuro-
logical disorder worldwide.
• Patients develop several mo-
tor and non-motor impairments.
(Hornykiewicz 1998).
Motor impairments
• Bradykinesia
• Rigidity
• Resting tremor
• Micrographia
• Dysartrhia
Evaluated by neurologist experts
according to the MDS-UPDRS-III
scale (Goetz et al. 2008).

Introduction: Justification
• If the motor evaluation is performed with bio–signals such as speech, hand-
writing and gait, the treatment could be followed in a more objective way
Hypothesis
PD patients have difficulties to begin and to stop the movement of limbs, and such
difficulties can be observed on signals by modeling the transitions between stop
and start movements1
1
J. C. Vásquez-Correa et al. (2018). “Multimodal assessment of Parkinson’s disease: a deep learning approach”. In: Journal of Biomedical and Health
Informatics (to appear).

Methods: Transitions detection – Speech
Onset transition Offset transition
• Onset and offset are detected according to the presence of the fundamental
frequency of the speech.
• The border is detected, and 80 ms are taken to the left and to the right, forming
chunks of 160 ms.

Methods: Transitions detection – Speech
25 50 75 100 125 150
Time (ms)
0
500
1000
1500
2000
2500
3000
3500
4000
Frequency(Hz)
25 50 75 100 125 150
Time (ms)
6
4
2
0
2
4
Energy(dB)
STFT for the speech onset for a PD patient (left) and a HC speaker (right) when
they pronounce the syllable /ka/2
.
2
Vásquez-Correa, J. C., Orozco-Arroyave, J. R. and Nöth, E. (2017). “Convolutional Neural Network to Model Articulation Impairments in Patients with
Parkinson’s Disease”. In: 18th International Conference of the Speech and Communication Association (INTERSPEECH).

Methods: Transitions detection – Handwriting
Tablet captures six signals: horizontal position, vertical position, pressure of the
pen, in air movement, azimuth angle, altitude angle3
.
3

Methods: Transitions detection – Handwriting
A. B. C. D.
0.0 0.5 1.0
Time (s)
0
500
1000
1500
Amplitude
0.0 0.5 1.0
Time (s)
0.0 0.5 1.0
Time (s)
0.0 0.5 1.0
Time (s)
x y z azimuth altitude pressure
Handwriting onset produced by: A) HC subject; B) PD patient in low state; C) PD
patient in intermediate state; and D) PD patient in severe state.4
4

Methods: Transitions detection – Gait
• Gait signals were captured with the eGaIT system5
• Three-axis accelerometer and gyroscope attached to the lateral heel of the
shoe.
5
Embedded Gait analysis using Intelligent Technology, http://www.egait.de/

Methods: Transitions detection – Gait
• Onset and offset are segmented based on the presence of the fundamental
frequency and an energy threshold.
• The border is detected, and 3 s are taken to the left and to the right.
2
1
0
1
2
3
acceleration(g)
0 1 2 3 4 5 6
Time (s)
10
20
30
40
50
Frequency(Hz)
0 1 2 3 4 5 6
Time (s)
12
10
8
6
4
2
Energy(dB)
Gait onset for a PD patient (left) and a HC subject (right)

Methods: Modeling Transitions with CNNs
Fully connected
layerConvolutional layer 1 Conv. layer 2Pooling layer 1 Pooling layer 2
Input
(c channels)
ො𝑦
d hidden
units 1
d hidden
units 2
ො𝑦
Raw Input
(c channels)
Conv.1
d1 feature
maps
ReLU
Pooling
Dropout Conv.2
d2 feature
maps
ReLU
Pooling
Dropout
Concatenate
Table: Number of inputs of the CNNs for speech, gait, and handwriting signals. c: Number of channels.
Input signal Convolution Input size c Num. inputs
Speech 2D 40 ×65 1 2600
Gait 2D 60 ×65 12 46800
Handwriting 1D 180 ×1 16 2880

Methods: Fusion of modalities
CNN
speech
CNN
handwriting
CNN gait
Feature maps from
last hidden layer
Avg. per
subject
TransitionTransitionTransition
Concatenate
rbf-SVM
6
6

Methods: Validation
Hyper-parameter Values
Filter size convolutional layers {3,5,7}
Depth of convolutional layers {4,8,16,32,64}
Hidden units in fully connected layers {16,32,64,128}
Learning rate {0.0001,0.0005,0.001}
Probability of dropout {0.1···0.9}
• A Bayesian optimization (BO) approach.
• BO use previous observations of a loss function f, to determine the next
(optimal) point to sample f.
• The loss function f is described as a Gaussian Process.
• Five-fold cross-validation.

Data
• Speech, handwriting and gait from:
• 39 HC subjects.
• 45 PD patients.
• Balanced in age and gender.
• Several exercises are performed by the participants in each modality.
• Speech: rapid repetition of syllables: /pa-ta-ka/, /pa-ka-ta/
• Handwriting: name, signature, sentence, ﬁxed drawings
• Gait: 40 meters walk in straight line with stops each 10 meters
• PD patients were labeled according to the MDS-UPDRS scale.
0 20 40 60 80 100
MDS-UPDRS-III score
0
2
4
6
0 10 20
MDS-UPDRS-III s
0
2
4
20Camilo Vásquez | Multimodal Assessment of Parkinson’s disease July 18, 2018 12

Classiﬁcation of PD patients and HC subjects7
Modality ACC
Speech 92.3±12.3
Handwriting 66.5± 5.5
Gait 80.3±10.3
Fusion 97.6± 2.9
• State-of-art results for the fusion
of modalities.
• High accuracies are obtained
from speech and gait.
• External factors may affect re-
sults for handwrting.
0.0 0.2 0.4 0.6 0.8 1.0
False positive rate
0.0
0.2
0.4
0.6
0.8
1.0
Truepositiverate
Speech AUC=0.925
Gait AUC=0.869
Handwriting AUC=0.699
Fusion AUC=0.954
7

Classification of PD patients in several stages of the disease8
HC PD1 PD2 PD3
HC 95.4% 2.3% 2.3% 0.0%
PD1 28.6% 21.4% 50.0% 0.0%
PD2 15.3% 3.9% 76.9% 3.9%
PD3 0.0% 0.0% 81.8% 18.2%
• Confusion Matrix classifying HC subjects and three groups of PD patients.
• HC are high accurate classified.
• PD patients in intermediate and severe state of the disease are commonly
miss-classified.
8

Conclusion
• A multimodal analysis of PD is proposed considering CNNs such that integrate
information from speech, handwriting and gait signals.

Conclusion
• The proposed method aims to model the difﬁculty of patients to start/stop the
movement of muscles in lower and upper limbs, and in speech.

Conclusion
• The proposed approach is highly accurate to classify PD patients and HC
subjects.

Conclusion
• The proposed approach is highly accurate to classify PD patients and HC
subjects.
• Classiﬁcation of PD patients in several stages of the disease is promising with
the proposed fusion strategy.

Ongoing work
• Prediction of the speech deficits of the patients according to a specific scale
to assess dysarthria9
.
• A multi-task learning approach to predict several speech deficits of the pa-
tients, e.g., lips, palate, larynx, and tongue movements10
.
• Unobtrusive monitoring of the speech impairments of the patients trough mo-
bile devices11
.
• Tracking the disease progression of the patients in a longitudinal study.
• Time-delay neural networks to track the temporal dynamics of the transitions
for speech, handwriting, and gait.
9
Vásquez-Correa, J. C., Orozco-Arroyave, J. R., Bocklet, T. et. al. (2018). “Towards an Automatic Evaluation of the Dysarthria Level of Patients with Parkinson’s
Disease”. In: Journal of Communication Disorders (to appear).
10
Vásquez-Correa, J. C., et. al. (2018). “A Multitask Learning Approach to Assess the Dysarthria Severity in Patients with Parkinson’s Disease”. In: 19th
International Conference of the Speech and Communication Association (INTERSPEECH).
11
Arias-Vergara, T., Vásquez-Correa, J. C., et. al. (2018). “Unontrusive Monitoring of Speech Impairments of Parkinson’s Disease Patients Through Mobile
Devices”. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

Thanks for listening.
Any questions?
juan.vasquez@fau.de
www5.cs.fau.de/en/our-team/vasquez-camilo
Training Network on Automatic Processing of PAthological Speech (TAPAS)
Horizon 2020 Marie Sklodowska-Curie Actions Initial Training Network European Training Network
(MSCA-ITN-ETN) project.

References I
Hornykiewicz, O. (1998). “Biochemical aspects of Parkinson’s disease”. In:
Neurology 51.2 Suppl 2, S2–S9.
Goetz, C. G. et al. (2008). “Movement Disorder Society-sponsored revision of the
Uniﬁed Parkinson’s Disease Rating Scale (MDS-UPDRS): Scale presentation
and clinimetric testing results”. In: Movement disorders 23.15, pp. 2129–2170.
Vásquez-Correa, J. C. et al. (2018). “Multimodal assessment of Parkinson’s
disease: a deep learning approach”. In: Journal of Biomedical and Health
Vásquez-Correa, J. C., Orozco-Arroyave, J. R., Bocklet, T. et. al. (2018). “Towards
an Automatic Evaluation of the Dysarthria Level of Patients with Parkinson’s
Disease”. In: Journal of Communication Disorders (to appear).
Vásquez-Correa, J. C., Orozco-Arroyave, J. R. and Nöth, E. (2017). “Convolutional
Neural Network to Model Articulation Impairments in Patients with Parkinson’s
Disease”. In: 18th International Conference of the Speech and Communication
Association (INTERSPEECH).

References II
Vásquez-Correa, et. al. (2018). “Phonological posteriors and GRU recurrent units
to assess speech impairments of patients with ParkinsonâC™s disease”. In:
21st International Conference on Text, Speech and Dialogue (TSD).
Vásquez-Correa, J. C., et. al. (2018). “A Multitask Learning Approach to Assess the
Dysarthria Severity in Patients with Parkinson’s Disease”. In: 19th International
Conference of the Speech and Communication Association (INTERSPEECH).
Arias-Vergara, T., Vásquez-Correa, J. C., et. al. (2018). “Unontrusive Monitoring of
Speech Impairments of Parkinson’s Disease Patients Through Mobile Devices”.
In: IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP).

Full results
Table: Multimodal classiﬁcation of PD patients and HC subjects. Acc. Test: accuracy in the test set, Acc. Dev.:
accuracy in the development set, AUC: area under the ROC curve, N.: Number of parameters in the CNN.
Bio-signal Acc. Test Acc. Dev. AUC N.
Speech baseline 74.5±1.7 77.0±2.4 0.841
Speech CNN 92.3±12.3 99.4±0.7 0.925 140055
Gait baseline 63.0±8.9 66.0±3.1 0.725
Gait CNN 80.3±10.3 83.3±8.9 0.878 326977
Handwriting baseline 67.1±4.2 67.7±1.7 0.725
Handwriting CNN 66.5±5.5 98.1±1.7 0.699 255560
Fusion baseline 89.0±7.8 87.8±3.1 0.944
Fusion CNN 97.6±2.9 98.8±0.6 0.988 609549

Appendix 212
Table: Parkinson’s disease classiﬁcation using gated-recurrent units (GRU) neural networks and phonological
features.
Features Classiﬁcation Num. ACC. AUC
Task Classes
Phonological PD vs HC 2 76.0±5.8 0.78
MFCC PD vs HC 2 65.0±4.7 0.66
Phonological+MFCC PD vs HC 2 64.0±5.6 0.69
12
Vásquez-Correa, et. al. (2018). “Phonological posteriors and GRU recurrent units to assess speech impairments of patients with ParkinsonâC™s disease”. In:
21st International Conference on Text, Speech and Dialogue (TSD).

Multimodal Assessment of Parkinson’s Disease: A Deep Learning Approach

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Multimodal Assessment of Parkinson’s Disease: A Deep Learning Approach

Similar to Multimodal Assessment of Parkinson’s Disease: A Deep Learning Approach (20)

More from Juan Camilo Vasquez

More from Juan Camilo Vasquez (6)

Recently uploaded

Recently uploaded (20)

Multimodal Assessment of Parkinson’s Disease: A Deep Learning Approach