This document summarizes challenges in developing machine learning models to detect cognitive diseases like Alzheimer's and aphasia from speech. Some key challenges discussed include the lack of appropriate labeled training data, especially for diseases other than Alzheimer's. The document also discusses how additional healthy speech data and techniques like consensus networks can help address the lack of labels. It notes that automatic speech recognition errors can significantly impact performance and that syntactic features are more robust to errors than lexical features. Finally, it discusses the importance of cross-language studies and domain adaptation techniques like optimal transport to overcome limitations of models trained only on English data.

1. Solving Real-life Challenges
in Detecting Cognitive Diseases
from Speech using ML
Jekaterina Novikova, Winterlight Labs
ODSC East (Virtual Ed.)
Boston, MA. April 14, 2020

2. Cognitive Diseases / Disorders
Alzheimer's disease (AD) is an irreversible disease that destroys brain cells, causing reduction in
thinking ability and memory loss. It is not a normal part of aging.
● Struggle with vocabulary, having trouble naming a familiar object or use the wrong name (e.g.
calling a “watch” a “hand-clock”),
● Difficulty in finding words,
● Loss of verbal fluency, etc.
Aphasia is a form of language impairment that affects speech production and/or comprehension.
Aphasia is always due to injury to the brain - most commonly from a stroke, but may also arise from
head trauma, brain tumors, or from infections. Ways aphasia may affect speech and language:
● Omission of short words (“a”, “is” ),
● Speech output is limited mainly to short utterances of less than four words,
● Sentences do not hang together and irrelevant words intrude,
● Difficulty in finding words, particularly significant nouns and verbs, etc.

4. Winterlight’s Product

5. Winterlight’s Product

6. Challenges
Some common challenges researchers are constantly dealing with when developing ML
models:
● Lack of appropriate training data,
● Absence of labels,
● Effects of automatic speech recognition,
● Limitations of English-only models.

7. Lack of Training Data
We need AD data for AD detection (obviously). But!
● Research shows that additional same-task data of
healthy (!) subjects improves ML model performance
by 13% (Noorian et al., 2017)
● We experimented with additional healthy samples
from different task. Increase of up to 9% in F1 scores.
Effect is especially pronounced when data come from
healthy subjects of age > 60.
A.Balagopalan, J.Novikova, F.Rudzicz and M.Ghassemi. The Effect of Heterogeneous Data for Alzheimer's Disease Detection from Speech. In: NeurIPS
Workshop on Machine Learning for Health ML4H, Montreal, 2018

8. Absence of Labels
We introduce Consensus Networks (CN):
Inspired by GANs (Generative Adversarial Networks), with
adversarial training but no generator.
Z. Zhu, J. Novikova, and F. Rudzicz. Detecting cognitive impairments by agreeing on interpretations of linguistic features. In Proceedings of NAACL, 2019
Z. Zhu, J. Novikova, and F. Rudzicz. Semi-supervised classification by reaching consensus among modalities. In: NeurIPS Workshop on Interpretability and
Robustness in Audio, Speech, and Language IRASL, Montreal, 2018
Three step algorithm:
1. ePhysicians try to produce
indistinguishable representations for each
modality
2. Discriminators recognize modal-specific
information retained in representations
3. Classifier trains the networks to make a
correct decision
Model F1-macro
QDA 0.5243
RF 0.6184
GP 0.6775
MLP 0.7528
CN 0.7998*

9. Effect of ASR Errors
Simulated deletion errors have a
strong effect on classification
performance when detecting
cognitive impairment from
speech and language.
J. Novikova, A. Balagopalan, K. Shkaruta and F. Rudzicz. Lexical Features Are More Vulnerable, Syntactic Features Have More Predictive Power.
In: The 5th Workshop on Noisy User-generated Text at EMNLP, Hong Kong, 2019
● Values of lexical features are easily affected by even
slight changes in text. Syntactic features, however, are
more robust to such modifications.
● But lower changes of syntactic features result in
stronger effects on classification performance.

10. Limitations of English-only Models
Why cross-language studies are important:
● Most developments are made in resource-rich languages
(especially English),
● Multi-language clinical speech datasets are small.
To overcome limitations:
● We use Optimal Transport (OT) domain adaptation
systems to adapt French and Mandarin to English,
Balagopalan, A., Novikova, J., McDermott, M., Nestor, B., Naumann, T., & Ghassemi, M. Cross-Language Aphasia Detection using Optimal Transport Domain
Adaptation. arXiv preprint arXiv:1912.04370. In: NeurIPS Workshop on Machine Learning for Health ML4H, Vancouver, 2019
● Utilize out-of-domain, single-speaker, healthy speech data,
● Train aphasia detection models on English data, test on
French and Mandarin,
● Such a model improves aphasia detection over unilingual
baselines and direct feature transfer.
Language Model F1-macro
French Unilingual 0.7400
Direct FT 0.7913
OT 0.8723
Mandarin Unilingual 0.6047
Direct FT 0.2308
OT 0.6625

11. Thank you!