AACL 2020 Renxuan Albert Li, Ihab Hajjar, Felicia Goldstein, Jinho D. Choi https://www.aclweb.org/anthology/2020.aacl-main.38/

1. Analysis of Hierarchical Multi-Content Text
Classi
fi
cation Model on B-SHARP Dataset
for Early Detection of Alzheimer’s Disease
Asia-Paci
fi
c Chapter of the Association for Computational Linguistic
s
Presented by Jinho D. Cho
i
October 28, 2020
♠
Renxuan A. Li, ♦
Ihab Hajjar, ♦
Felicia Goldstein, ♠
Jinho D. Choi
♠
Department of Computer Science, ♦
Department of Neurolog
y
Emory University, Atlanta GA, USA
jinho.choi@emory.edu

2. Detection of Alzheimer’s Disease
2
Cerebrospinal Fluid Analysis (CFA)
Positron Emission Tomography (PET)

3. Mild Cognition Impairment (MCI)
3
Mild Cognitiv
e
Impairment
Mil
d
Dementia
Moderat
e
Dementia
Sever
e
Dementia
Impairment does not Interfere
with activities or daily living
First work to detec
t
MCI using NLP
Impairment starts Interfering
with activities or daily living

4. B-SHARP Dataset
4
Brain, Stress, Hypertension, and Aging Research Program
Collect 1-2 minute recordings for 3 tasks from MCI patients and Control subjects.
Task 1
:
Daily Activity
Task 2
:
Room Environment
Task 3
:
Picture Description

5. B-SHARP Dataset
5
1st-visit 2nd-visit 3rd-visit Recordings MoCA BNT
Control 185 100 50 385 26.2 (±2.6) 14.2 (±1.2)
MCI 141 68 28 265 21.5 (±3.5) 13.4 (±1.5)
Total 326 168 78 650 24.2 (±3.8) 13.9 (±1.4)
Subject make multiple visits to take more recordings.
The term between the previous and each visit is 1 year.
Tokens Sentences Nouns Verbs Conjuncts Complex Discourse
Q1
Control 186.6 (±60.4) 10.4 (±4.5) 28.1 (±9.6) 30.4 (±11.5) 8.5 (±4.5) 2.3 (±1.7) 8.1 (±5.4)
MCI 175.6 (±54.5) 9.8 (±4.1) 23.7 (±8.3) 29.3 (±10.4) 8.5 (±4.2) 2.0 (±1.6) 9.2 (±6.0)
Q2
Control 191.5 (±11.8) 11.7 (±4.7) 41.1 (±13.3) 24.3 (±11.2) 6.6 (±4.5) 3.6 (±2.7) 7.1 (±4.8)
MCI 178.6 (±11.7) 11.6 (±4.7) 36.7 (±12.1) 23.2 (±10.6) 6.4 (±4.4) 2.9 (±2.3) 8.4 (±5.3)
Q3
Control 193.4 (±63.4) 12.6 (±5.4) 39.5 (±13.5) 28.4 (±10.1) 8.0 (±4.8) 3.3 (±2.1) 6.1 (±5.5)
MCI 187.8 (±63.4) 12.7 (±5.1) 36.2 (±13.2) 27.7 (±10.9) 7.2 (±4.2) 2.6 (±2.0) 7.3 (±5.5)
All
Control 578.1 (±149.8) 34.5 (±10.7) 110.5 (±27.9) 84.2 (±25.4) 23.5 (±10.1) 9.3 (±4.5) 21.4 (±13.0)
MCI 548.7 (±140.6) 34.0 (±10.5) 98.1 (±26.1) 81.2 (±24.1) 22.5 (±9.7) 7.7 (±4.2) 25.3 (±15.0)
p 0.0110 0.5541 < 0.0001 0.1277 0.2046 < 0.0001 0.0006
Table 1: Average counts and their standard deviations of linguistic features per transcript in the B-SHARP dataset.

6. Hierarchical Multi-Content Classi
fi
cation
6
w11 w12 ⋯ w1n
[CLS1] w21 w22 ⋯ w2n
[CLS2] w31 w32 ⋯ w3n
[CLS3]
c1 e11 e12 ⋯ e1n c2 e21 e22 ⋯ e2n c3 e31 e32 ⋯ e3n
c1 c2 c3
MLP1 MLP2 MLP3
o2
o1
o3
⊕ ⊕
MLPe oe
Transformer1 (T1) Transformer2 (T2) Transformer3 (T3)
Task 1 Task 2 Task 3

7. Experiments
7
5-folds Cross Validation
Transformer Encoder
s
BERT (Devlin et al., 2019
)
RoBERTa (Liu et al., 2020
)
ALBERT (Lan et al., 2019)
CV0 CV1 CV2 CV3 CV4 ALL
Control 77 77 77 77 77 385
MCI 53 53 53 53 53 265
Control 37 37 37 37 37 185
MCI 27 28 28 29 29 141
Recordings
Subjects
Subjects in each set are mutually exclusive to the other sets.

8. Evaluation
8
BERT RoBERTa ALBERT
Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3
ACC 67.6 (±0.4) 69.0 (±1.2) 67.7 (±0.7) 69.0 (±1.5) 69.9 (±0.2) 65.2 (±0.3) 67.6 (±1.5) 69.5 (±0.3) 66.6 (±1.3)
SEN 48.9 (±1.8) 57.1 (±2.5) 41.5 (±3.6) 44.3 (±4.5) 55.3 (±1.2) 37.1 (±3.7) 45.9 (±1.9) 52.2 (±0.6) 37.4 (±3.3)
SPE 80.4 (±1.2) 77.3 (±2.8) 85.2 (±3.0) 85.8 (±2.1) 79.7 (±0.7) 84.5 (±3.0) 82.6 (±3.7) 81.4 (±0.3) 86.8 (±3.3)
Table 3: Model performance on the individual tasks. ACC: accuracy, SEN: sensitivity, SPE: specificity.
CNN BERTe RoBERTae ALBERTe Be + Re Ae + Re Be + Ae + Re
ACC 69.5 (±0.2) 69.9 (±1.1) 71.6 (±1.5) 69.7 (±2.9) 72.2 (±0.7) 71.5 (±1.9) 74.1 (±0.3)
SEN 49.2 (±0.8) 57.6 (±3.4) 48.5 (±6.1) 46.2 (±8.3) 56.5 (±2.5) 51.7 (±1.3) 60.9 (±5.2)
SPE 83.5 (±0.9) 77.4 (±4.8) 87.5 (±1.8) 85.4 (±0.5) 83.1 (±0.9) 86.7 (±3.4) 84.0 (±2.4)
Table 4: Performance of ensemble models. Berte/RoBERTae/ALBERTe use transcript embeddings from all 3 tasks
trained by the BERT/RoBERTa/ALBERT models in Table 3, respectively. Be+Re uses transcript embeddings from
both Berte and RoBERTae (so the total of 6 embeddings), Ae+Re uses transcript embeddings from both ALBERTe
and RoBERTae (6 embeddings), and Be+Ae+Re uses transcript embeddings from all three models (9 embeddings).
BERT RoBERTa ALBERT
Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3
ACC 67.6 (±0.4) 69.0 (±1.2) 67.7 (±0.7) 69.0 (±1.5) 69.9 (±0.2) 65.2 (±0.3) 67.6 (±1.5) 69.5 (±0.3) 66.6 (±1.3)
SEN 48.9 (±1.8) 57.1 (±2.5) 41.5 (±3.6) 44.3 (±4.5) 55.3 (±1.2) 37.1 (±3.7) 45.9 (±1.9) 52.2 (±0.6) 37.4 (±3.3)
SPE 80.4 (±1.2) 77.3 (±2.8) 85.2 (±3.0) 85.8 (±2.1) 79.7 (±0.7) 84.5 (±3.0) 82.6 (±3.7) 81.4 (±0.3) 86.8 (±3.3)
Table 3: Model performance on the individual tasks. ACC: accuracy, SEN: sensitivity, SPE: specificity.
CNN BERTe RoBERTae ALBERTe Be + Re Ae + Re Be + Ae + Re
ACC 69.5 (±0.2) 69.9 (±1.1) 71.6 (±1.5) 69.7 (±2.9) 72.2 (±0.7) 71.5 (±1.9) 74.1 (±0.3)
SEN 49.2 (±0.8) 57.6 (±3.4) 48.5 (±6.1) 46.2 (±8.3) 56.5 (±2.5) 51.7 (±1.3) 60.9 (±5.2)
SPE 83.5 (±0.9) 77.4 (±4.8) 87.5 (±1.8) 85.4 (±0.5) 83.1 (±0.9) 86.7 (±3.4) 84.0 (±2.4)
Table 4: Performance of ensemble models. Berte/RoBERTae/ALBERTe use transcript embeddings from all 3 tasks
trained by the BERT/RoBERTa/ALBERT models in Table 3, respectively. Be+Re uses transcript embeddings from
both Berte and RoBERTae (so the total of 6 embeddings), Ae+Re uses transcript embeddings from both ALBERTe
and RoBERTae (6 embeddings), and Be+Ae+Re uses transcript embeddings from all three models (9 embeddings).
Performance on the Individual Tasks
Performance of the Ensemble Models

9. Conclusion
9
Introduced the new dataset, B-SHARP
,
for the detection of Mild Cognitive Impairment (MCI)
Presented Hierarchical Multi-Content Classi
fi
cation Mode
l
to jointly learn multiple documents from different tasks
Achieved the state-of-the-art results with an ensemble mode
l
using three types of transformer encoders
Please visit our lab webpag
e
http://nlp.cs.emory.edu