The electronic health record (EHR) contains a large amount of multi-dimensional and unstructured clinical data of significant operational and research value. Distinguished from previous studies, our approach embraces a double-annotated dataset and strays away from obscure ``black-box'' models to comprehensive deep learning models. In this paper, we present a novel neural attention mechanism that not only classifies clinically important findings. Specifically, convolutional neural networks (CNN) with attention analysis are used to classify radiology head computed tomography reports based on five categories that radiologists would account for in assessing acute and communicable findings in daily practice. The experiments show that our CNN attention models outperform non-neural models, especially when trained on a larger dataset. Our attention analysis demonstrates the intuition behind the classifier's decision by generating a heatmap that highlights attended terms used by the CNN model; this is valuable when potential downstream medical decisions are to be performed by human experts or the classifier information is to be used in cohort construction such as for epidemiological studies.

1. Classification of Radiology Reports
Using Neural Attention Models
Bonggun Shin, Falgun H. Chokshi, Timothy Lee and Jinho D. Choi
Emory University

2. Electronic Health Record
(EHR)
• EHR?
• Lab values, vital sign - Structured
• Clinical reports, Radiology reports - Unstructured
• If all EHR records are structured
• Evaluating cancer treatment outcomes
• Identifying patient phenotype cohorts
• Predicting clinical outcomes

3. Problem Definition
Acute Blood
Radiology
Report #1
The patient has
hemorrhage.
Radiology
Report #1
The patient has
hemorrhage.
Some Blood
Radiology
Report #3
The patient
some
hemorrhage.
No Blood
Radiology
Report #7
The patient has
no
hemorrhage.
Task 1: Severity
Task 2: Blood
Task 3: Mass
Task 4: Stroke
Task 5: Hydro

4. Previous Methods
• Query based methods
• Rules-based approach
• Example
• Attempt to categorize bleeding patients
• The query word “hemorrhage”
• Expected - The patient has hemorrhage.
• False negative - “no more hemorrhage”
• NLP based methods
• n-grams - low performances
• Neural network models
• No CNN based models
• Promising but black box

5. Neural Attention model
Radiology
Report
Radiology
Report
Radiology
Report
Black-Box
Model
Normal
Symptomatic
Radiology
Report
Radiology
Report
Radiology
Report
Normal
Symptomatic
Explanation
Other Neural Models
Proposed Models

6. Dataset
• 1400 annotated radiology head CT reports
• Split into training, development, and evaluation sets (1000/200/200)
• 80,000 unannotated reports for creating word embedding
• Annotated according to each task

7. Convolutional Neural
Networks Basics (1)
• Tokenization: NLP4J
• Train W2V: Miklov
• Params
• Model: CBOW, or SKIP
• Dimension: 100, 200, or 400
• # of doc: 20k, 40k, 60k, or 80k (Unannotated reports)
Unannotated
Radiology Report
#1
xxxxxxxxxxxx
xxxxxxx
Word2Vec

8. Convolutional Neural
Networks Basics (2)
• Each word corresponds to a respective vector
• The number of tokens is set to the maximum document length
The
patient
has
hemorrhage
.
<pad>
<pad>
<pad>
<pad>
Radiology
Report #1
The patient has
hemorrhage.
Word2Vec

9. Convolutional Neural
Networks Basics (3)
• Filters: 2, 3, 4, 5
• Maxpooling over each feature vector.
• Dense vector is creating the softmax layer
Input Matrix
Feature
Vectors
Dense
Vector Prediction

10. Attention vector
• Filter: 1
• Maxpooling across all feature vectors.
Document
Matrix
Attention Matrix
(Filter Lenth=1)
Attention
Vector
(MaxPool)

11. Embedding Attention Vector
(EAV)
• EAV can be seen as document vector
• EAV is concatenated to the dense vector
Document
Matrix
(Transposed)
Attention
Vector
Embedding
Attention
Vector

12. Explanatory Features
The
patient
has
hemorrhage
.
<pad>
<pad>
<pad>
<pad>
Attention
Vector
Heat map
Input Matrix

13. Attention Heat Map
CT HEAD W/O CONTRAST HEAD CT WITHOUT IV CONTRAST CLINICAL INDICATION : Altered mental status
−1
0
1
TECHNIQUE : Axial CT images skull base vertex IV contrast . COMPARISON : Date ,
−1
0
1
MRI brain Date FINDINGS : Interval blooming demonstrated greater 20 foci intraparenchymal hemorrhage involving 4
−1
0
1
cerebral lobes surrounding edema . For example , intraparenchymal hemorrhage frontal lobe vertex measures 1.6
−1
0
1
1.7 cm , 1.3 1.4 cm ( series 4 image 43 ) , corpus callosum
−1
0
1
hemorrhage measures measures 4.2 2.2 cm , 3.0 1.9 cm ( series 4 image 42
−1
0
1
) , hemorrhage posterior temporal lobe measures 2.3 1.5 cm , 2.1 1.3 cm (
−1
0
1
series 4 image 34 ) . Additionally , worsening mass increasing sulcal effacement mild effacement
−1
0
1
suprasellar quadrigeminal plate cisterns . No interval change low − lying tonsils . Minimal left
−1
0
1
− − midline shift 2 mm. There persistent effacement lateral ventricles , unchanged size .
−1
0
1
No hydrocephalus . The skull base calvarium demonstrate abnormality . Redemonstrated mucus retention cyst maxillary
−1
0
1
sinus . The remaining included paranasal sinuses mastoid air cells clear . IMPRESSION : 1.
−1
0
1
Interval increase size / blooming greater 20 intraparenchymal hemorrhages surrounding edema involving 4 cerebral lobes
−1
0
1
. Please report details . 2. Interval worsening diffuse cerebral edema sulcal effacement , mild
−1
0
1
effacement cisterns , midline shift stable low lying tonsils . 3. No acute large territory
−1
0
1
infarction definite foci hemorrhage . Important findings communicated name name page info Date name name name
−1
0
1
This final report , dictated radiology name name name name , agrees preliminary report dictated
−1
0
1
overnight name . These images reviewed interpreted name name name , name
−1
0
1
CT HEAD W/O CONTRAST HEAD CT WITHOUT IV CONTRAST CLINICAL INDICATION : Altered mental status
−1
0
1
TECHNIQUE : Axial CT images skull base vertex IV contrast . COMPARISON : Date ,
−1
0
1
MRI brain Date FINDINGS : Interval blooming demonstrated greater 20 foci intraparenchymal hemorrhage involving 4
−1
0
1
cerebral lobes surrounding edema . For example , intraparenchymal hemorrhage frontal lobe vertex measures 1.6
−1
0
1
1.7 cm , 1.3 1.4 cm ( series 4 image 43 ) , corpus callosum
−1
0
1
hemorrhage measures measures 4.2 2.2 cm , 3.0 1.9 cm ( series 4 image 42
−1
0
1
) , hemorrhage posterior temporal lobe measures 2.3 1.5 cm , 2.1 1.3 cm (
−1
0
1
series 4 image 34 ) . Additionally , worsening mass increasing sulcal effacement mild effacement
−1
0
1
suprasellar quadrigeminal plate cisterns . No interval change low − lying tonsils . Minimal left
−1
0
1
− − midline shift 2 mm. There persistent effacement lateral ventricles , unchanged size .
−1
0
1
No hydrocephalus . The skull base calvarium demonstrate abnormality . Redemonstrated mucus retention cyst maxillary
−1
0
1
sinus . The remaining included paranasal sinuses mastoid air cells clear . IMPRESSION : 1.
−1
0
1
Interval increase size / blooming greater 20 intraparenchymal hemorrhages surrounding edema involving 4 cerebral lobes
−1
0
1
. Please report details . 2. Interval worsening diffuse cerebral edema sulcal effacement , mild
−1
0
1
effacement cisterns , midline shift stable low lying tonsils . 3. No acute large territory
−1
0
1
infarction definite foci hemorrhage . Important findings communicated name name page info Date name name name
−1
0
1
This final report , dictated radiology name name name name , agrees preliminary report dictated
−1
0
1
overnight name . These images reviewed interpreted name name name , name
−1
0
1
Acute Blood Mass Effect

14. Performance Comparison
on the Test data
• Annotation: Two trained medical doctors
• Accuracy is comparable to the human agreement scores
• Both CNN and NAM outperform the baseline

15. Performance Trends
Number of Documents
for training w2v
Number of Documents
to train the models

16. Discussion
• Comparable to human agreement scores
• Explanatory features
• Big data is good for a complex model
• Annotation is expensive
• data synthesis?
• Transfer learning
• n-gram attention?