Is multitask learning worthy in PICO recognition? We explored this question in out paper with the same name (Read our paper here https://arodes.hes-so.ch/record/8949?ln=FR). These slides correspond to the paper and were presented in CLEF2021 Romania, Bucharest.

1. Presenter: Anjani K. Dhrangadhariya
Joint work with: G. Aguilar, T. Solorio, R. Hilfiker, H. Müller
CLEF 2021, Bucharest, Romania, 23.09.2021
1
End-to-end Fine-grained Neural
Entity Recognition of Patients,
Interventions, Outcomes

2. Introduction
2
…A semi-structured interview
was used to obtain qualitative
information on the effect of the
aerobics intervention vs.
conventional exercise. The
convenience sample included 15
adult Oncology outpatients, 13
female and 2 male, ranging in
age from 20 to 87. Quality of
life, fatigue, and side-effects
were the measured outcomes…
Participant
Intervention
Comparator
Outcome

3. Introduction
3
…A semi-structured interview
was used to obtain qualitative
information on the effect of the
aerobics intervention vs.
conventional exercise. The
convenience sample included 15
adult Oncology outpatients, 13
female and 2 male, ranging in
age from 20 to 87. Quality of
life, fatigue, and side-effects
were the measured outcomes…
P
I
C
O
aerobics intervention
Quality of life
fatigue
conventional
exercise
15
Adult, age from 20 to 87
Oncology outpatients
13 female and 2 male
side-effects
Sample size
Age
Condition
Gender
Physical
Control
Physical
Physical
Adverse
effects
Coarse-grained
labels
Fine-grained
labels

4. 4
P
I
C
O
aerobics intervention
Quality of life
fatigue
conventional
exercise
15
Adult, age from 20 to 87
Oncology outpatients
13 female and 2 male
side-effects
Sample size
Age
Condition
Gender
Physical
Control
Physical
Physical
Adverse
effects
Systematic Reviews (SR)
Inclusion into the SR Exclude from the SR
• Evidence-based clinical practice
• Health policy-making
Introduction

5. Challenges
• Coarse-grained PICO - Disagreement between human
annotators
• Fine-grained PICO subtypes - are neither clearly identified
nor standardized as semantic units.
• Domain-specific challenges
• Naming conventions are not established in the health vs. pharma
literature
5
Brockmeier, Austin J., et al. "Improving reference prioritisation with PICO recognition." BMC medical informatics and decision making 19.1
(2019): 1-14.

6. Challenges
• Sentence-level PICO extraction
• Only available “large” entity annotation PICO dataset – EBM-PICO
• ~5000 PICO annotated RCT abstract
• Manually annotated – PICO and subtypes
• Training and Test sets
• Annotators: Medical students and physicians
• Only available with multi-level annotation
6
Nye, Benjamin, et al. "A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical
literature." Proceedings of the conference. Association for Computational Linguistics. Meeting. Vol. 2018. NIH Public Access, 2018.

7. State of the art
Nye et al. Hand-engineered features
Poor performance on the fine-grained PICO
Ananiadou et al. Do not focus on the fine-grained recognition
Beltagy et al. SciBERT domain-adaptation improved coarse-grained recognition.
Zhang et al. Disease NER - Participant condition entity only
Xu et al. Focus on patient demographics (gender, sex, ethnicity) alone
Chung et al. Focus on the Intervention arms in the RCTs
7

8. 1. Most studies concentrate on the coarse-grained PICO recognition
but not on the extraction of PICO subtypes and are not end-to-end.
• PICO recognition = Semi-automation
• PICO subtype recognition = full automation
2. No focus on domain differences
3. No exploration of using coarse and fine-grained labels annotation
together – Multi-task learning (MTL)?
8
Motivation

9. Methodology: MTL
Single-task learning Multi-task learning
• Task A, Task B
• Labeled dataset d(xa ,ya), d(xb ,yb)
• Loss function La, Lb
• ModelA, ModelB
• Task A, Task B
• Labeled dataset d(x,ya), d(x,yb)
• A loss function L
• Model
Models trained separately and do not
share the L and the internal parameters
Models trained in unison and share the L
and the internal parameters
9
• Train one neural network to do multiple related tasks where each task hopefully
helps improve the performance on the other task.
https://ruder.io/multi-task/index.html

10. Methodology: MTL
• The key idea is that one task (auxiliary task) can help another task (main task).
• Assumption: Fine-grained entities nested under the coarse-grained entities =
related tasks.
• Mutual sources of inductive bias
10
Coarse-grained Participant labels Fine-grained Participant labels

11. Methodology: Datasets
12
• EBM-PICO training set
• 5000 PICO annotated RCT abstracts
• PICO coarse- and fine-grained
• Training set: 4,993 documents
• Annotated by medical students
EBM-PICO evaluation corpus Hilfiker corpus
• 191 PICO documents (fine and coarse)
• Annotated by physicians
• Pharma domain
• 153 PICO annotated (fine and coarse)
• Annotated by the first author
• Physiotherapy and Rehab domain
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6174533/bin/NIHMS988059-supplement-Appendix.pdf
Documents = RCT abstracts

12. 13
Prepares the input text
tokens by subword
tokenizing them using
BERT tokenizer
Methodology: MTL architecture

13. 14
Methodology: MTL architecture
Used to extract deep
contextualized BERT
vectors from the
tokenized text

14. 15
Methodology: MTL architecture
Average over the last
four BERT layers are
calculated and the
representation is fed to
the BiLSTM layer

15. 16
Methodology: MTL architecture
BiLSTM outputs are
concatenated and self-
attention is applied

16. 17
Methodology: MTL architecture
Emission
probabilities from
the self-attention
layer are fed into
the two separate
CRF heads along
with the true
labels

17. 18
Methodology: MTL architecture
=
Decoder layers
separately learn
the coarse- and
fine- grained
labels on the
data and
calculate
individual losses
1. MTL
2. STL

18. Experiments conducted
19
Experiment Model setup
I BERT Linear
II BERT LSTM CRF
III BERT BiLSTM CRF
IV BERT LSTM Atten CRF
V BERT BiLSTM Atten CRF
V BERT BiLSTM Multihead Atten CRF
VI BERT BiLSTM Multihead Atten CRF (fine-)/Linear(coarse-)

19. Experiments conducted
20
Experiment Model setup
I BERT Linear
II BERT LSTM CRF
III BERT BiLSTM CRF
IV BERT LSTM Atten CRF
V BERT BiLSTM Atten CRF
Ablation 1 BERT BiLSTM attn (on coarse) CRF
Ablation 2 BERT BiLSTM attn (on fine) CRF
V BERT BiLSTM Multihead Atten CRF
VI BERT BiLSTM Multihead Atten CRF (fine-)/Linear(coarse-)
1. MTL
2. STL

20. Results
21
Underline = Significant improvement compared to the counterpart
MTL performs better
(BOLD) for some
experimental setups
but it is not a
significant
improvement
(UNDERLINE)

21. Results: Ablation experiments
22

22. Error analysis (EBM-PICO eval set)
23
• Intervention entity : Poor performance
• “Psychological” and “Other” classes
• Fewer annotations
• Consistently confused between “Educational” and
“Psychological” – Annotation exercise
• Plenty PICO entities get classified as out-of-the-
span label and vice versa.
• Class overlap + Class imbalance

23. Conclusion
1. Propose two end-to-end neural model setups for fine-grained PICO
recognition.
2. Both model setups outperform the previous approaches for the
fine-grained ``Participant'' and ``Outcome'' entity recognition.
3. MTL is a good alternative to STL but not directly.
4. We contribute a PICO annotated corpus for the physiotherapy
domain and explore the domain difference
• Our error analysis warrants rethinking of semantically solid class
definitions for fine-grained PICO entities along with ontology
development for the healthcare domain.
24

24. Training setup
1. Epochs 15
2. Max. seq. length BERT 512
3. BERTs’ Last four layers summed
4. BERT not freezed
5. LSTM/BiLSTM hidden size – 512/1025
6. Optimizer – AdamW, Learning rate 5e-5
7. Gradients clipped to one.
8. Weighted cross-entropy loss using Sklearn suite
9. A single Tesla K80 GPU, PyTorch
25

25. References
• Boudin, Florian, et al. "Combining classifiers for robust PICO element detection." BMC
medical informatics and decision making 10.1 (2010): 1-6.
• Huang, Ke-Chun, et al. "Classification of PICO elements by text features systematically
extracted from PubMed abstracts." 2011 IEEE International Conference on Granular
Computing. IEEE, 2011.
• Jin, Di, and Peter Szolovits. "Pico element detection in medical text via long short-term
memory neural networks." Proceedings of the BioNLP 2018 workshop. 2018.
• Beltagy, Iz, Kyle Lo, and Arman Cohan. "Scibert: A pretrained language model for
scientific text." arXiv preprint arXiv:1903.10676 (2019).
• Zhang, Tengteng, et al. "Unlocking the power of deep pico extraction: Step-wise medical
ner identification." arXiv preprint arXiv:2005.06601 (2020).
• Xu, Rong, et al. "Extracting subject demographic information from abstracts of
randomized clinical trial reports." Medinfo 2007: Proceedings of the 12th World Congress
on Health (Medical) Informatics; Building Sustainable Health Systems. IOS Press, 2007.
26

26. Thank you for your attention
Questions?
Anjani Dhrangadhariya
anjani.dhrangadhariya@hevs.ch
https://www.linkedin.com/in/anjani-dhrangadhariya/
More information
http://medgift.hevs.ch/wordpress/
https://www.examode.eu/
27