CBMS 2017 presentation on multi-Label modality classification for figures in biomedical literature. Presenting three different multi-label approaches to classify biomedical figures from PubMed Central and MEDIEVAL, a web application where you can easily search and filter by modality PMC figures.

1. Multi-Label Modality
Classification for
Figures in Biomedical
Literature
Athanasios Lagopoulos, PhD Student
School of Informatics, Aristotle University of Thessaloniki
lathanag@csd.auth.gr
Anestis Fachantidis (afa@csd.auth.gr), Grigorios Tsoumakas (greg@csd.auth.gr)
30th International Symposium
on Computer-Based Medical
Systems - IEEE CBMS 2017

2. The Case
PubMed Central (PMC):
• >4 million figures available
• Great source of information for biomedical
research, education and clinical decision.
Lack of associated meta-data
Inaccessible information

3. 30 different modalities/categories
as proposed by ImageCLEF

4. Simple Vs Compound
40% are compound figures (multi-panel format)
Simple (60%) Compound (40%)

5. The Standard Approach
Compound
Figure
Detection
Multi-class
Model
Simple Figure
Compound Subfigures
Figure Separation
Algorithm
Figure Separation is not perfect (~85%)
Figure Isolation - Information Loss

6. Our Approach: Multi-Label Classification
No use of figure separation algorithm
Three different multi-label learning approaches:
• Simple
• Standard
• Extended

7. Simple multi-label approach
Multi-label
model
Compound Simple
Training
Prediction

8. Standard multi-label approach
Compound
Figure
Detection
Simple
Compound
Multi-class
Model
Multi-label
Model
Compound
Figure
Detection
Multi-class
Model
Multi-label
Model
Compound Simple
PredictionTraining

9. Extended multi-label approach
Compound
Figure
Detection
Simple
Compound
Multi-class
Model
Multi-label
Model
Compound
Figure
Detection
Multi-class
Model
Multi-label
Model
Compound Simple
PredictionTraining

10. Model Training
Feature Extraction from JPEG
• BVLC model - Caffe1
• Deep learning (1.2 million images)
• 4096 visual features/figure
Linear Support Vector Machines (SVMs)
• scikit-learn2
• One-vs-Rest transformation (multiple binaries)
1
http://caffe.berkeleyvision.org/
2
http://scikit-learn.org/

11. ImageCLEF 2016 dataset
20.985 Figures
1.568 Compound
No simple figures with categories
Extracted subfigures as simple
Split 40% - 60% (compound –
subfigures) in order to resemble
the distribution of PMC

12. Results
Approach F1-Macro F1-Micro F1-Samples
Standard 0.3569 0.7786 0.7912
Simple multi-label 0.3139 0.7581 0.7215
Standard multi-label 0.3270 0.7667 0.7726
Extended multi-label 0.3309 0.7666 0.7728
Perfect (100%)
figure separation
Compound Figure Detection: 88,83% (Balanced Accuracy)
Multi-class model: 79,54% (F1-micro)

13. The System
Web app
Weekly updates from PMC
Extended multi-label approach
Easy search & filtering by modality
Build with Apache Solr & AngularJS
Available @
atypon.csd.auth.gr/medieval/

14. Conclusion
No information loss
Model redundancy
Promising results
Web application

15. Future Work
Backend:
• Textual features + Visual features
Frontend:
• User feedback, crowdsourcing
• Active learning

16. THANK YOU!
intelligence.csd.auth.gr
Questions?
atypon.csd.auth.gr/medieval
Partially funded by
A. Lagopoulos, A. Fachantidis, G. Tsoumakas
Multi-Label Modality Classification for
Figures in Biomedical Literature
{lathanag,afa,greg}@csd.auth.gr
Atypon Systems Inc.