There is a pressing need for advanced semantic annotation technologies of medical content, in particular medical publications, clinical trials and clinical records. Search engines and information retrieval systems require semantic annotation and indexing systems to support more advanced user search queries. Considering the relevance of disease concepts for clinical coding, automated processing of clinical trials and even patents, it is critical to provide access to high quality manually annotated documents labelled by clinicians for the development and benchmarking of disease mention recognition and grounding tools. This is particularly important for medical content beyond English, where even fewer annotated corpora have been released. To address these issues, we have organized the DisTEMIST (DISease TExt MIning Shared Task) track at BioASQ 2022. It represents the first community effort to evaluate and promote the development of resources for automatic detection and normalization of disease mentions from clinical case documents in Spanish. For this track we have released the DisTEMIST corpus, a collection of 1000 clinical case documents carefully selected by clinicians and annotated manually by a team of healthcare professionals following annotation guidelines and quality control analysis for consistency. Disease mentions were exhaustively mapped by these experts to their corresponding SNOMED CT concept identifiers. Moreover, we have created additional multilingual Silver Standard versions of the corpus for 7 languages (English, Portuguese, French, Italian, Romanian, Catalan and Galician), as well as mention normalization cross-mappings to 4 additional highly used terminologies. We received 38 systems or runs from 9 teams, obtaining very competitive results. Most participants implemented sophisticated AI approaches, mainly deep learning algorithms based on pre- trained transformer-like language models (BERT, BETO, RoBERTa, etc.), with a classifier layer for named entity recognition and embedding distance metrics for entity linking. Finally, some initial explorations of applicability and adaptation of disease taggers trained on the DisTEMIST corpus to different clinical records (discharge summaries, radiology reports and emergency records) were performed. DisTEMIST corpus: https://doi.org/10.5281/zenodo.6408476

1. National Center for Scientific Research
“Demokritos” Athens
• Anastasios Nentidis
• Anastasia Krithara
• Georgios Katsimpras
• Georgios Paliouras
Barcelona Supercomputing Center (BSC):
• Antonio Miranda-Escalada
• Luis Gascó
• Salvador Lima-López
• Eulàlia Farré-Maduell
• Darryl Estrada
• Martin Krallinger
Overview of DisTEMIST at BioASQ:
Automatic detection and
normalization of diseases from
clinical texts: results, methods,
evaluation and multilingual
resources
BioASQ @ CLEF
DisTEMIST corpus: doi.org/10.5281/zenodo.6408476
1

2. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
Importance of disease NER/Text Mining
De − Allice Hunter - File:Hispanophone global world map language.png,CC BY-SA 4.0,
https://commons.wikimedia.org/w/index.php?curid=69323596
Snomed CT
● Relevant for health structured data: clinical coding or indexing with “Diagnosis” codes (ICD-10)
● Relevant search query/IR: 20% of PubMed queries are related to diseases, disorders, and
anomalies (2nd most popular search after authors)
● Relevant to clinical use cases and IE: etiology (e.g. gene–disease relationships), clinical aspects
(diagnosis, prevention and treatment), etc.
● Key concept across multiple data sources/types: literature , clinical records, clinical trials,
patents, or social media (e.g. SMM4H/COLING2022 track - SocialDisNER 1)
(temu.bsc.es/socialdisner/)
Spanish

3. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
Previous disease extraction & normalization efforts
2022
< 2010 2010 2013 2015 2017 2019 2020 2021
● i2b2/VA [Disease
mention corpus]
● IxaMed-GS [Disease mention
corpus in Spanish]
● DrugSemantics corpus [Disease
mention corpus in Spanish]
● CodiEsp [Diagnosis mention corpus
normalized to ICD-10 in Spanish]
● Campillos-Llanos corpus [Disease
corpus partially normalized to
UMLS in Spanish]
● UMLS Metathesaurus creation
[Terminological resource] [1993]
● Snomed-CT creation [Terminological
resource] [1999]
● MetaMap [Tool] [2001]
● NCBI-Disease Corpus [Disease mention corpus
normalized to MeSH and OMIM]
● ShARe/CLEF eHealth [Disease corpus
normalized to SNOMED CT]
● DNorm [Disease finding and normalization
Tool]
● i2b2/VA partial normalization
[Disease mention corpus
normalized to SNOMED CT and
RxNorm]
● DisTEMIST

4. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST/BioASQ track overview

5. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST resources
Webpage: temu.bsc.es/distemist
DisTEMIST corpus: doi.org/10.5281/zenodo.6408476
DisTEMIST annotation guidelines: doi.org/10.5281/zenodo.6458078
DisTEMIST Multilingual Silver Standard: doi.org/10.5281/zenodo.6408476
DisTEMIST entity normalization cross-mappings: doi.org/10.5281/zenodo.6408476
DisTEMIST gazetteer: doi.org/10.5281/zenodo.6458114
DisTEMIST Silver Standard: TBP
DisTEMIST evaluation library:
github.com/TeMU-BSC/distemist_evaluation_library
DisTEMIST participant systems:
temu.bsc.es/distemist/participant-systems/
DisTEMIST YouTube playlist: tinyurl.com/distemist-playlist

6. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST Corpus - data, format & annotation

7. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST Corpus - Overview
● Manual entity mention labels/annotations
● Manual normalization or mapping to SNOMED CT codes
● Inter-Annotator Agreement (IAA): 82.3
● Random training and test split
Most common disease mentions
Manual

8. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST Multilingual Silver Standard
7 languages:
• English
• Italian
• Portuguese
• French
• Romanian
• Catalan
• Galician

9. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST Multilingual Silver Standard
Spanish Gold Standard English Silver Standard
Snomed ID:
416098002
Snomed ID:
416098002
Snomed ID:
13644009
Snomed ID:
13644009
Online side-by side browser (BRAT): temu.bsc.es/mDistemist/diff.xhtml#/translations/en/train/S0004-06142005000500011-1?diff=/gold-standard/train/

10. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST participating teams
● Registrations: 159
● DisTEMIST-entities: 9 participating teams, 19 submissions
● DisTEMIST-linking: 7 teams, 13 submissions

11. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST participant results
● MiF: micro-averaged F-score (main metric)
● MiP: micro-avg. Precision
● MiR: micro-avg. Recall
https://github.com/TeMU-BSC/distemist_evaluation_library

12. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST participant results
● MiF: micro-averaged F-score (main metric)
● MiP: micro-avg. Precision
● MiR: micro-avg. Recall
https://github.com/TeMU-BSC/distemist_evaluation_library

13. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST top-performing teams methods
● Domain and language specific
transformers-based LM
● Simple decoding layer: linear layer
● Two step-process: candidate
generation & candidate ranking
● 2 dictionaries: DisTEMIST
Gazetteer & UMLS
● 2 candidate generation
engines based on different
technologies
● Smart candidate combination
using weights and thresholds
E PICUSLab
L HPI-DHC
Borchert et al. HPI-DHC @ BioASQ DisTEMIST: Spanish
Biomedical Entity Linking with Pre-trained Transformers
and Cross-lingual Candidate Retrieval
Moscato et al. Biomedical Spanish Language
Models for entity recognition and linking at
BioASQ DisTEMIST
DISTEMIST-entities
DISTEMIST-linking
Biomedical Spanish RoBERTa
Tokenized input sentence
Linear Token Classification Layer
O
El paciente … Sarcoma de Kaposi …
O B-ENF I-ENF
… I-ENF …

14. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST Spanish Silver Standard
Definition: DisTEMIST participants made predictions for the test set + 2750 background set documents
(oncology, cardiology, urology, nefrology, etc).
Harmonize background set predictions → DisTEMIST Spanish Silver Standard corpus.
Statistics: 615235 disease predictions (146396 unique)
Usage: the DisTEMIST Spanish Silver Standard is useful to improve the deep learning systems (shown in table
below), as well as to create a disease Knowledge Graph.
System improvement using
Silver Standard data!
Documents Mentions Unique mentions Tokens
2,750 615,235 146,396 1,206,711

15. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
• Increasing participation in the Spanish task of BioASQ
(5 [2020], 7 [2021], 9 [2022],…)
• Increasing interest in Spanish clinical NLP tasks
• DisTEMIST Resources
○ DisTEMIST Corpus: Disease entity Gold Standard corpus normalised to SNOMED-CT.
○ DisTEMIST Multilingual Silver Standard Corpus: Disease entity corpora normalised
to SNOMED-CT in several languages.
○ DisTEMIST Spanish Silver Standard (from participants’ predictions)
• Disease Entity Recognition difficulties:
○ Complex entities (4%)
○ Heterogeneous diseases
○ Mention length: e.g. "tortuosidad de tronco celíaco y arteria hepática" (tortuosity of
the celiac trunk and hepatic artery)
Conclusions

16. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
•Expand the DisTEMIST Multilingual Silver Standard to other languages
•Correct the DisTEMIST Multilingual Silver Standard to generate a Gold Standard subset
of each language to create high-quality benchmarks in the seven languages.
•Prepare the same Gold Standard corpus for other medical entities (symptoms and
medical procedures).
Future directions

17. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
● Pablo Serrano
Acknowledgements
DisTEMIST Participants &
DisTEMIST Scientific Committee
BioASQ organisers
● Anastasios Nentidis
● Georgios Katsimpras
● Anastasia Krithara
● Georgios Paliouras
CLEF organisers
Funding:
• Plan de Tecnologías del Lenguaje
• AI4PROFHEALTH (PID2020-119266RA-I00)
• BioMATDB Horizon Europe Grant
Agreement No 101058779 Hospital 12 de Octubre
BSC Text Mining Unit

18. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
DisTEMIST resources
Webpage: temu.bsc.es/distemist
DisTEMIST corpus: doi.org/10.5281/zenodo.6408476
DisTEMIST annotation guidelines: doi.org/10.5281/zenodo.6458078
DisTEMIST Multilingual Silver Standard: doi.org/10.5281/zenodo.6408476
DisTEMIST cross-mappings: doi.org/10.5281/zenodo.6408476
DisTEMIST gazetteer: doi.org/10.5281/zenodo.6458114
DisTEMIST Silver Standard: TBP
DisTEMIST evaluation library:
github.com/TeMU-BSC/distemist_evaluation_library
DisTEMIST participant systems:
temu.bsc.es/distemist/participant-systems/
DisTEMIST YouTube playlist: tinyurl.com/distemist-playlist

19. BioASQ - DisTEMIST: DISease TExt Mining Shared Task - krallinger.martin@gmail.com; antoniomiresc@gmail.com
Thank you!
Questions?