Knowledge graphs have been shown to significantly improve search results. Usually populated by subject matter experts, relations therein need to keep up to date with medical literature in order for search to remain relevant. Dynamically identifying text snippets in literature that confirm or deny knowledge graph triples is increasingly becoming the differentiator between trusted and untrusted medical decision support systems. This work describes our approach to mapping triples to medical text. A medical knowledge graph is used as a source of triples that are used to find matching sentences in reference text. Our unsupervised approach uses phrase embeddings and cosine similarity measures, and boosts candidate text snippets when certain key concepts exist. Using this approach, we can accurately map semantic relations within the medical knowledge graph to text snippets with a precision of 61.4% and recall of 86.3%. This method will be used to develop a novel application in the future to retrieve medical relations and corroborating snippets from medical text given a user query.

1. Maulik R. Kamdar, Craig E. Stanley Jr.,
Michael Carroll, Linda Wogulis, Will Dowling,
Helena F. Deus, Mevan Samarasinghe
Elsevier Health and Commercial Markets
Twitter: @maulikkamdar
Text Snippets to Corroborate Medical Relations
An Unsupervised Approach using a Knowledge Graph and Embeddings
VS02: Machine Learning & Predictive Modeling

2. Disclosure
I disclose the following relevant relationship with commercial interests:
• Senior Data Scientist at Elsevier Health and Commercial Markets
2AMIA 2018 | amia.org

3. What types of questions do clinicians ask?
3AMIA 2018 | amia.org
Entity Types:
• Drugs
• Diagnoses
• Diseases
• Phenotypes
• Symptoms
• ...
Del Fiol, G., Workman, T. E., & Gorman, P. N. (2014). Clinical questions raised by clinicians at the point of care:
a systematic review. JAMA internal medicine, 174(5), 710-718.

4. Elsevier’s Healthcare Knowledge Graph
4AMIA 2018 | amia.org
400k concepts
5m relationships
75k diseases
46k drugs
63k procedures
90k symptoms
Elsevier’s Healthcare Knowledge Graph
connects the world’s healthcare concepts and
relationships supported by evidence in content,
and unlocks the knowledge as a scalable,
easily-navigable information service.

5. Elsevier’s Healthcare Knowledge Graph
5AMIA 2018 | amia.org
The Visualizer is a prototype of synoptic content in Elsevier’s Healthcare Knowledge Graph.

6. Elsevier’s Healthcare Knowledge Graph
6AMIA 2018 | amia.org

7. Medical Knowledge Increases …
7AMIA 2018 | amia.org
https://www.clinicalkey.com/#!/browse/books

8. 8AMIA 2018 | amia.org
Candidate Snippets for a
Semantic Relation
• Phrase Embedding Generation
• Boosting using Concept Mapper
and HG Hierarchy
• Evaluation Methodology
Evaluation and Application
• Evaluation over relation—snippet
pairs for 3 diseases
• HG knowledge coverage
• Future Work and Application

9. 9AMIA 2018 | amia.org
Candidate Snippets for a
Semantic Relation
• Phrase Embedding Generation
• Boosting using Concept Mapper
and HG Hierarchy
• Evaluation Methodology
Evaluation and Application
• Evaluation over relation—snippet
pairs for 3 diseases
• HG knowledge coverage
• Future Work and Application

10. Methods
10AMIA 2018 | amia.org
• Ferri’s Clinical Advisor and Conn’s Current
Therapy are widely used (>100K searches)
medical textbooks accessible through the
Clinical Key search engine.
• Ferri has 1037 chapters (799 single disease
chapters) and Conn has 331 chapters (235
single disease chapters).
• The chapters cover diseases, classes or
combinations, differential diagnosis, lab tests,
medical algorithms, etc.
(Sentence) Most commonly used agents are
methotrexate (MTX), hydroxychloroquine
(HCQ), sulfasalazine (SSZ), andleflunomide
(LEF).
(Chapter) Rheumatoid Arthritis
(Section) Treatment
A)Snippet from Ferri’s Clinical
Advisor 2019 Medical Textbook
B) Elsevier’s Healthcare
Knowledge Graph (HG)
Semantic
Relation
https://www.clinicalkey.com/#!/

11. Methods
11AMIA 2018 | amia.org
(Sentence) Most commonly used agents are
methotrexate (MTX), hydroxychloroquine
(HCQ), sulfasalazine (SSZ), andleflunomide
(LEF).
(Chapter) Rheumatoid Arthritis
(Section) Treatment
A)Snippet from Ferri’s Clinical
Advisor 2019 Medical Textbook
C)PhraseEmbeddingGeneration
Sentence Embedding
Section Embedding
Relation Type Embedding
Relation Embedding
B) Elsevier’s Healthcare
Knowledge Graph (HG)
Semantic
Relation
• Embedding vectors generated from
a corpus of biomedical abstracts
from the MEDLINE database using
GloVe software.
• 2.5M words, 30M biomedical
abstracts, 100 dimensions
• Phrase embedding vector using a
weighted average of words in that
phrase (Arora, et al.)
https://figshare.com/articles/Biomedical_Word_Vectors/9598760
Arora, S., Liang, Y., & Ma, T. (2016). A simple but tough-to-beat baseline for sentence embeddings.

12. Methods
12AMIA 2018 | amia.org
(Sentence) Most commonly used agents are
methotrexate (MTX), hydroxychloroquine
(HCQ), sulfasalazine (SSZ), andleflunomide
(LEF).
(Chapter) Rheumatoid Arthritis
(Section) Treatment
A)Snippet from Ferri’s Clinical
Advisor 2019 Medical Textbook
C)PhraseEmbeddingGeneration
Sentence Embedding
Section Embedding
Relation Type Embedding
Relation Embedding
D) Cosine Similarity
Computation
• simb
• sima
B) Elsevier’s Healthcare
Knowledge Graph (HG)
Semantic
Relation
Similarity Matrices:
- Section titles and
relation types
- Book sentences and
HG relations

13. Methods
13AMIA 2018 | amia.org
(Sentence) Most commonly used agents are
methotrexate (MTX), hydroxychloroquine
(HCQ), sulfasalazine (SSZ), andleflunomide
(LEF).
(Chapter) Rheumatoid Arthritis
(Section) Treatment
A)Snippet from Ferri’s Clinical
Advisor 2019 Medical Textbook
C)PhraseEmbeddingGeneration
Sentence Embedding
Section Embedding
Relation Type Embedding
Relation Embedding
D) Cosine Similarity
Computation
• simb
• sima
boostO
boostS
B) Elsevier’s Healthcare
Knowledge Graph (HG)
Semantic
Relation
E) Concept Mapper
Snippet-Relation
Mapping Score
Boosting of a Score
- Hierarchy traversal
- Presence of parents or
descendants in a text snippet

14. Evaluation Methodology
14AMIA 2018 | amia.org
• Candidate relation–sentence mappings for 3 diseases: Diabetes Mellitus, Asthma,
and Rheumatoid Arthritis, using content from Ferri and Conn.
• Mappings selected with a threshold score ≥ 2.5 (decided heuristically).
• 3 domain experts, with biomedical knowledge, independently review 2 random
samples of 250 mappings on a 4-point scale:
• 0: No association between snippet and HG relation
• 1: Negative association between snippet and HG relation
• 2: Positive association between snippet and HG relation
• 3: Uncertain association between snippet and HG relation
• Inter-reviewer agreement computed using the Fleiss’ kappa metric.

15. 15AMIA 2018 | amia.org
Candidate Snippets for a
Semantic Relation
• Phrase Embedding Generation
• Boosting using Concept Mapper
and HG Hierarchy
• Evaluation Methodology
Evaluation and Application
• Evaluation over relation—snippet
pairs for 3 diseases
• HG knowledge coverage
• Future Work and Application

16. Results – Moderate Agreement
16AMIA 2018 | amia.org
• Ferri: 245,809 sentences, 22,246 sections, 1,037 chapters.
• Conn: 88,711 sentences, 5,278 sections, 331 chapters.
• 1,015 candidate snippets from Ferri and 873 candidate snippets from Conn for 3
diseases with score threshold ≥ 2.5:
• Diabetes Mellitus (161 relations),
• Asthma (274 relations), and
• Rheumatoid Arthritis (223 relations).
• Considering the 4-point evaluation scale, Fleiss’ kappa metric for the inter-reviewer
agreement was 0.404 for Ferri and 0.378 for Conn.
• Assuming a binary scale (i.e., only a positive association is valid), Fleiss’ kappa
metric increased to 0.533 for Ferri and 0.492 for Conn.

17. Results – 61.4% Precision
17AMIA 2018 | amia.org
Type of Association Number of Mappings
Positive Association (Consensus) 307 (500 sample mappings - 61.4%)
Uncertain Association (Consensus) 57 (11.4%)
Negative Association (Consensus) 7 (1.4%)
No Association (Consensus) 90 (18%)
Positive Association (Unanimous) 213 (69.8% w.r.t. 307 mappings)
Divergence (No Consensus) 39
• At a precision of 61.4%, recall can be computed to be 86.3%.
• That is, 568 out of 658 semantic relations can be corroborated by at least one text
snippet from either Ferri or Conn at a mapping score >= 2.5.

18. Results – Elsevier HG Coverage in Books
18AMIA 2018 | amia.org
• A large proportion of mappings for
relation type “has differential
diagnosis” has a consensus score
of 0 (no association).
• 384 semantic relations for 3
diseases can be corroborated by
multiple text snippets across both
the medical textbooks.

19. Examples of Divergence between Reviewers
19AMIA 2018 | amia.org
Relation Chapter Section Sentence
Asthma
has drug Epinephrine
Asthma Basic
Information
Status asthmaticus, or acute severe asthma,
is a refractory state that does not respond to
standard therapy such as inhaled beta-
agonists or subcutaneous epinephrine.
Rheumatoid Arthritis
has drug Anakinra
Juvenile
Idiopathic
Arthritis
Treatment IL-1 and IL-6 antagonists, anakinra, and
tocilizumab for those with systemic JIA: Meta-
analysis of randomized controlled trials did
not show statistically significant differences in
the efficacy or safety profile of these agents.
Diabetes Mellitus
has clinical finding Nausea
Diabetes
Mellitus
Treatment -
General Rx
Nausea is its major side effect.
Diabetes Mellitus
has clinical finding Vomiting
Diabetes
Mellitus
Treatment Major manifestations are postprandial
fullness, nausea, vomiting, and bloating.

20. Future Work and Application
20AMIA 2018 | amia.org
• Use of graph embedding vectors and contextualized word embedding
vectors for better precision and recall.
• Comparison against baseline methods, unstructured literature (unlike
medical textbooks), and using external resources (e.g., UMLS).
• Search Application that is powered by Elsevier’s Healthcare Knowledge
Graph to retrieve medical relations and corroborating snippets from
medical textbooks given a user query (e.g., “Drugs to treat Asthma in a
pregnant adult” or “Diagnostic procedures for Rheumatoid Arthritis”).
Acknowledgments: Cailey Fitzgerald, David Childs, Sravanthi Tummala, Connor Skiro,
Danielle Walsh, Paul Snyder, Steve Ross, Doug Anderson, David Conrad, Dru Henke

21. Thank you!
Contact:
Maulik R. Kamdar
Senior Data Scientist, Elsevier Health
m.kamdar@elsevier.com