The document discusses medical conversations and the potential for AI-enabled conversational systems to improve healthcare. It covers: 1) How doctors currently conduct medical diagnosis through a question-and-answer process to understand patients' symptoms and medical histories. 2) Challenges doctors face in making accurate diagnoses given time constraints and the growing complexity of medical data and knowledge. 3) Opportunities for AI to enhance healthcare through personalized recommendations, multimodal data integration, and proactive diagnosis via conversational systems and sensors. 4) Recent advances in natural language processing, machine learning and medical language representations that enable building such conversational medical assistants.

1. Learning
To Speak
Medicine
Xavier Amatriain

2. Part I. The Medical Conversation

3. Medical Diagnosis
● Diagnosis (R.A. Miller 1990):
○ Mapping from patient’s data
(history, examination, lab exams…)
to a possible condition.
○ It depends on ability to:
■ Evoke history
■ Surface symptoms and
findings
■ Generate hypotheses that
suggest how to refine or
pursue different hypothesis
○ In a compassionate,
cost-effective manner
● Doctor: “How can I help?”
● Patient: Chief complaint
● Doctor: “Anything else?”
● Patient: ….
● Start of a doctor-led Q&A:
○ E.g. “Do you have X?”
● Doctor communicates actionable
recommendation (diagnosis +
treatment, triage, referral…)

4. Medicine is hard(er)
● Doctors have ~15 minutes to capture
information* about a patient, diagnose,
and recommend treatment
● *Information
○ Patient’s history
○ Patient’s symptoms
○ Medical knowledge
■ Learned years ago
■ Latest research findings
■ Different demographics
● Data is growing over time, so is complexity

5. The future of healthcare
● Patient-centered, AI-driven, doctor-in-the-loop
● Increased value of patient/doctor interaction
● Data-enabled applications
● Individual/personalized recommendations
● Multimodal (images, voice, audio…)
● Pro-active Prognosis/Diagnosis (eg. when
sensor reading abnormal or new information
available)
● Identifying and warning about side effect of
medications
● Conversational systems are a core
component
Personalized AI agent
Sensors
Conversational
System
(eliciting and providing
information, recommending
outcomes, treatments, prevention)

6. Part II.
Why now?

7. Is it a new idea?
● Internist -1 (1971) led by Jack Myers
considered (one of) the best clinical
diagnostic experts in the US
● Process for adding a disease requires
2-4 weeks of full-time effort and doctors
reading 50 to 250 relevant publications
● Very structured heuristic-driven dialog
system

8. Is it still hot?

9. Research efforts

10. Data Availability
● Large-scale patient-level clinical data
○ Electronic health records
○ Electronic imaging: x-rays, scans
○ Genes
● Smart devices as health sensors
○ Wearables
○ FDA-approved phone apps
○ High-quality images
● Electronic access to medical research
● Much of this only available in the last few
years

11. The Language of Medicine
● ICD(10)
○ 10th revision of the International Statistical Classification of Diseases and
Related Health Problems (ICD)
○ Codes for diseases, symptoms, findings, complaints...
○ Evolved from Bertillon Classification of Causes of Death (1893)
○ UN gave WHO responsibility for the ICD in 1946
● Snomed Clinical Terms
○ Computer processable collection of medical terms used in clinical
documentation and reporting.
○ Clinical findings, symptoms, diagnoses, procedures, body structures,
organisms substances, pharmaceuticals, devices…
○ Started in 1965, has had multiple revisions and mutations
● UMLS
○ Compendium of many controlled vocabularies (1986)
○ Mapping structure among vocabularies
○ Allows to translate among the various terminology systems

12. Electronic health records
● Digital information about
patients encounters with
doctors or the health
system
● An encounter may include
○ Doctor notes, medications,
procedures, diagnosis
● Mix of structured data and
unstructured text

13. Open Data
● Hnlp
● I2b2
● Mimic-3
● ….

14. Part III. Building Medical
Dialogue Systems

15. Medical conversation as a task-oriented dialogue

16. Intent classification

17. Text representation
● Infer structured representation
An 18-year-old male student presents with severe headache and fever
that he has had for 3 days. Examination reveals fever, photophobia,
and neck stiffness
Demographics:
● Age: 18 years
● Gender: Male
Symptoms:
● headache
○ Severity: severe
● Fever
○ Duration: 3 days
● Photophobia
● Neck stiffness

18. Slot filling in frames

19. Text representation
● Infer latent semantic space through embeddings
A. L. Beam, B. Kompa, I. Fried, N. P. Palmer, X. Shi, T. Cai, and I. S. Kohane. 2018. Clinical Concept Embeddings Learned from Massive
Sources of Medical Data. ArXiv e-prints (April 2018).
108,477 medical
concepts using:
● insurance claims
database of 60 million
members
● 20 million clinical notes
● 1.7 million full text
biomedical journal
articles

20. Entity recognition

21. Entity recognition

22. The Deep Learning promise

23. Learning to infer structure in text
A. Jagannatha and H. Yu, Bidirectional Recurrent Neural Networks for Medical Event Detection in Electronic Health Records, 2016
A. Vani et.al. Grounded Recurrent Neural Networks, arXiv, 2017
bleeding due to warfarin
Adverse
drug effect
- - medication

24. End-to-end Deep Learning Dialog Systems SOTA
Language model input:
User: got home and when i 've been at work i 've been experiencing a
pain like a 4 on my left side by my rib and some days i feel where i had
my iv in my arm on march 2 when i went to the er for my constipation
Dr: ah i see . have you tried anything for relief ?
User: took an acetaminophen but that only last for about an hour my body
burns off medicine really fast i learned that at a young age
Language model output:
Dr: i see . how bad is the pain ? on a scale of 1 - 10 10 being the
worst how would you rate it ?
User: 8
Dr: sounds rough ! applying a warm compress 2 - 3 times a day warm baths
with epsom salt massaging...

25. End-to-end Deep Learning Dialog Systems

26. References

27. ● “An interlingua for electronic interchange of medical information: using frames to map between clinical vocabularies.” Masarie. 1991
● “Health dialog systems for patients and consumers.” Bickmore. 2006
● “Designing a Chatbot for diabetic patients”, Lokman. 2007
● “Computer-assisted diagnostic decision support: history, challenges, and possible paths forward” Miller. 2009.
● “Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)” Murphy. 2010
● “Mining Biomedical Ontologies and Data Using RDF Hypergraphs” Liu et al. 2013.
● “From health search to healthcare: explorations of intention and utilization via query logs and user surveys” White. 2014
● “Pharmabot: A Pediatric Generic Medicine Consultant Chatbot”, Comendador. 2015
● “Identifiable Phenotyping using Constrained Non-Negative Matrix Factorization”. Joshi et al. 2016.
● “Clinical Tagging with Joint Probabilistic Models” . Halpern et al. 2016.
● “MIMIC-III, a freely accessible critical care database” Johnson. 2016
● “Disease named entity recognition by combining conditional random fields & bidirectional recurrent neural networks” Wei. 2016
● “Named Entity Recognition Over Electronic Health Records Through a Combined Dictionary-based Approach”, Pomares. 2016
● “Bidirectional Recurrent Neural Networks for Medical Event Detection in Electronic Health Records”, Jagannatha. 2016
● “Bringing semantic structures to user intent detection in online medical queries”, Zhang. 2017
● “A Conversational Chatbot Based on Kowledge-Graphs for Factoid Medical Questions”, A. Minutolo. 2017
● “MANDY: Towards a Smart Primary Care Chatbot Application”. Ni. 2017
● “Entity recognition from clinical texts via recurrent neural network”, Liu. 2017
● “Knowledge-driven Entity Recognition and Disambiguation in Biomedical Text”. Siu. 2017
● “Deep Learning for Dialogue Systems”. Vivian Chen. 2017
● “Universal Language Model Fine-tuning for Text Classification”, Howard. 2018
● “Clinical Concept Embeddings Learned from Massive Sources of Medical Data” A. L. Beam, et al. ArXiv e-prints (April 2018).
References

28. Yes, we’re hiring!