1. Towards Smart Chatbots for Enhanced Health:
Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing
for Augmented Personalized Health
Keynote: DEEP-DIAL @ AAAI 2019, Honolulu, 27 Feb 2019
Amit Sheth
LexisNexis Ohio Eminent Scholar
The Ohio Center of Excellence in Knowledge-enabled Computing & BioHealth Innovations (Kno.e.sis)
Wright State, USA
1
Icon source used in the entire presentation -
https://thenounproject.com
Presentation template by SlidesCarnival

2. 2
Figure:Avisualhistoryofchatbots
Source:https://chatbotsmagazine.com/a-visual-history-of-chatbots-8bf3b31dbfb2
Chatbot 3.0
Next-Generation Smart Bots
● NATURAL communication
● MULTIMODAL interactions
● Ability to maintain the system, task, and people
CONTEXTS
● PERSONALIZATION
● ABSTRACTION along DIKW
Chatbot 2.0
Current Bots
● Driven by back-and-forth communication between
the system & people
● Automation at the task level
● Ability to maintain both system and task contexts
Chatbot 1.0
Traditional Bots
● System-driven
● Scripted-automation
● Ability to maintain only system context
Evolution of
CHATBOTS

3. Next-Generation Smart Bots
Computing for Human Experience
Promising domain: Computing for Healthcare
3
Source: http://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=1919&context=knoesis
http://wiki.knoesis.org/index.php/Computing_For_Human_Experience
“Computing for Human Experience will employ
a suite of technologies to nondestructively and
unobtrusively complement and enrich normal
human activities, with minimal explicit concern
or effort on the humans’ part.”

4. “
Understanding and managing health is complex!
Throughout the last few decades of modern medicine,
we have relied on clinicians
for most health-related decision making.
4

5. 5
LIMITED DATA due to episodic visits
TIME CONSTRAINT during clinical visits
● Significant information seeking time is required every time
● Comprehending clinical notes which contains only text is difficult
Each individual is DIFFERENT and thus,
personalized treatment is needed.
Insufficient time and data for personalization
Image Source: https://www.istockphoto.com/gb/vector/woman-doctor-examining-patient-by-stethoscope-gm541296730-96809003
WHY Healthcare? [a technology take]
CHALLENGES
TRADITIONAL Healthcare [a technology take]

6. “
New technologies have enabled a growing involvement of
patients in their own health management.
Chatbot could play a pivotal role throughout the
unfolding data & knowledge-driven, AI-supported
ecosystem for ENHANCED HEALTH.
6

7. 7
MULTISENSORY Sensing
Semantic-Cognitive-Perceptual Computing
COGNITIVE UNDERPINNING & EXPLAINABILITY with
Domain Model & Protocols
CONTEXTUALIZATION
PERSONALIZATION
ABSTRACTION
AUGMENTED Personalized Health
Self-monitoring, Self-appraisal, Self-management, Intervention,
Prediction of disease progression and planning
Towards SMART Chatbots for ENHANCED Health
Domain & User-specific
Knowledge Graphs
Natural Language Processing
Machine with Deep Learning

8. Use Cases & Prototypes
Experience & observations based on ongoing collaborative
healthcare projects @ KNO.E.SIS
8

9. Health Related Studies at KNO.E.SIS [Overview]
HealthChallenges
(Also Dementia,
Obesity,
Parkinson’s, Liver
Cirrhosis, ADHF)
Public Policy/ Population Epidemiology Personalized Health
PCS + EMR + Multimodal
(Speech + Image)
kHealth
Asthma in Children
Bariatric Surgery
Nutrition
Physical(IoT)/Cyber/
Social (PCS)+ EMR
Marijuana Social
Drug Abuse Social
Mental Health
Depression & Suicide Social + Public + EMR
Health
Knowledge Graph
Services
Social + Clinical Data
...and infrastructure technologies:
Context-aware KR (SP),
KG development,
Smart Data from PCS Big Data,
Twitris
9

10. 10
HCI: Mobile Applications & Chatbots @ KNO.E.SIS
kHealth
Asthma
kHealth
Bariatrics
Depression
Active (Subset)
Healthcare Projects
@ KNO.E.SIS with
mApps/chatbot
kHealth Framework: a knowledge-enabled semantic platform
that captures the data and analyzes it to produce actionable
information.
3 Chatbots (Alpha Stage)
1. NOURICH: A Google Assistant based
Conversational Nutrition Management System
1. Knowledge-enabled (kHealth) Personalized
ChatBot for Asthma: Contextualized &
Personalized Conversations involving
Multimodal data (IoT & Devices)
1. ReaCTrack: Personalized Adverse Reaction
Conversation-based Tracker for Clinical
Depression
3 Applications
1. NOURICH: Food image-recognition app.
2. kHealth Asthma (patient evals)
3. kHealth Bariatrics (patient evals)

11. 11
Physical-Cyber-Social (PCS) Data
Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1),
peak expiratory flow (PEF), indoor temperature, indoor humidity,
particulate matter, volatile organic compound, carbon dioxide,
air quality index, pollen level, outdoor temperature, outdoor
humidity, number of steps, heart rate and number of hours of
sleep. Also clinical notes.
kHealth Asthma Nutrition
Depression
Active Healthcare Projects
in Kno.e.sis (Subset)
Modality of Data
kHealth Bariatrics
For monitoring asthma control and predict vulnerability
Pre and Post Surgery monitoring and self adherence
Mobile app Q/A (tablet), weighing scale, pill bottle sensor, water
bottle sensor for reminder to drink water, number of steps, heart
rate and number of hours of sleep. Also clinical notes.
Q/A, diet, images, food profile, nutrition
knowledge base, user knowledge graph
For nutrition tracking and diet monitoring
Modeling Social Behavior for Healthcare Utilization in Depression
Q/A, social media profile (Twitter, Reddit)

12. Multisensory Sensing &
Multimodal Data
Text, Speech, Image Processing Framework for
“natural” communication
12

13. 13
Figure: An illustration of how a basic chatbot can be
extended with multimodal data and input
Multisensory Sensing Framework

14. 14
Use Case: kHealth Asthma
Many Sources of Highly Diverse Data
(& collection methods: Active + Passive):
Up to 1852 data points/ patient /day
http://bit.ly/kHealth-Asthma
kBot with screen
interface for conversation
Images
Text
Speech
★ Episodic to Continuous Monitoring
★ Clinician-centric to Patient-centric
★ Clinician controlled to Patient-empowered
★ Disease Focused to Wellness-focused
★ Sparse data to Multimodal Big Data
*(Asthma-Obesity)

15. Semantic-Cognitive-Perceptual
Computing
Knowledge-Infused AI
with Contextualization (Knowledge Graphs), Personalization, Abstraction
15

16. 16
Semantic Browsing
Extraction
Data Integration and Interlinking
Entity
Complex Extraction
Aberrant
Drug-related
Behaviour
Neuro-Cognitive
Symptoms
Adverse
Drug
Reaction
Relatio
n
Event Severity
Personal Sensor Data De-identified EMR Blog Post
Context Representation Relevant Subgraph Selection
Semantic Search
Disease-specific
Chatbot
Visualization
Health
Knowledge Graph
Intent
Open Health Knowledge Graph

17. 17
SOCIAL -MEDIA TEXT
(July 12,2016)
EVENT-SPECIFIC
SCHEMA-BASED
KNOWLEDGE

18. 18
Application: Evolving Patient Knowledge Graph (PKG)
Figure: A healthcare assistant bot interacts with the patient via various conversational interfaces (voice,
text, and visual) to acquire and disseminate information, and provide recommendation (validated by
physician). The core functionalities of the chatbot (Component C boxed in blue) are extended with a
background HKG (Component A boxed in green) and a evolving PKG (Component B boxed in orange).
★ Smarter & engaging agent
★ Minimize active sensing
(Questions to be asked)
★ Ask only informed & intelligent
questions
★ Relevant & Contextualized
conversations
★ Personalized & Human-Like

19. Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition (allergic reaction to high
ragweed pollen level) via the
personalized patient’s knowledge
graph generated from EMR, PGHD,
and prior interactions with the kBOT.
Generates predictions or
recommended course of actions.
Inference based on patient’s
historical records and background
health knowledge graph containing
contextualized (domain-specific)
knowledge.
Figure: Example kBOT conversation which
utilizes background health knowledge graph
and patient’s knowledge graph to infer and
generate recommendation to patients.
★ Conversing only information relevant
to the patient
19
Context enabled by relevant
healthcare knowledge including
clinical protocols.

20. 20
Contextualization
refers to data interpretation in terms of knowledge (context).
Without Domain Knowledge With Domain Knowledge
Chatbot with domain
(drug) knowledge is
potentially more natural
and able to deal with
variations.

21. 21
Personalization
refers to future course of action by taking into account the contextual factors such as
user’s health history, physical characteristics, environmental factors, activity, and lifestyle.
Without
Contextualized Personalization
With
Contextualized Personalization
Chatbot with
contextualized (asthma)
knowledge is potentially
more personalized and
engaging.

22. 22
Abstraction
A computational technique that maps and associates raw data to action-related
information.
With AbstractionWithout Abstraction
.

23. 23
Smarter Chatbot with Semantically-Abstracted Information
Smarterdata
Data Sophistication
Smart (semantically-abstracted)
data should answer:
★ What causes my disease severity?
★ How well am I doing with respect to prescribed
care plan?
★ Am I deviating from the care plan? I am
following the care plan but my disease
is not well controlled.
★ Do I need treatment adjustments?
★ How well controlled is my disease over time?
Example of Abstraction

24. 24
Utkarshani Jaimini, Krishnaprasad
Thirunarayan, Maninder Kalra, Revathy
Venkataramanan, Dipesh Kadariya, Amit
Sheth, “How Is My Child’s Asthma?”
Digital Phenotype and Actionable Insights
for Pediatric Asthma”, JMIR Pediatr Parent
2018;1(2):e11988, DOI: 10.2196/11988.

25. 25
Semantic, Cognitive, Perceptual Computing:
Paradigms That Shape Human Experience
http://bit.ly/SCPComputing
Humans are interested in high-level
concepts (phenotypic characteristics).
Semantic Computing: Assign labels and
associate meanings (representation &
contextualization).
Cognitive Computing: Interpretation of
data with respect to perspectives, constraints,
domain knowledge, and personal context.
Perceptual Computing: A cyclical process
of semantic-cognitive computing for higher
level of perception and reasoning (abstraction
& action).

26. 26
Use Case: NOURICH (diet management assistant)
A sample video demo of NOURICH: https://www.youtube.com/watch?v=b2OgFuEAik4

27. 27
NOURICH
An Android app to support food image recognition

28. 28
Scenario: NOURICH (diet management chatbot)
Figure: Architectural process of NOURICH (http://bit.ly/NOURICH)
Scenario
User
Age: 49
Height: 5 ft
Weight: 120 lbs
Diet Plan: Ketogenic
Food Allergies: Peanuts
Diet/Recipe/Article Recommendation System with
Semantic-Cognitive-Perceptual Computing framework
(a) Using domain knowledge, the system searches for and
filters articles related to ketogenic diet.
(a) Using personalized knowledge graph, the system
understands the user is allergic to peanuts.
(a) Combining
● domain and user KGs
● the concept allergy <-> avoid (rule embedded in the
ontology, beyond keywords-matching) and
● diet, calorie constraints, and gender profile
The system will be able to interpret and will not recommend
keto-recipes that have peanuts

29. Augmented Personalized
Health (APH)
Self-monitoring, Self-appraisal, Self-management, Intervention,
Disease Progression and Tracking
29

30. Knowledge-Infused
Learning with
Semantic,
Cognitive,
Perceptual
Computing
Framework
30
Overarching Theory
Knowledge
Domain (Ontology)
Personalized KG
Multisensory
Sensing &
Multimodal
Data Interactions
ImagesText Speech Videos
IoTs
Natural Language
Processing,
Machine with
Deep Learning
AUGMENTED PERSONALIZED
HEALTH (APH)Modeling broader disease context, and
personalized user behavior
Reasoning & decision-
making framework
Minimize data overload, assist in making
choices, appraisal, recommendations

31. 31
This not only prevent the disease, but also enhances the patient’s health
BariatricsAsthma
Use Cases: APH for Asthma and Bariatrics: Patient-centric drivers

32. 32
❖ Health management is complex.
❖ Knowledge-infused learning could give use the power need to match
complex requirements.
❖ Multisensory and Multimodal data interactions are essential for
natural communications.
❖ Semantic-Cognitive-Perceptual Computing enables contextualization,
personalization, and abstraction for Augmented Personalized Health.
In Short,
This research is supported by NICHD/NIH under the Grant Number: 1R01HD087132.
The content is solely the responsibility of the authors and does not necessarily
represent the official views of the National Institutes of Health.

33. 33
Special Thanks
Hong Yung (Joey) Yip
(Graduate Student)