Keynote at the SWAT4HCLS (Semantic Web Applications and Tools for Healthcare and Life Sciences), 12 Jan 2022. Event info: http://www.swat4ls.org/workshops/leiden2022/keynotes/ Video: https://youtu.be/nwGAv9q2wsY Healthcare as we know it is in the process of going through a massive change – from episodic to continuous, from disease-focused to wellness and quality of life focused, from clinic centric to anywhere a patient is, from clinician controlled to patient empowered, and from being driven by limited data to 360-degree, multimodal personal-public-population physical-cyber-social big data-driven. While the ability to create and capture data is already here, the upcoming innovations will be in converting this big data into smart data through contextual and personalized processing such that patients and clinicians can make better decisions and take timely actions. The exploitation of all relevant data, relevant medical knowledge, and explainable AI techniques will also support better communications between patients, clinicians, and virtual health assistants with higher-level abstractions (rather than low-level data) representing health choices, decisions and actions. Augmented Personalized Healthcare (APH) strategy we are developing empowers patients with self-monitoring (collecting relevant data), self-appraisal (interpreting data in the patient’s context), self-management (assisting the patient in following personalized care plan to maintain health), to intervention (when the clinical help is needed) and disease progression tracking and prediction (http://bit.ly/AI-APH, http://bit.ly/APH-TED). We currently apply APH using mobile Apps and virtual health assistants for patients managing pediatric asthma (http://bit.ly/kAsthma), mental health, carbohydrate management for type 1 diabetes, hypertension, etc. In this talk, I will describe some of the technical components that incorporate context, personalization, and abstraction for supporting advanced capabilities such as patient engagement through meaningful question generation, chatbot safety, and explainable decision-making using knowledge-infused learning, a neurosymbolic AI strategy that utilizes many types and levels of explicit knowledge.

1. Augmented Personalized Health
An explicit knowledge enhanced neurosymbolic dHealth approach to patient empowerment
for managing chronic disease burden
Background: http://bit.ly/APH-TED, http://bit.ly/k-APH, http://bit.ly/kAsthma
SWAT4HCLS 2022
Semantic Web Applications and Tools for Health Care and Life Science
Keynote on 12 Jan 2022

2. 2
Amit Sheth
Founding Director,
Artificial Intelligence Institute http://aiisc.ai
The University of South Carolina
amit@sc.edu http://amit.aiisc.ai
Special Thanks
Some of the Healthcare collaborators:
Maninder Kalra (Dayton Children’s Hospital)
Phillis Raynor, Ronda Huges, Sara Donevant(UofSC)
Lisa Knight (Prisma Health)
Some of the CS collaborators:
Manas Gaur (AIISC)
Krishnaprasad Thirunarayan (Wright State University)
Kaushik Roy
AIISC, kaushikr@email.sc.edu
Utkarshani Jaimini
AIISC, ujaimini@email.sc.edu
Ack: NIH/NICHD 1 R01 HD087132-01: KHealth: Semantic Multisensory
Mobile Approach to Personalized Asthma Care
Hong Yung (Joey) Yip
AIISC, hyip@email.sc.edu
Revathy Venkataramanan
AIISC, revathy@email.sc.edu

3. HCLS Research/Applications/Tools at #AIISC
Types of Data:
◎ Patient Generated Health Data (e.g.,
food images, IoT/sensors, Q/A)
○ Multimodal (fMRI, speech,
biosignals)
◎ Social Media, DarkNets, Web Forums,
satellite/drone images,
◎ Scientific literature
◎ EMR/clinical notes
Medical Conditions/Collaborations:
Autism, Aphasia, Diabetes,
Hypertension, Asthma, Cardiovascular
health, Mental Health, Drug
Abuse/Addiction, Radiology reports,
Colorectal Cancer, spread of malaria,
COVID-19
Applications & Tools: mApp,
Virtual Health Assistants
(Chatbots), Recommender
Systems, Clinical Support
Objectives: Surveillance, Diagnosis,
Disease Management, Decision
Making

4. Augmented Personalized Health (APH)
Big Data to Smart Data
APH is a vision to enhance the healthcare by using AI techniques on semantically
integrated PGHD, environmental data, clinical data, public health data & social data.

5. APH: Health Management Strategies

6. APH/Knowledge-enabled Healthcare:
kHealth Asthma
A multisensory approach for personalized asthma care for children
◎ 6.3 million children in USA are affected by Asthma; 300 million adults & children worldwide
◎ Multifactorial disease, difficult to diagnose based on episodic visits and clinical records
◎ Stage 1: Mobile App with 29 parameters collected
◎ Stage 2: Virtual Health Assistant

7. Augmented Personalized Health for Asthma
Smart data can answer
- What causes my disease severity?
- How well am I doing with respect to
prescribed care plan?
- Am I deviating from the care plan?
- I am following the care plan but my
disease is not well controlled. Do I
need treatment adjustments?
- How well controlled is my disease
over the time?

8. kHealth Asthma Overview

9. Self Monitoring with kHealthDash:
Knowledge enabled personalized DASHboard for Asthma Management
Video link - https://youtu.be/yUgXCPwc55M

10. Self Appraisal with Digital Phenotype Score
Jaimini U, Thirunarayan K, Kalra M, Venkataraman R, Kadariya D, Sheth A, “How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma, JMIR Pediatric Parent 2018;1(2):e11988
https://medium.com/leoilab/digital-phenotyping-turning-our-smartphones-inward-141a75b2f2a3
● Digital Phenotype Score (DPS) is defined as the score
to quantify the digital phenotypes collected from the
social media, smartphones, wearables, and sensors
streams.
● DPS acts as a cumulative measure for the abstraction
of knowledge and information from the raw digital
phenotypic data.
● The integration of the DPS can enable personalized
interventions in real time which are directly responsive
to the healthcare need of a patient.

11. Digital Phenotype Score vs Asthma Control Test Score
Jaimini U, Thirunarayan K, Kalra M, Venkataraman R, Kadariya D, Sheth A, “How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma, JMIR Pediatric Parent 2018;1(2):e11988

12. Determining Personalized Asthma Triggers: Seasonal Dependency
Venkataramanan R, Thirunarayan K, Jaimini U, Kadariya D, Yip HY, Kalra M, Sheth A. Determination of Personalized Asthma Triggers From Multimodal Sensing and a Mobile App: Observational Study. JMIR Pediatr Parent 2019;2(1):e14300. doi: 10.2196/14300
PMID: 31518318 PMCID: 6716491

13. Evidence based Path to Personalization
Patient-A was monitored for 13 weeks encompassing winter to spring 2018. Type: Severe, low medication compliance.
Absence of Pollen
First 6 weeks
Presence of Pollen
Rest of the 7 weeks
Pre (observe)
4 weeks
Post (validate)
2 weeks
Pre
4 weeks
Post
3 weeks
Pollen 0 Pollen 0 days Pollen 17 days Pollen 3 days
PM2.5 20 days PM2.5 5 days PM2.5 14 days PM2.5 2 days
Ozone 1 day Ozone 0 Ozone 0 Ozone 1 day
Asthma
Episodes*
21 days Asthma
Episodes
5 days Asthma
Episodes
17 days Asthma
Episodes
3 days
● Absence of Pollen - PM2.5 is the trigger
● Presence of Pollen - Pollen and PM2.5. Severe symptoms occurred in this period. Presence of both PM2.5 and
Pollen increased the intensity of asthma episodes.
Venkataramanan R, Thirunarayan K, Jaimini U, Kadariya D, Yip HY, Kalra M, Sheth A. Determination of Personalized Asthma Triggers From Multimodal Sensing and a Mobile App: Observational Study. JMIR Pediatr Parent 2019;2(1):e14300. doi: 10.2196/14300
PMID: 31518318 PMCID: 6716491

14. Personalized Health Knowledge Graph

15. kHealth Chatbot (With Personalized Health Knowledge Graph)
Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition (allergic reaction to high
ragweed pollen level) via the
personalized patient’s knowledge
graph generated from EMR, PGHD,
and prior interactions with the kBOT.
Generates predictions or
recommended course of actions.
Inference based on patient’s
historical records and background
health knowledge graph containing
contextualized (domain-specific)
knowledge.
Figure: Example kBOT conversation which utilizes
background health knowledge graph and patient’s
knowledge graph to infer and generate
recommendation to patients.
★ Conversing only information relevant to the
patient
Context enabled by relevant
healthcare knowledge including
clinical protocols.
kBOT initiates greeting
conversation.
Generic Chatbot
(Without Personalized Health Knowledge Graph)

16. kHealth Chatbot Demo
https://youtu.be/oZCaM7rhJAg
More on: https://www.slideshare.net/aiisc/kbot-knowledgeenabled-personalized-chatbot-for-selfmanagement-of-asthma-in-pediatric-population

17. CausalKG
Causal Knowledge Graph
Utkarshani Jaimini and Amit Sheth, "CausalKG: Causal Knowledge Graph: Explainability using interventional and counterfactual reasoning, " IEEE Internet Computing, 26 (1), Jan-Feb 2022

18. Why CausalKG?
● Humans have innate understanding of causality
● Causality is a complex relationship
● Asthma trigger causes-
➔ An occurrence of a symptom
➔ Intake of medication

19. Personalized Causal Bayesian Network
Personalized Causal Bayesian Network, a graphical representation of the causal relations and interaction between a
patient’s asthma triggers (such as pollen, AQI, Ozone), asthma symptom (such as cough, wheeze, chest tightness,
hard and fast breathing, and nose opens wide), and medication intake (such as controller, rescue, and allergy
medication)

20. CausalKG Architecture
CausalKG Architecture consists of three main steps: 1) a Causal Bayesian Network and kHealth: pediatric asthma patient
observational dataset, 2) Causal Ontology creation and enriching the asthma ontology with causal relationships, and 3) Estimating
the causal effects of the trigger, symptom, and medication variable in the pediatric asthma patient.

21. CausalKG:
Personalized Causal Knowledge Graph for the asthma patient
CausalKG for a pediatric asthma scenario represented as a hyper-relational graph. Each node in the CausalKG is a
concept in knowledge graph and is associated with a conditional probability estimated using the causal Bayesian
network. The edges between the nodes represent the causal relationships between the concepts.

22. Current AI to Future Hybrid AI
From statistical explainability based on data representation and associational
support to context explain- ability based on causal relation and interventional
support eventually leading to domain explainability based on causal
representation in KG leading to counterfactual support.

23. APH: Virtual Health Assistant assisted Self
Management using Reinforcement Learning and
Personalized Knowledge Graph
Use case: Mental Health

24. Construction of Personalized Knowledge Graph
PKG <- Patient Data + DS +
interactions

25. AMR - Parser
Discharge Summary
Spacy NER
AMR Graph
Chemicals
(Medicine)
Diseases
Knowledge Base
Personalized Context
Ms. D meets criteria for recurrent Major Depressive Disorder and Generalized Anxiety Disorder. She
also is currently receiving treatment for ADHD and has some traits consistent with Obsessive
Compulsive Personality Disorder, although she does not appear to meet full OCPD traits. She is
currently completing a taper of Prozac (reached dose of 40mg daily) and Abilify (reached dose of
10mg daily) for depressive symptoms in anticipation of starting a different medication.

26. Task: Predicting disease related
aspects

27. Task: Predicting drug-drug
interaction related aspects

28. Really struggling with my bisexuality which
is causing chaos in my relationship with a
girl. I am equal to worthless for her. I’m now
starting to get drunk because I can’t cope
with the obsessive, intrusive thoughts, and
need to get out of my head.
288291000119102: High risk bisexual behavior
365949003: Health-related behavior finding 365949003: Health-related behavior finding
307077003: Feeling hopeless
365107007: level of mood
225445003: Intrusive thoughts
55956009: Disturbance in content of thought
26628009: Disturbance in thinking
1376001: Obsessive compulsive personality disorder
Multi-hop traversal on
medical knowledge
graphs
<is symptom>
Obsessive-compulsive disorder is a disorder in
which people have obsessive, intrusive thoughts,
ideas or sensations that make them feel driven to do
something repetitively
Explain interpretation through
linking to KG and definitions

29. Knowledge-infused Reinforcement Learning
● The input to the agent is sequential through many steps, it gets an input and a reward at every step and
learns the right output gradually through reinforcement.

30. Insomnia
restlessness
Lack
Of
sleep
prescribe(MsD, “Melatonin”).
use(MsD, 0.5mg, “Melatonin”).
frequency(0.5mg, “Melatonin”, 1, day).
Clinical
guidelines
+ plan
Abstraction

31. APH Self Management: Reinforcement Learning

32. Information Gathering Process Knowledge

33. Yes/no? frequency/
degree?
Causes? Remedies ….
Do you feel nervous?
Yes/no?
How often do you feel nervous?
Question Probability: 0.95
Heuristic Score - Process: 4
- N - R = 5 - 1 = 4
Heuristic Score - Validity: 1
- “nervous” matches
nervousness or being easily
startled in physical symptom of
Generalized Anxiety Disorder in
the KB of Mayo Clinic
Heuristic Score - Safety: 0
- |LSafety| = 0 unsafe matches
frequency/
degree?
Causes? ...
More Informative Evaluation Metrics

34. Information Gathering
Hello X
Hi
I want to ask you some questions regarding
your Anxiety Disorder. I understand that
you’ve mentioned feeling nervous, but how
likely do you feel this way?
These thoughts cross my
mind hundreds of times.
Any ideas on what might
be causing this?
I have been in a very bad
financial condition because of a
huge debt, which I am unable to
return.
. . . .
. . . .
GAD – 7
Questionnaire
Feeling Nervous, anxious or on edge
Not being able to stop or control
worrying
Worrying, too much about
different things
Trouble Relaxing
Being so restless that it is hard to
sit still
Becoming easily annoyed or
irritable
Feeling afraid, as if something
awful might happen
1. Do you feel nervous or on
edge?
2. How likely do you feel this
way?
3. Any ideas on what maybe
causing this?
4. Have you tried any remedies to
feel any better?
5. Are you also feeling any other
symptoms such as jitters or
dread.
Explanation
Question Probability: 0.95
Heuristic Score - Process: 2
Heuristic Score - Validity: 1
Heuristic Score - Safety: 0

35. Detect Suicidal Thought (Parallel to Act 01)
Hello X
Hey
How are you doing today?
I feel low today. I wished
that I never wake up. So, I
never have to face it.
Have you actually had any
thoughts of killing yourself?
Yes, I’ve had it for a long time
now.
*Contacts 911 Emergency Hotline*
CSSR
S

36. Food plays pivotal role in managing
(preventing, treating) many diseases.
Dietary Guideline of Americans:
Focus on reducing excess calorie
consumption and making an informed
decision about food choices and physical
activity can help attain a healthier weight and
reduce the risk of chronic illness.
Key questions:
What is in the food?
How is the food
prepared?
What is the
nutrition break
down?
Is this food good for
my health?
Food that matches
my preference
APH: Self-Management of Diet for Chronic Conditions with
Knowledge Graph for Explainability
(an example of Precision Nutrition)

37. System capabilities
- Recognize food and estimate volume using computer vision
techniques
- Extract ingredients and cooking methods to analyze the recipe
- Estimate calorie intake and nutrition break down of the
food using domain knowledge from Edamam
- Recommend meals based on user’s health condition and food preferences using
Personalized Health Knowledge Graph also utilizing gut microbiome
- Trend of weight vs calorie intake (or against any other parameters)
Image source: https://hips.hearstapps.com/hmg-prod.s3.amazonaws.com/images/woman-taking-photo-of-diet-food-with-mobile-phone-royalty-free-image-1141629022-1562089609.jpg?crop=0.670xw:1.00xh;0.166xw,0&resize=640:*

38. CNN Recognition
model
Volume Estimator
(Computer Vision
techniques)
Food name
Food
Volume
NUTRITION KG
Calorie Count
ANALYZER
REASONER
Personalized KG
- Glucose level
- Gut microbiome*
- Comorbidities*
Diabetes KG
- Diabetes rules
- ConceptNet (food-type)
- Effects of cooking
YES
Healthy CHO: Carrot, Kale, Broccoli
Unhealthy CHO: None
Cooking method: Boiling
Your DV of CHO: within limit
Allergy: None
CAN I EAT THIS FOOD OR NOT? WHY?
Knowledge Graph
(explainability and personalization)
Rule based +
ML/DL
TAGGER
(Cooking Methods)
Ingredients
Recipe
Frying, boiling,
stir-fry
2.3M recipes
900K food items
40+ major diets/allergens
150+ nutrients
Cooking
method KG

39. Can I eat this? Yes
Healthy CHO: carrot, kale, broccoli,
white bean
Unhealthy CHO: None
Cooking method: Boiling
Your DV of CHO: Within limit
Allergy: None
Can I eat this? No
Healthy CHO: Potato
Unhealthy CHO: None
Cooking method: Frying
Your DV of CHO: Within limit
Allergy: None
SAMPLE RESPONSES
Recipe name: Slow-Cooker Chicken &
White Bean Stew Recipe Name: Potato Fries
Can I eat this? No
Healthy CHO: None
Unhealthy CHO: Added sugar
Cooking method: Baking
Your DV of CHO: Within limit
Allergy: Dairy
Recipe Name: Cheesecake
Cooking method is acceptable
and CHO daily value is within the
limit. But it contains unhealthy
CHO.
Cooking method is frying which
introduces unhealthy fat.
All good!

40. BARIATRICS
TYPE-1 DIABETES
Bariatric surgery patients need to adhere to
their post surgical protocol which mainly
involves diet and weight monitoring.
Type-1 diabetes patients need to know their
carbohydrate intake for every meal to
calculate their insulin dosage.
HYPERTENSION
Hypertension patients need to avoid high
sodium foods. Additionally, hypertension
occurs with co-morbidities
APPLICATIONS

41. “
42
AIISC in core AI areas, and
interdisciplinary AI/AI applications

42. Thanks!
Open to Questions?
You can find me at:
amit@sc.edu
https://aiisc.ai/
https://www.linkedin.com/company/1054055/
http://bit.ly/AIISC
43
Acknowledgement
This research is supported by National Institutes of
Health under the Grant Number 1 R01HD087132. The
content of this study is solely the responsibility of the
authors and does not necessarily represent the official
views of the National Institutes of Health.