Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Towards Smart Chatbots for Enhanced Health:
Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing
for Augme...
2
Figure:Avisualhistoryofchatbots
Source:https://chatbotsmagazine.com/a-visual-history-of-chatbots-8bf3b31dbfb2
Chatbot 3....
Next-Generation Smart Bots
Computing for Human Experience
Promising domain: Computing for Healthcare
3
Source: http://core...
“
Understanding and managing health is complex!
Throughout the last few decades of modern medicine,
we have relied on clin...
5
LIMITED DATA due to episodic visits
TIME CONSTRAINT during clinical visits
● Significant information seeking time is req...
“
New technologies have enabled a growing involvement of
patients in their own health management.
Chatbot could play a piv...
7
MULTISENSORY Sensing
Semantic-Cognitive-Perceptual Computing
COGNITIVE UNDERPINNING & EXPLAINABILITY with
Domain Model &...
Use Cases & Prototypes
Experience & observations based on ongoing collaborative
healthcare projects @ KNO.E.SIS
8
Health Related Studies at KNO.E.SIS [Overview]
HealthChallenges
(Also Dementia,
Obesity,
Parkinson’s, Liver
Cirrhosis, ADH...
10
HCI: Mobile Applications & Chatbots @ KNO.E.SIS
kHealth
Asthma
kHealth
Bariatrics
Depression
Active (Subset)
Healthcare...
11
Physical-Cyber-Social (PCS) Data
Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1),
peak expiratory flow (...
Multisensory Sensing &
Multimodal Data
Text, Speech, Image Processing Framework for
“natural” communication
12
13
Figure: An illustration of how a basic chatbot can be
extended with multimodal data and input
Multisensory Sensing Fram...
14
Use Case: kHealth Asthma
Many Sources of Highly Diverse Data
(& collection methods: Active + Passive):
Up to 1852 data ...
Semantic-Cognitive-Perceptual
Computing
Knowledge-Infused AI
with Contextualization (Knowledge Graphs), Personalization, A...
16
Semantic Browsing
Extraction
Data Integration and Interlinking
Entity
Complex Extraction
Aberrant
Drug-related
Behaviou...
17
SOCIAL -MEDIA TEXT
(July 12,2016)
EVENT-SPECIFIC
SCHEMA-BASED
KNOWLEDGE
18
Application: Evolving Patient Knowledge Graph (PKG)
Figure: A healthcare assistant bot interacts with the patient via v...
Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition (al...
20
Contextualization
refers to data interpretation in terms of knowledge (context).
Without Domain Knowledge With Domain K...
21
Personalization
refers to future course of action by taking into account the contextual factors such as
user’s health h...
22
Abstraction
A computational technique that maps and associates raw data to action-related
information.
With Abstraction...
23
Smarter Chatbot with Semantically-Abstracted Information
Smarterdata
Data Sophistication
Smart (semantically-abstracted...
24
Utkarshani Jaimini, Krishnaprasad
Thirunarayan, Maninder Kalra, Revathy
Venkataramanan, Dipesh Kadariya, Amit
Sheth, “H...
25
Semantic, Cognitive, Perceptual Computing:
Paradigms That Shape Human Experience
http://bit.ly/SCPComputing
Humans are ...
26
Use Case: NOURICH (diet management assistant)
A sample video demo of NOURICH: https://www.youtube.com/watch?v=b2OgFuEAi...
27
NOURICH
An Android app to support food image recognition
28
Scenario: NOURICH (diet management chatbot)
Figure: Architectural process of NOURICH (http://bit.ly/NOURICH)
Scenario
U...
Augmented Personalized
Health (APH)
Self-monitoring, Self-appraisal, Self-management, Intervention,
Disease Progression an...
Knowledge-Infused
Learning with
Semantic,
Cognitive,
Perceptual
Computing
Framework
30
Overarching Theory
Knowledge
Domain...
31
This not only prevent the disease, but also enhances the patient’s health
BariatricsAsthma
Use Cases: APH for Asthma an...
32
❖ Health management is complex.
❖ Knowledge-infused learning could give use the power need to match
complex requirement...
33
Special Thanks
Hong Yung (Joey) Yip
(Graduate Student)
Upcoming SlideShare
Loading in …5
×

Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health

228 views

Published on

https://sites.google.com/view/deep-dial-2019/keynotes

Understanding and managing health is complex. Throughout the last few decades of modern medicine, we have relied clinicians on most health-related decision making. New technologies have enabled a growing involvement of patients in their own health management, aided by increasing variety and amount of patient-generated health data. Augmented personalized health [http://bit.ly/k-APH, http://bit.ly/APH-HI] strategy has outlined a broad variety of patient and clinician engagement in devising an increasingly more sophisticated and powerful health management solutions - from self-monitoring, self-appraisal, self-management, intervention to the prediction of disease progression and planning. Chatbot could play a pivotal role throughout the unfolding data-driven, AI-supported ecosystem [http://bit.ly/H-Chatbot] that engages patients and clinicians in collecting data, in driving their actions, informing them of their choices, and even delivering part of the clinical care (e.g., Cognitive Behavioral Therapy (CBT) for mental health patients). Nevertheless, this will require quite a few advances in making a more intelligent technology. In this talk, we will share some experience and observations based on our ongoing collaborative projects that usually involve clinicians and patients targeting pediatric asthma management, pre-and-post bariatric surgery care regimen, depression and other mental health issues, and nutrition. Using use cases and prototypes, we will elucidate the need, support, and use of domain- and user-specific knowledge graphs, Natural Language Processing (NLP), machine learning, and conversational AI for:

- multimodal interactions including text, voice, and other media, along with the use of diverse devices and software platforms for “natural” communication

- context enabled by deep relevant medical/healthcare knowledge including clinical protocols

- personalization by collecting and using the history of the individual patient from IoT health devices, open data, and Electronic Medical Record (EMR)

- abstraction by aggregating and correlating diverse streams data to draw plausible explanation(s) based on public (cohort-level) data (for example percentage of asthmatic patient who gets symptom when exposed to certain triggers) and personal data

- smart dialogue (intent) management and response generations by causal relations and inference of association

  • Be the first to comment

  • Be the first to like this

Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health

  1. 1. Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health Keynote: DEEP-DIAL @ AAAI 2019, Honolulu, 27 Feb 2019 Amit Sheth LexisNexis Ohio Eminent Scholar The Ohio Center of Excellence in Knowledge-enabled Computing & BioHealth Innovations (Kno.e.sis) Wright State, USA 1 Icon source used in the entire presentation - https://thenounproject.com Presentation template by SlidesCarnival
  2. 2. 2 Figure:Avisualhistoryofchatbots Source:https://chatbotsmagazine.com/a-visual-history-of-chatbots-8bf3b31dbfb2 Chatbot 3.0 Next-Generation Smart Bots ● NATURAL communication ● MULTIMODAL interactions ● Ability to maintain the system, task, and people CONTEXTS ● PERSONALIZATION ● ABSTRACTION along DIKW Chatbot 2.0 Current Bots ● Driven by back-and-forth communication between the system & people ● Automation at the task level ● Ability to maintain both system and task contexts Chatbot 1.0 Traditional Bots ● System-driven ● Scripted-automation ● Ability to maintain only system context Evolution of CHATBOTS
  3. 3. Next-Generation Smart Bots Computing for Human Experience Promising domain: Computing for Healthcare 3 Source: http://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=1919&context=knoesis http://wiki.knoesis.org/index.php/Computing_For_Human_Experience “Computing for Human Experience will employ a suite of technologies to nondestructively and unobtrusively complement and enrich normal human activities, with minimal explicit concern or effort on the humans’ part.”
  4. 4. “ Understanding and managing health is complex! Throughout the last few decades of modern medicine, we have relied on clinicians for most health-related decision making. 4
  5. 5. 5 LIMITED DATA due to episodic visits TIME CONSTRAINT during clinical visits ● Significant information seeking time is required every time ● Comprehending clinical notes which contains only text is difficult Each individual is DIFFERENT and thus, personalized treatment is needed. Insufficient time and data for personalization Image Source: https://www.istockphoto.com/gb/vector/woman-doctor-examining-patient-by-stethoscope-gm541296730-96809003 WHY Healthcare? [a technology take] CHALLENGES TRADITIONAL Healthcare [a technology take]
  6. 6. “ New technologies have enabled a growing involvement of patients in their own health management. Chatbot could play a pivotal role throughout the unfolding data & knowledge-driven, AI-supported ecosystem for ENHANCED HEALTH. 6
  7. 7. 7 MULTISENSORY Sensing Semantic-Cognitive-Perceptual Computing COGNITIVE UNDERPINNING & EXPLAINABILITY with Domain Model & Protocols CONTEXTUALIZATION PERSONALIZATION ABSTRACTION AUGMENTED Personalized Health Self-monitoring, Self-appraisal, Self-management, Intervention, Prediction of disease progression and planning Towards SMART Chatbots for ENHANCED Health Domain & User-specific Knowledge Graphs Natural Language Processing Machine with Deep Learning
  8. 8. Use Cases & Prototypes Experience & observations based on ongoing collaborative healthcare projects @ KNO.E.SIS 8
  9. 9. Health Related Studies at KNO.E.SIS [Overview] HealthChallenges (Also Dementia, Obesity, Parkinson’s, Liver Cirrhosis, ADHF) Public Policy/ Population Epidemiology Personalized Health PCS + EMR + Multimodal (Speech + Image) kHealth Asthma in Children Bariatric Surgery Nutrition Physical(IoT)/Cyber/ Social (PCS)+ EMR Marijuana Social Drug Abuse Social Mental Health Depression & Suicide Social + Public + EMR Health Knowledge Graph Services Social + Clinical Data ...and infrastructure technologies: Context-aware KR (SP), KG development, Smart Data from PCS Big Data, Twitris 9
  10. 10. 10 HCI: Mobile Applications & Chatbots @ KNO.E.SIS kHealth Asthma kHealth Bariatrics Depression Active (Subset) Healthcare Projects @ KNO.E.SIS with mApps/chatbot kHealth Framework: a knowledge-enabled semantic platform that captures the data and analyzes it to produce actionable information. 3 Chatbots (Alpha Stage) 1. NOURICH: A Google Assistant based Conversational Nutrition Management System 1. Knowledge-enabled (kHealth) Personalized ChatBot for Asthma: Contextualized & Personalized Conversations involving Multimodal data (IoT & Devices) 1. ReaCTrack: Personalized Adverse Reaction Conversation-based Tracker for Clinical Depression 3 Applications 1. NOURICH: Food image-recognition app. 2. kHealth Asthma (patient evals) 3. kHealth Bariatrics (patient evals)
  11. 11. 11 Physical-Cyber-Social (PCS) Data Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1), peak expiratory flow (PEF), indoor temperature, indoor humidity, particulate matter, volatile organic compound, carbon dioxide, air quality index, pollen level, outdoor temperature, outdoor humidity, number of steps, heart rate and number of hours of sleep. Also clinical notes. kHealth Asthma Nutrition Depression Active Healthcare Projects in Kno.e.sis (Subset) Modality of Data kHealth Bariatrics For monitoring asthma control and predict vulnerability Pre and Post Surgery monitoring and self adherence Mobile app Q/A (tablet), weighing scale, pill bottle sensor, water bottle sensor for reminder to drink water, number of steps, heart rate and number of hours of sleep. Also clinical notes. Q/A, diet, images, food profile, nutrition knowledge base, user knowledge graph For nutrition tracking and diet monitoring Modeling Social Behavior for Healthcare Utilization in Depression Q/A, social media profile (Twitter, Reddit)
  12. 12. Multisensory Sensing & Multimodal Data Text, Speech, Image Processing Framework for “natural” communication 12
  13. 13. 13 Figure: An illustration of how a basic chatbot can be extended with multimodal data and input Multisensory Sensing Framework
  14. 14. 14 Use Case: kHealth Asthma Many Sources of Highly Diverse Data (& collection methods: Active + Passive): Up to 1852 data points/ patient /day http://bit.ly/kHealth-Asthma kBot with screen interface for conversation Images Text Speech ★ Episodic to Continuous Monitoring ★ Clinician-centric to Patient-centric ★ Clinician controlled to Patient-empowered ★ Disease Focused to Wellness-focused ★ Sparse data to Multimodal Big Data *(Asthma-Obesity)
  15. 15. Semantic-Cognitive-Perceptual Computing Knowledge-Infused AI with Contextualization (Knowledge Graphs), Personalization, Abstraction 15
  16. 16. 16 Semantic Browsing Extraction Data Integration and Interlinking Entity Complex Extraction Aberrant Drug-related Behaviour Neuro-Cognitive Symptoms Adverse Drug Reaction Relatio n Event Severity Personal Sensor Data De-identified EMR Blog Post Context Representation Relevant Subgraph Selection Semantic Search Disease-specific Chatbot Visualization Health Knowledge Graph Intent Open Health Knowledge Graph
  17. 17. 17 SOCIAL -MEDIA TEXT (July 12,2016) EVENT-SPECIFIC SCHEMA-BASED KNOWLEDGE
  18. 18. 18 Application: Evolving Patient Knowledge Graph (PKG) Figure: A healthcare assistant bot interacts with the patient via various conversational interfaces (voice, text, and visual) to acquire and disseminate information, and provide recommendation (validated by physician). The core functionalities of the chatbot (Component C boxed in blue) are extended with a background HKG (Component A boxed in green) and a evolving PKG (Component B boxed in orange). ★ Smarter & engaging agent ★ Minimize active sensing (Questions to be asked) ★ Ask only informed & intelligent questions ★ Relevant & Contextualized conversations ★ Personalized & Human-Like
  19. 19. Contextualization and Personalization kBOT initiates greeting conversation. Understands the patient’s health condition (allergic reaction to high ragweed pollen level) via the personalized patient’s knowledge graph generated from EMR, PGHD, and prior interactions with the kBOT. Generates predictions or recommended course of actions. Inference based on patient’s historical records and background health knowledge graph containing contextualized (domain-specific) knowledge. Figure: Example kBOT conversation which utilizes background health knowledge graph and patient’s knowledge graph to infer and generate recommendation to patients. ★ Conversing only information relevant to the patient 19 Context enabled by relevant healthcare knowledge including clinical protocols.
  20. 20. 20 Contextualization refers to data interpretation in terms of knowledge (context). Without Domain Knowledge With Domain Knowledge Chatbot with domain (drug) knowledge is potentially more natural and able to deal with variations.
  21. 21. 21 Personalization refers to future course of action by taking into account the contextual factors such as user’s health history, physical characteristics, environmental factors, activity, and lifestyle. Without Contextualized Personalization With Contextualized Personalization Chatbot with contextualized (asthma) knowledge is potentially more personalized and engaging.
  22. 22. 22 Abstraction A computational technique that maps and associates raw data to action-related information. With AbstractionWithout Abstraction .
  23. 23. 23 Smarter Chatbot with Semantically-Abstracted Information Smarterdata Data Sophistication Smart (semantically-abstracted) data should answer: ★ What causes my disease severity? ★ How well am I doing with respect to prescribed care plan? ★ Am I deviating from the care plan? I am following the care plan but my disease is not well controlled. ★ Do I need treatment adjustments? ★ How well controlled is my disease over time? Example of Abstraction
  24. 24. 24 Utkarshani Jaimini, Krishnaprasad Thirunarayan, Maninder Kalra, Revathy Venkataramanan, Dipesh Kadariya, Amit Sheth, “How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma”, JMIR Pediatr Parent 2018;1(2):e11988, DOI: 10.2196/11988.
  25. 25. 25 Semantic, Cognitive, Perceptual Computing: Paradigms That Shape Human Experience http://bit.ly/SCPComputing Humans are interested in high-level concepts (phenotypic characteristics). Semantic Computing: Assign labels and associate meanings (representation & contextualization). Cognitive Computing: Interpretation of data with respect to perspectives, constraints, domain knowledge, and personal context. Perceptual Computing: A cyclical process of semantic-cognitive computing for higher level of perception and reasoning (abstraction & action).
  26. 26. 26 Use Case: NOURICH (diet management assistant) A sample video demo of NOURICH: https://www.youtube.com/watch?v=b2OgFuEAik4
  27. 27. 27 NOURICH An Android app to support food image recognition
  28. 28. 28 Scenario: NOURICH (diet management chatbot) Figure: Architectural process of NOURICH (http://bit.ly/NOURICH) Scenario User Age: 49 Height: 5 ft Weight: 120 lbs Diet Plan: Ketogenic Food Allergies: Peanuts Diet/Recipe/Article Recommendation System with Semantic-Cognitive-Perceptual Computing framework (a) Using domain knowledge, the system searches for and filters articles related to ketogenic diet. (a) Using personalized knowledge graph, the system understands the user is allergic to peanuts. (a) Combining ● domain and user KGs ● the concept allergy <-> avoid (rule embedded in the ontology, beyond keywords-matching) and ● diet, calorie constraints, and gender profile The system will be able to interpret and will not recommend keto-recipes that have peanuts
  29. 29. Augmented Personalized Health (APH) Self-monitoring, Self-appraisal, Self-management, Intervention, Disease Progression and Tracking 29
  30. 30. Knowledge-Infused Learning with Semantic, Cognitive, Perceptual Computing Framework 30 Overarching Theory Knowledge Domain (Ontology) Personalized KG Multisensory Sensing & Multimodal Data Interactions ImagesText Speech Videos IoTs Natural Language Processing, Machine with Deep Learning AUGMENTED PERSONALIZED HEALTH (APH)Modeling broader disease context, and personalized user behavior Reasoning & decision- making framework Minimize data overload, assist in making choices, appraisal, recommendations
  31. 31. 31 This not only prevent the disease, but also enhances the patient’s health BariatricsAsthma Use Cases: APH for Asthma and Bariatrics: Patient-centric drivers
  32. 32. 32 ❖ Health management is complex. ❖ Knowledge-infused learning could give use the power need to match complex requirements. ❖ Multisensory and Multimodal data interactions are essential for natural communications. ❖ Semantic-Cognitive-Perceptual Computing enables contextualization, personalization, and abstraction for Augmented Personalized Health. In Short, This research is supported by NICHD/NIH under the Grant Number: 1R01HD087132. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
  33. 33. 33 Special Thanks Hong Yung (Joey) Yip (Graduate Student)

×