Successfully reported this slideshow.

Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health

0

Share

1 of 33
1 of 33

Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health

0

Share

Download to read offline

https://sites.google.com/view/deep-dial-2019/keynotes

Understanding and managing health is complex. Throughout the last few decades of modern medicine, we have relied clinicians on most health-related decision making. New technologies have enabled a growing involvement of patients in their own health management, aided by increasing variety and amount of patient-generated health data. Augmented personalized health [http://bit.ly/k-APH, http://bit.ly/APH-HI] strategy has outlined a broad variety of patient and clinician engagement in devising an increasingly more sophisticated and powerful health management solutions - from self-monitoring, self-appraisal, self-management, intervention to the prediction of disease progression and planning. Chatbot could play a pivotal role throughout the unfolding data-driven, AI-supported ecosystem [http://bit.ly/H-Chatbot] that engages patients and clinicians in collecting data, in driving their actions, informing them of their choices, and even delivering part of the clinical care (e.g., Cognitive Behavioral Therapy (CBT) for mental health patients). Nevertheless, this will require quite a few advances in making a more intelligent technology. In this talk, we will share some experience and observations based on our ongoing collaborative projects that usually involve clinicians and patients targeting pediatric asthma management, pre-and-post bariatric surgery care regimen, depression and other mental health issues, and nutrition. Using use cases and prototypes, we will elucidate the need, support, and use of domain- and user-specific knowledge graphs, Natural Language Processing (NLP), machine learning, and conversational AI for:

- multimodal interactions including text, voice, and other media, along with the use of diverse devices and software platforms for “natural” communication

- context enabled by deep relevant medical/healthcare knowledge including clinical protocols

- personalization by collecting and using the history of the individual patient from IoT health devices, open data, and Electronic Medical Record (EMR)

- abstraction by aggregating and correlating diverse streams data to draw plausible explanation(s) based on public (cohort-level) data (for example percentage of asthmatic patient who gets symptom when exposed to certain triggers) and personal data

- smart dialogue (intent) management and response generations by causal relations and inference of association

https://sites.google.com/view/deep-dial-2019/keynotes

Understanding and managing health is complex. Throughout the last few decades of modern medicine, we have relied clinicians on most health-related decision making. New technologies have enabled a growing involvement of patients in their own health management, aided by increasing variety and amount of patient-generated health data. Augmented personalized health [http://bit.ly/k-APH, http://bit.ly/APH-HI] strategy has outlined a broad variety of patient and clinician engagement in devising an increasingly more sophisticated and powerful health management solutions - from self-monitoring, self-appraisal, self-management, intervention to the prediction of disease progression and planning. Chatbot could play a pivotal role throughout the unfolding data-driven, AI-supported ecosystem [http://bit.ly/H-Chatbot] that engages patients and clinicians in collecting data, in driving their actions, informing them of their choices, and even delivering part of the clinical care (e.g., Cognitive Behavioral Therapy (CBT) for mental health patients). Nevertheless, this will require quite a few advances in making a more intelligent technology. In this talk, we will share some experience and observations based on our ongoing collaborative projects that usually involve clinicians and patients targeting pediatric asthma management, pre-and-post bariatric surgery care regimen, depression and other mental health issues, and nutrition. Using use cases and prototypes, we will elucidate the need, support, and use of domain- and user-specific knowledge graphs, Natural Language Processing (NLP), machine learning, and conversational AI for:

- multimodal interactions including text, voice, and other media, along with the use of diverse devices and software platforms for “natural” communication

- context enabled by deep relevant medical/healthcare knowledge including clinical protocols

- personalization by collecting and using the history of the individual patient from IoT health devices, open data, and Electronic Medical Record (EMR)

- abstraction by aggregating and correlating diverse streams data to draw plausible explanation(s) based on public (cohort-level) data (for example percentage of asthmatic patient who gets symptom when exposed to certain triggers) and personal data

- smart dialogue (intent) management and response generations by causal relations and inference of association

More Related Content

Slideshows for you

Similar to Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health

Related Books

Free with a 30 day trial from Scribd

See all

Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health

  1. 1. Towards Smart Chatbots for Enhanced Health: Using Multisensory Sensing & Semantic-Cognitive-Perceptual Computing for Augmented Personalized Health Keynote: DEEP-DIAL @ AAAI 2019, Honolulu, 27 Feb 2019 Amit Sheth LexisNexis Ohio Eminent Scholar The Ohio Center of Excellence in Knowledge-enabled Computing & BioHealth Innovations (Kno.e.sis) Wright State, USA 1 Icon source used in the entire presentation - https://thenounproject.com Presentation template by SlidesCarnival
  2. 2. 2 Figure:Avisualhistoryofchatbots Source:https://chatbotsmagazine.com/a-visual-history-of-chatbots-8bf3b31dbfb2 Chatbot 3.0 Next-Generation Smart Bots ● NATURAL communication ● MULTIMODAL interactions ● Ability to maintain the system, task, and people CONTEXTS ● PERSONALIZATION ● ABSTRACTION along DIKW Chatbot 2.0 Current Bots ● Driven by back-and-forth communication between the system & people ● Automation at the task level ● Ability to maintain both system and task contexts Chatbot 1.0 Traditional Bots ● System-driven ● Scripted-automation ● Ability to maintain only system context Evolution of CHATBOTS
  3. 3. Next-Generation Smart Bots Computing for Human Experience Promising domain: Computing for Healthcare 3 Source: http://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=1919&context=knoesis http://wiki.knoesis.org/index.php/Computing_For_Human_Experience “Computing for Human Experience will employ a suite of technologies to nondestructively and unobtrusively complement and enrich normal human activities, with minimal explicit concern or effort on the humans’ part.”
  4. 4. “ Understanding and managing health is complex! Throughout the last few decades of modern medicine, we have relied on clinicians for most health-related decision making. 4
  5. 5. 5 LIMITED DATA due to episodic visits TIME CONSTRAINT during clinical visits ● Significant information seeking time is required every time ● Comprehending clinical notes which contains only text is difficult Each individual is DIFFERENT and thus, personalized treatment is needed. Insufficient time and data for personalization Image Source: https://www.istockphoto.com/gb/vector/woman-doctor-examining-patient-by-stethoscope-gm541296730-96809003 WHY Healthcare? [a technology take] CHALLENGES TRADITIONAL Healthcare [a technology take]
  6. 6. “ New technologies have enabled a growing involvement of patients in their own health management. Chatbot could play a pivotal role throughout the unfolding data & knowledge-driven, AI-supported ecosystem for ENHANCED HEALTH. 6
  7. 7. 7 MULTISENSORY Sensing Semantic-Cognitive-Perceptual Computing COGNITIVE UNDERPINNING & EXPLAINABILITY with Domain Model & Protocols CONTEXTUALIZATION PERSONALIZATION ABSTRACTION AUGMENTED Personalized Health Self-monitoring, Self-appraisal, Self-management, Intervention, Prediction of disease progression and planning Towards SMART Chatbots for ENHANCED Health Domain & User-specific Knowledge Graphs Natural Language Processing Machine with Deep Learning
  8. 8. Use Cases & Prototypes Experience & observations based on ongoing collaborative healthcare projects @ KNO.E.SIS 8
  9. 9. Health Related Studies at KNO.E.SIS [Overview] HealthChallenges (Also Dementia, Obesity, Parkinson’s, Liver Cirrhosis, ADHF) Public Policy/ Population Epidemiology Personalized Health PCS + EMR + Multimodal (Speech + Image) kHealth Asthma in Children Bariatric Surgery Nutrition Physical(IoT)/Cyber/ Social (PCS)+ EMR Marijuana Social Drug Abuse Social Mental Health Depression & Suicide Social + Public + EMR Health Knowledge Graph Services Social + Clinical Data ...and infrastructure technologies: Context-aware KR (SP), KG development, Smart Data from PCS Big Data, Twitris 9
  10. 10. 10 HCI: Mobile Applications & Chatbots @ KNO.E.SIS kHealth Asthma kHealth Bariatrics Depression Active (Subset) Healthcare Projects @ KNO.E.SIS with mApps/chatbot kHealth Framework: a knowledge-enabled semantic platform that captures the data and analyzes it to produce actionable information. 3 Chatbots (Alpha Stage) 1. NOURICH: A Google Assistant based Conversational Nutrition Management System 1. Knowledge-enabled (kHealth) Personalized ChatBot for Asthma: Contextualized & Personalized Conversations involving Multimodal data (IoT & Devices) 1. ReaCTrack: Personalized Adverse Reaction Conversation-based Tracker for Clinical Depression 3 Applications 1. NOURICH: Food image-recognition app. 2. kHealth Asthma (patient evals) 3. kHealth Bariatrics (patient evals)
  11. 11. 11 Physical-Cyber-Social (PCS) Data Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1), peak expiratory flow (PEF), indoor temperature, indoor humidity, particulate matter, volatile organic compound, carbon dioxide, air quality index, pollen level, outdoor temperature, outdoor humidity, number of steps, heart rate and number of hours of sleep. Also clinical notes. kHealth Asthma Nutrition Depression Active Healthcare Projects in Kno.e.sis (Subset) Modality of Data kHealth Bariatrics For monitoring asthma control and predict vulnerability Pre and Post Surgery monitoring and self adherence Mobile app Q/A (tablet), weighing scale, pill bottle sensor, water bottle sensor for reminder to drink water, number of steps, heart rate and number of hours of sleep. Also clinical notes. Q/A, diet, images, food profile, nutrition knowledge base, user knowledge graph For nutrition tracking and diet monitoring Modeling Social Behavior for Healthcare Utilization in Depression Q/A, social media profile (Twitter, Reddit)
  12. 12. Multisensory Sensing & Multimodal Data Text, Speech, Image Processing Framework for “natural” communication 12
  13. 13. 13 Figure: An illustration of how a basic chatbot can be extended with multimodal data and input Multisensory Sensing Framework
  14. 14. 14 Use Case: kHealth Asthma Many Sources of Highly Diverse Data (& collection methods: Active + Passive): Up to 1852 data points/ patient /day http://bit.ly/kHealth-Asthma kBot with screen interface for conversation Images Text Speech ★ Episodic to Continuous Monitoring ★ Clinician-centric to Patient-centric ★ Clinician controlled to Patient-empowered ★ Disease Focused to Wellness-focused ★ Sparse data to Multimodal Big Data *(Asthma-Obesity)
  15. 15. Semantic-Cognitive-Perceptual Computing Knowledge-Infused AI with Contextualization (Knowledge Graphs), Personalization, Abstraction 15
  16. 16. 16 Semantic Browsing Extraction Data Integration and Interlinking Entity Complex Extraction Aberrant Drug-related Behaviour Neuro-Cognitive Symptoms Adverse Drug Reaction Relatio n Event Severity Personal Sensor Data De-identified EMR Blog Post Context Representation Relevant Subgraph Selection Semantic Search Disease-specific Chatbot Visualization Health Knowledge Graph Intent Open Health Knowledge Graph
  17. 17. 17 SOCIAL -MEDIA TEXT (July 12,2016) EVENT-SPECIFIC SCHEMA-BASED KNOWLEDGE
  18. 18. 18 Application: Evolving Patient Knowledge Graph (PKG) Figure: A healthcare assistant bot interacts with the patient via various conversational interfaces (voice, text, and visual) to acquire and disseminate information, and provide recommendation (validated by physician). The core functionalities of the chatbot (Component C boxed in blue) are extended with a background HKG (Component A boxed in green) and a evolving PKG (Component B boxed in orange). ★ Smarter & engaging agent ★ Minimize active sensing (Questions to be asked) ★ Ask only informed & intelligent questions ★ Relevant & Contextualized conversations ★ Personalized & Human-Like
  19. 19. Contextualization and Personalization kBOT initiates greeting conversation. Understands the patient’s health condition (allergic reaction to high ragweed pollen level) via the personalized patient’s knowledge graph generated from EMR, PGHD, and prior interactions with the kBOT. Generates predictions or recommended course of actions. Inference based on patient’s historical records and background health knowledge graph containing contextualized (domain-specific) knowledge. Figure: Example kBOT conversation which utilizes background health knowledge graph and patient’s knowledge graph to infer and generate recommendation to patients. ★ Conversing only information relevant to the patient 19 Context enabled by relevant healthcare knowledge including clinical protocols.
  20. 20. 20 Contextualization refers to data interpretation in terms of knowledge (context). Without Domain Knowledge With Domain Knowledge Chatbot with domain (drug) knowledge is potentially more natural and able to deal with variations.
  21. 21. 21 Personalization refers to future course of action by taking into account the contextual factors such as user’s health history, physical characteristics, environmental factors, activity, and lifestyle. Without Contextualized Personalization With Contextualized Personalization Chatbot with contextualized (asthma) knowledge is potentially more personalized and engaging.
  22. 22. 22 Abstraction A computational technique that maps and associates raw data to action-related information. With AbstractionWithout Abstraction .
  23. 23. 23 Smarter Chatbot with Semantically-Abstracted Information Smarterdata Data Sophistication Smart (semantically-abstracted) data should answer: ★ What causes my disease severity? ★ How well am I doing with respect to prescribed care plan? ★ Am I deviating from the care plan? I am following the care plan but my disease is not well controlled. ★ Do I need treatment adjustments? ★ How well controlled is my disease over time? Example of Abstraction
  24. 24. 24 Utkarshani Jaimini, Krishnaprasad Thirunarayan, Maninder Kalra, Revathy Venkataramanan, Dipesh Kadariya, Amit Sheth, “How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma”, JMIR Pediatr Parent 2018;1(2):e11988, DOI: 10.2196/11988.
  25. 25. 25 Semantic, Cognitive, Perceptual Computing: Paradigms That Shape Human Experience http://bit.ly/SCPComputing Humans are interested in high-level concepts (phenotypic characteristics). Semantic Computing: Assign labels and associate meanings (representation & contextualization). Cognitive Computing: Interpretation of data with respect to perspectives, constraints, domain knowledge, and personal context. Perceptual Computing: A cyclical process of semantic-cognitive computing for higher level of perception and reasoning (abstraction & action).
  26. 26. 26 Use Case: NOURICH (diet management assistant) A sample video demo of NOURICH: https://www.youtube.com/watch?v=b2OgFuEAik4
  27. 27. 27 NOURICH An Android app to support food image recognition
  28. 28. 28 Scenario: NOURICH (diet management chatbot) Figure: Architectural process of NOURICH (http://bit.ly/NOURICH) Scenario User Age: 49 Height: 5 ft Weight: 120 lbs Diet Plan: Ketogenic Food Allergies: Peanuts Diet/Recipe/Article Recommendation System with Semantic-Cognitive-Perceptual Computing framework (a) Using domain knowledge, the system searches for and filters articles related to ketogenic diet. (a) Using personalized knowledge graph, the system understands the user is allergic to peanuts. (a) Combining ● domain and user KGs ● the concept allergy <-> avoid (rule embedded in the ontology, beyond keywords-matching) and ● diet, calorie constraints, and gender profile The system will be able to interpret and will not recommend keto-recipes that have peanuts
  29. 29. Augmented Personalized Health (APH) Self-monitoring, Self-appraisal, Self-management, Intervention, Disease Progression and Tracking 29
  30. 30. Knowledge-Infused Learning with Semantic, Cognitive, Perceptual Computing Framework 30 Overarching Theory Knowledge Domain (Ontology) Personalized KG Multisensory Sensing & Multimodal Data Interactions ImagesText Speech Videos IoTs Natural Language Processing, Machine with Deep Learning AUGMENTED PERSONALIZED HEALTH (APH)Modeling broader disease context, and personalized user behavior Reasoning & decision- making framework Minimize data overload, assist in making choices, appraisal, recommendations
  31. 31. 31 This not only prevent the disease, but also enhances the patient’s health BariatricsAsthma Use Cases: APH for Asthma and Bariatrics: Patient-centric drivers
  32. 32. 32 ❖ Health management is complex. ❖ Knowledge-infused learning could give use the power need to match complex requirements. ❖ Multisensory and Multimodal data interactions are essential for natural communications. ❖ Semantic-Cognitive-Perceptual Computing enables contextualization, personalization, and abstraction for Augmented Personalized Health. In Short, This research is supported by NICHD/NIH under the Grant Number: 1R01HD087132. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
  33. 33. 33 Special Thanks Hong Yung (Joey) Yip (Graduate Student)

×