Collaborative Assistant/Virtual Assistant/ Conversational AI agents at AIISC

The document discusses collaborative assistants being developed at AIISC focused on health and education. Examples include a COVID-19 symptom checker, assistants for pediatric asthma, nutrition, mental health, autism, and more. Complex assistants for health domains require context using deep domain knowledge, personalization with personalized health graphs, sensing data from multiple modalities like text, voice and images. The virtual health assistant for asthma aims to do self-monitoring, self-appraisal, management, and disease tracking using sensors, social media, clinical data and knowledge graphs to have informed intelligent conversations. Developing such assistants is quite challenging due to big data issues, need for domain context, personalization, explainability, and ease of use.

Collaborative Assistant/Virtual Assistant/
Conversational AI agents at AIISC
Panel @ Collaborative Assistants for the Society (CASY 2020)
Amit Sheth
Director, Artificial Intelligence Institute of UofSC (#AIISC, aiisc.ai)

 Focus on two domains: Health & Education
 Broad variety: Simple chatbot (flexible interactions) to quite complex (manage
chronic disease)
 Uniqueness of complex examples: a. Context with deep domain knowledge, b.
Personalization with personalized health knowledge graph, c. active/passive
sensing, d. multimodal (text, voice, image)
Examples – at various stages of development
 Health-e Gamecock COVID-10 daily symptom checker
 Pediatric Asthma
 Nutrition
 Mental Health
 Autism, After Cancer Exercise, Diabetes, and more in planning.
http://wiki.aiisc.ai/index.php/Covid19

1. Self
Monitoring
2. Self
Appraisal
3. Self
Management
4.
Intervention
5. Disease
Progression
and
Tracking
Virtual Health Assistant for Augmented
Personalized Health: example of Asthma
• Complex- multifactorial disease, personalized
• Access- challenges in working with patients
(IRB, privacy, medical/clinical knowledge and
lack of gold standards, etc.

4
Sensor, Social, Clinical Datastreams: Informed &
Intelligent Questions
Weather information
(temperature, pollen,
humidity, etc)
Elasticsearch (ES)
Database Query & Rule Abstract raw values
into information
Asthma Domain Knowledge
https://bioportal.bioontology.org/ontologies/AO
http://www.childhealthservicemodels.eu
Patient Data from EMR & PGHD
(Compliance score, prescribed
medications, asthma control level)
IoTs (Foobot & Fitbit)
Conversation Rules & Scripts
(DialogFlow)
Sensor,Social,Clinical Architectural Framework for Intelligent & Informed Conversations: kBOT Asthma
Sensor (IoTs) & Cyber
Datastream
Clinical (Baseline) Datastream
Patient Consented Social Data
(Facebook, Instagram, Twitter
Activity)
Social Datastream
Knowledge Datastream
★ Smarter & engaging agent
★ Minimize active sensing
(Questions to be asked)
★ Ask only informed & intelligent questions
★ Relevant & Contextualized conversations
★ Personalized & Human-Like
Human-Like Aspect

Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition (allergic reaction to high
ragweed pollen level) via the
personalized patient’s knowledge
graph generated from EMR, PGHD, and
prior interactions with the kBOT.
Generates predictions or
recommended course of actions.
Inference based on patient’s historical
records and background health
knowledge graph containing
contextualized (domain-specific)
knowledge.
Figure: Example kBOT conversation which
utilizes background health knowledge graph
and patient’s knowledge graph to infer and
generate recommendation to patients.
★ Conversing only information relevant to
the patient
5

“Huge need in several fields – health and education among the important areas.
• Examples from health domain: not enough available clinical expertise to meet mental
health needs; chronic disease need continuous care; massive growth in patient generate
data with demonstrated value for making decisions on health, wellness and fitness.
But,
Quite challenging: big data challenges – esp variety of data, need for context to interpret
data (and relevant deep domain knowledge), personalization for each patient, need to be
able to explain results before clinicians will take the technology seriously, ease of use/UX,
incentives in validation studies (patients, clinicians, insurance companies), etc.

https://sites.google.com/view/deep-dial-2019/keynotes Understanding and managing health is complex. Throughout the last few decades of modern medicine, we have relied clinicians on most health-related decision making. New technologies have enabled a growing involvement of patients in their own health management, aided by increasing variety and amount of patient-generated health data. Augmented personalized health [http://bit.ly/k-APH, http://bit.ly/APH-HI] strategy has outlined a broad variety of patient and clinician engagement in devising an increasingly more sophisticated and powerful health management solutions - from self-monitoring, self-appraisal, self-management, intervention to the prediction of disease progression and planning. Chatbot could play a pivotal role throughout the unfolding data-driven, AI-supported ecosystem [http://bit.ly/H-Chatbot] that engages patients and clinicians in collecting data, in driving their actions, informing them of their choices, and even delivering part of the clinical care (e.g., Cognitive Behavioral Therapy (CBT) for mental health patients). Nevertheless, this will require quite a few advances in making a more intelligent technology. In this talk, we will share some experience and observations based on our ongoing collaborative projects that usually involve clinicians and patients targeting pediatric asthma management, pre-and-post bariatric surgery care regimen, depression and other mental health issues, and nutrition. Using use cases and prototypes, we will elucidate the need, support, and use of domain- and user-specific knowledge graphs, Natural Language Processing (NLP), machine learning, and conversational AI for: - multimodal interactions including text, voice, and other media, along with the use of diverse devices and software platforms for “natural” communication - context enabled by deep relevant medical/healthcare knowledge including clinical protocols - personalization by collecting and using the history of the individual patient from IoT health devices, open data, and Electronic Medical Record (EMR) - abstraction by aggregating and correlating diverse streams data to draw plausible explanation(s) based on public (cohort-level) data (for example percentage of asthmatic patient who gets symptom when exposed to certain triggers) and personal data - smart dialogue (intent) management and response generations by causal relations and inference of association

Knowledge-enhanced Learning @ Kno.e.sis

Waterloo Adv Bd Dec 00brighteyes

Healthcare over Internet Protocol, Web 2.0, Health 2.0 and the Personal Healt...

Rowan Purdy

Augmented Personalized Health: using AI techniques on semantically integrated...

The National Collaborating Centre for Methods and Tools

Keynote @ 2018 AAAI Joint Workshop on Health Intelligence (W3PHIAI 2018), 2 February 2018, New Orleans, LA [Video: https://youtu.be/GujvoWRa0O8] Related article: https://ieeexplore.ieee.org/document/8355891/ Abstract Healthcare as we know it is in the process of going through a massive change - from episodic to continuous, from disease-focused to wellness and quality of life focused, from clinic centric to anywhere a patient is, from clinician controlled to patient empowered, and from being driven by limited data to 360-degree, multimodal personal-public-population physical-cyber-social big data-driven. While the ability to create and capture data is already here, the upcoming innovations will be in converting this big data into smart data through contextual and personalized processing such that patients and clinicians can make better decisions and take timely actions for augmented personalized health. In this talk, we will discuss how use of AI techniques on semantically integrated patient-generated health data (PGHD), environmental data, clinical data, and public social data is exploited to achieve a range of augmented health management strategies that include self-monitoring, self-appraisal, self-management, intervention, and Disease Progression Tracking and Prediction. We will review examples and outcomes from a number of applications, some involving patient evaluations, including asthma in children, bariatric surgery/obesity, mental health/depression, that are part of the Kno.e.sis kHealth personalized digital health initiative. Background: Background: http://bit.ly/k-APH, http://bit.ly/kAsthma, http://j.mp/PARCtalk

Sun==big data analytics for health care

Aravindharamanan S

Abstract Kno.e.sis (http://knoesis.org) is a world-class research center that uses semantic, cognitive, and perceptual computing for gathering insights from physical/IoT, cyber/Web, and social and enterprise (e.g., clinical) big data. We innovate and employ semantic web, machine learning, NLP/IR, data mining, network science and highly scalable computing techniques. Our highly interdisciplinary research impacts health and clinical applications, biomedical and translational research, epidemiology, cognitive science, social good, policy, development, etc. A majority of our $12+ million in active funds come from the NSF and NIH. In this talk, I will provide an overview of some of our major research projects. Kno.e.sis is highly successful in its primary mission of exceptional student outcomes: our students have exceptional publication and real-world impact and our PhDs compete with their counterparts from top 10 schools for initial jobs in research universities, top industry research labs, and highly competitive companies. A key reason for Kno.e.sis' success is its unique work culture involving teamwork to solve complex problems. Practically all our work involves real-world challenges, real-world data, interdisciplinary collaborators, path-breaking research to solve challenges, real-world deployments, real-world use, and measurable real-world impact. In this talk, I will also seek to discuss our choice of research topics and our unique ecosystem that prepares our students for exceptional careers.

Population health informatics DME SoPH 20151006Kathleen Gray

NCCMT Spotlight Webinar: Clear Communication Index

Interested in evidence-based criteria for clear communication of public health material? Do you develop and review public health material? Are you interested in enhancing the clarity and ease of understanding of these materials? The Clear Communication Index can help! The U.S. Centers for Disease Control and Prevention developed the Index. How can the Clear Communication Index help you? The Index helps professionals develop and review public health communication materials for clarity, encourages collaboration between writers and reviewers to ensure accuracy, and assesses ease of use of communication materials. The tool consists of criteria that enhance clarity and understanding. Use of the Index yields a quantitative score based on assessment of seven areas: main message and call to action, language, information design, state of the science, behavioural recommendations, numbers, and risk. A widget is available for placement on webpages. This webinar includes an overview of the Index by its developers, followed by a presentation from the National Resource Center for Lupus in the Lupus Foundation of America. To see the summary statement of this method developed by NCCMT, click here: http://www.nccmt.ca/resources/search/247 The National Collaborating Centre for Methods and Tools is funded by the Public Health Agency of Canada and affiliated with McMaster University. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada. NCCMT is one of six National Collaborating Centres (NCCs) for Public Health. The Centres promote and improve the use of scientific research and other knowledge to strengthen public health practices and policies in Canada.

Tackling societal challenges through digital transformation

Games for Health Europe

Artificial Intelligence and ChatGPT: Impacts and Challenges for Medical Educa...

tbirdcymru

Introduction to Health Informatics and Health IT (Part 1) (February 10, 2021)

Nawanan Theera-Ampornpunt

Exploiting Multimodal Information for Machine Intelligence and Natural Intera...

http://iwma.lnmiit.ac.in/speakers.html Third International Workshop on Multimedia Applications ( IWMA ), March 02-06, 2021. The Holy Grail of machine intelligence is the ability to mimic the human brain. In computing, we have created silos in dealing with each modality (text/language processing, speech processing, image processing, video processing, etc.). However, the human brain’s cognitive and perceptual capability to seamlessly consume (listen and see) and communicate (writing/typing, voice, gesture) multimodal (text, image, video, etc.) information challenges machine intelligence research. Emerging chatbots for demanding health applications present the requirements for these capabilities. To support the corresponding data analysis and reasoning needs, we have explored a pedagogical framework consisting of semantic computing, cognitive computing, and perceptual computing. In particular, we have been motivated by the brain’s amazing perceptive power that abstracts massive amounts of multimodal data by filtering and processing them into a few concepts (representable by a few bits) to act upon. From the information processing perspective, this requires moving from syntactic and semantic big data processing to actionable information that can be weaved naturally into human activities and experience. Exploration of the above research agenda, including powerful use cases, is afforded in a growing number of emerging technologies and their applications - such as chatbots and robotics for healthcare. In this talk, I will provide these examples and share the early progress we have made towards building health chatbots that consume contextually relevant multimodal data and support different forms/modalities of interactions to achieve various alternatives for digital health. I will also demonstrate the strong role of domain knowledge and personalization using domain and personalized knowledge graphs as part of various reasoning and learning techniques.

Healthcare innovations at Kno.e.sis sept2016

HEC 2016 Panel: Putting User-Generated Data in Action: Improving Interpretabi...

Pei-Yun Sabrina Hsueh

Chair/Moderator: Pei-Yun Sabrina HSUEH, PhD (IBM T.J. Watson Research Center) Panelists: XinXin ZHU, Bian YANG, Ying-Kuen CHEUNG , Thomas WETTER, and Sanjoy DEY a IBM T.J. Watson Research Center, USA b Norwegian University of Science and Technology, Norway c Mailman School of Public health, Columbia University, USA d, Department of Biomedical Informatics, University of Washington, USA e Department of Medical Informatics, University of Heidelberg, Germany The rise of consumer health awareness and the recent advent of personal health management tools (including mobile and health wearable devices) have contributed to another shift transforming the healthcare landscape. Despite the rise of health consumers, the impact of user-generated health data remains to be validated. In fact, many applications are hinged on the interpretability issues of this sort of data. The aim of this panel is two-fold. First, this panel aims to review the key dimensions in the interpretability, spanning from quality and reliability to information security and trust management. Secondly, since similar issues and methodologies have been proposed in different application areas ranging from clinical decision support to behavioral interventions and clinical trials, the panelists will also discuss both the success stories and the areas that fall short. The opportunities and barriers identified can then serve as guidelines or action items individuals can bring to their organizations to further improve the interpretability of user-generated data.

Improving health care outcomes with responsible data science

k-BOT: Knowledge-driven Chatbot for Health @ CASY2020

Professor Jim McManus AFBPsS,FFPH,CSci, FRSB, CPsychol

Mental Health at Work during and beyond the Pandemic

ON EXPLOITING MULTIMODAL INFORMATION FOR MACHINE INTELLIGENCE AND NATURAL IN...

Keynote: SECOND INTERNATIONAL WORKSHOP IN MULTIMEDIA PRAGMATICSMMPrag 2019, San Jose, California, 28-30 March 2019 http://mipr.sigappfr.org/19/keynote-speakers/ The Holy Grail of machine intelligence is the ability to mimic the human brain. In computing, we have created silos in dealing with each modality (text/language processing, speech processing,image processing, video processing, etc.). However, the human brain’s cognitive and perceptual capability to seamlessly consume (listen and see) and communicate (writing/typing, voice, gesture) multimodal (text, image, video, etc.) information challenges the machine intelligence research. Emerging chatbots for demanding health applications present the requirements for these capabilities. To support the corresponding data analysis and reasoning needs, we have to explore a pedagogical framework consisting of semantic computing, cognitive computing, and perceptual computing (http://bit.ly/w-SCP). In particular, we have been motivated by the brain’s amazing perceptive power that abstracts massive amounts of multimodal data by filtering and processing them into a few concepts (representable by a few bits) to act upon. From the information processing perspective, this requires moving from syntactic and semantic big data processing to actionable information that can be weaved naturally into human activities and experience (http://bit.ly/w-CHE). Exploration of the above research agenda, including powerful use cases, is afforded in a growing number of emerging technologies and their applications - such as chatbots and robotics. In this talk, I will provide these examples and share the early progress we have made towards building health chatbots (http://bit.ly/H-Chatbot) that consume contextually relevant multimodal data and support different forms/modalities of interactions to achieve various alternatives for digital health (http://bit.ly/k-APH). I will also discuss the indispensable role of domain knowledge and personalization using domain and personalized knowledge graphs as part of various reasoning and learning techniques.

The shared value of personal and population data

Health Informatics New Zealand

MedInfo 2010 Report

Running head MANUAL FOR SURGICAL FACE MASKS .docx

jeanettehully

Running head: MANUAL FOR SURGICAL FACE MASKS 1 MANUAL FOR SURGICAL FACE MASKS 5 Production of Disposable Surgical and Medical Face Masks Student’s Name Institutional Affiliation Production of Disposable Surgical and Medical Face Masks The process that needs better explanation and instruction Surgical and medical face masks are worn by healthcare personnel and professionals when attending to patients with either bacterial or viral infections to avoid contact that may lead to them being infected. In times of an outbreak of a highly infectious viral and bacterial disease like Covid-19, health practitioners and even the public might require to wear medical face masks to prevent the spread of the virus or bacteria (Centers for Disease Control and Prevention, 2019). However, the process of producing the masks is not clear, and that explains the reason surgical and medical face masks are only limited to use by health professionals. The medical masks are highly effective in preventing the spread of viral diseases, and in case of an outbreak, the public should have adequate access to these masks. The process of how they are made needs a clear explanation so that cloth manufacturers and tailors can embark on mass production when necessary. Impact of the process on manufacturers of face masks Manufacturers of medical face masks are limited in number. The reason is due to the lack of an elaborate procedure on how to manufacture masks that meet the required standards. Many of them know how to make cloth face masks but which are not effective in controlling the spread of viruses and bacteria like the surgical masks. Outlining a manual for making surgical masks will help manufacturers and tailors to embark on mass production of the same. Such an act will reduce the shortage of surgical face masks and make them available not only to the medical personnel but also to the public. The research to be conducted The student should embark on a study to understand how surgical face masks are made. What special features do they possess compared to cloth, non-medical face masks? The student should clearly outline the process involved, the materials required, the average duration taken to produce a single piece, as well as any other relevant resources for the procedure. It should be precise and elaborate such that any interested manufacturer can embark on the production of the masks without struggling on the process. Existing documentation and information about the process Much of the available information on how to manufacture masks focuses on non-surgical cloth masks. However, despite cloth masks and surgical masks being closely related, the procedure for making medical masks is more complicated than cloth masks. That is because surgical masks are more efficient in disease protection than cloth masks. Making cloth mas ...

Role of AI in Transforming the Healthcare Industry

HammadAfzal23

Best practices in HCSMnwenderlich

ICH Guidelines for Pharmacovigilance.pdf

NEHA GUPTA

The "ICH Guidelines for Pharmacovigilance" PDF provides a comprehensive overview of the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) guidelines related to pharmacovigilance. These guidelines aim to ensure that drugs are safe and effective for patients by monitoring and assessing adverse effects, ensuring proper reporting systems, and improving risk management practices. The document is essential for professionals in the pharmaceutical industry, regulatory authorities, and healthcare providers, offering detailed procedures and standards for pharmacovigilance activities to enhance drug safety and protect public health.

Feeding plate for a newborn with Cleft Palate.pptx

SatvikaPrasad

A feeding plate is a prosthetic device used for newborns with a cleft palate to assist in feeding and improve nutrition intake. From a prosthodontic perspective, this plate acts as a barrier between the oral and nasal cavities, facilitating effective sucking and swallowing by providing a more normal anatomical structure. It helps to prevent milk from entering the nasal passage, thereby reducing the risk of aspiration and enhancing the infant's ability to feed efficiently. The feeding plate also aids in the development of the oral muscles and can contribute to better growth and weight gain. Its custom fabrication and proper fitting by a prosthodontist are crucial for ensuring comfort and functionality, as well as for minimizing potential complications. Early intervention with a feeding plate can significantly improve the quality of life for both the infant and the parents.

Similar to Collaborative Assistant/Virtual Assistant/Conversational AI agents at AIISC

ISYS90069 eHealth & biomedical informatics systems flyer

Health and Biomedical Informatics Centre @ The University of Melbourne

Precision and Participatory Medicine - MEDINFO 2015 Panel on big data

Health and Biomedical Informatics Centre @ The University of Melbourne

Kno.e.sis Approach to Impactful Research & Training for Exceptional Careers

The National Collaborating Centre for Methods and Tools

Population health informatics DME SoPH 20151006Kathleen Gray

NCCMT Spotlight Webinar: Clear Communication Index

Tackling societal challenges through digital transformation

Games for Health Europe

Artificial Intelligence and ChatGPT: Impacts and Challenges for Medical Educa...

tbirdcymru

Introduction to Health Informatics and Health IT (Part 1) (February 10, 2021)

Nawanan Theera-Ampornpunt

Exploiting Multimodal Information for Machine Intelligence and Natural Intera...

Healthcare innovations at Kno.e.sis sept2016

HEC 2016 Panel: Putting User-Generated Data in Action: Improving Interpretabi...

Pei-Yun Sabrina Hsueh

Improving health care outcomes with responsible data science

k-BOT: Knowledge-driven Chatbot for Health @ CASY2020

Professor Jim McManus AFBPsS,FFPH,CSci, FRSB, CPsychol

Mental Health at Work during and beyond the Pandemic

ON EXPLOITING MULTIMODAL INFORMATION FOR MACHINE INTELLIGENCE AND NATURAL IN...

The shared value of personal and population data