Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Augmented Personalized Health: using AI techniques on semantically integrated multimodal data for patient empowered health management strategies


Published on

Keynote @ 2018 AAAI Joint Workshop on Health Intelligence (W3PHIAI 2018), 2 February 2018, New Orleans, LA [Video:]

Related article:


Healthcare as we know it is in the process of going through a massive change - from episodic to continuous, from disease-focused to wellness and quality of life focused, from clinic centric to anywhere a patient is, from clinician controlled to patient empowered, and from being driven by limited data to 360-degree, multimodal personal-public-population physical-cyber-social big data-driven. While the ability to create and capture data is already here, the upcoming innovations will be in converting this big data into smart data through contextual and personalized processing such that patients and clinicians can make better decisions and take timely actions for augmented personalized health. In this talk, we will discuss how use of AI techniques on semantically integrated patient-generated health data (PGHD), environmental data, clinical data, and public social data is exploited to achieve a range of augmented health management strategies that include self-monitoring, self-appraisal, self-management, intervention, and Disease Progression Tracking and Prediction. We will review examples and outcomes from a number of applications, some involving patient evaluations, including asthma in children, bariatric surgery/obesity, mental health/depression, that are part of the Kno.e.sis kHealth personalized digital health initiative.

Background: Background:,,

Published in: Healthcare
  • Be the first to comment

Augmented Personalized Health: using AI techniques on semantically integrated multimodal data for patient empowered health management strategies

  1. 1. Augmented Personalized Health using AI techniques on semantically integrated multimodal data for patient empowered health management strategies 2018 AAAI Joint Workshop on Health Intelligence (W3PHIAI 2018) @ AAAI 2018 Keynote on Feb 2, 2018 Prof. Amit. P. Sheth LexisNexis Ohio Eminent Scholar, Executive Director, Kno.e.sis Wright State University Background:,, Icon source used in the entire presentation -
  2. 2. Kno.e.sis: Ohio Center of Excellence in Knowledge-enabled Computing & BioHealth Innovation NSF ● Harassment on Social Media ● Citizen & Physical Sensing ● Twitris - Collective Intelligence ● Aerial Surveillance ● Visual Experience ● Web Robot Traffic NIH ● kHealth - Asthma ● eDrugTrends ● eDarkTrends ● Depression on Social Media ● Drug Abuse Early Warning DoD & Industry • Metabolomics & Proteomics • Medical Info Decisions • Human Detection on Synthetic FMV • Sensor & Information • Material Genomics • Cardiology Semantic Analysis 15 faculty from 4 colleges + ~60 Funded Students • 40 PhD • 16 MS • 10 BS Kno.e.sis conducts research in AI techniques that convert physical- cyber-social big data into smart data, enabling building of intelligent systems for clinical, biomedical, policy, and epidemiological applications. Example clinical/healthcare applications include major diseases such as asthma, obesity, depression, cardiology, dementia and GI. This is complemented by social and development challenges such as marijuana legalization policy, harassment on social media, gender-based violence, and disaster coordination. Kno.e.sis’ research in World Wide Web ranks Wright State University among the top 10 organizations in the world based on 10-yr impact [MAS: 2016]. Its total budget for currently active projects is $13+ million [2017]. World-class interdisciplinary research is complemented by exceptional student outcomes and commercialization with local economic impact. 2
  3. 3. Active Healthcare Projects in Kno.e.sis kHealth for Asthma kHealth for Bariatrics kHealth for Dementia Depression EDrugTrends EDarkNets Alchemy for Healthcare Knowledge Graph Development Platform Contextual Knowledge Representation 3
  4. 4. HealthChallenges (Also, Dementia, Obesity, Parkinson’s, Liver Cirrhosis, ADHF ) Public Policy/ Population Epidemiology Personalized Health PCS + EMR kHealth Asthma in Children Bariatric Surgery Physical(IoT)/Cyber/ Social (PCS)+ EMR Marijuana Social Drug Abuse Social Mental Health Depression (Suicide) Social + Public + EMR Health Knowledge Graph Services Social + Clinical Data Health Related Studies at Kno.e.sis [overview] ...and infrastructure technologies: Context- aware KR (SP), KG development, Smart Data from PCS Big Data, Twitris,.... 4
  5. 5. Goal of Medicine [a revisionist view] 1) Ultimate goal of medicine: fight and prevent diseases 3) Predict: must know how one reacts in order to predict 2) How? Outcome has to be predicted in order to be prevented 5) But, each patient is different. Generalised approach is not efficient 6) Personalised data collection - Augmented Personalised Health 7) Personalised data - accurate prediction - better prevention 4) Data must be collected over a period of time to know the reaction of the patients to particular disease 5
  6. 6. Traditional Healthcare [a technology take] - Limited data due to episodic visits - Time constraints in knowing the patient during clinical visits - Significant information seeking time is required every time - Comprehending clinical notes every time which contains only text is difficult - Each individual is different and thus, personalised treatment is needed - Insufficient time and data for personalization Image Source - 6
  7. 7. Health Care: Transition Material Paternalism Therapeutic and Preventive Alliance Certain health decisions are best left in the hands of a clinician Clinician-patient interaction for effective treatment to control and prevent disease To Image source(left) - 7
  8. 8. Therapeutic and Preventive Alliance - Relationship between health care professional and the patient - Patient and clinician hope to engage in making clinical decisions for the patient - Improves patients outcome as the patient is educated about the disease Challenges - Difficult to know patients well only with clinical visits due to time constraints - Frequent interaction about the patient health with clinician is necessary Solution - Insights from Patient Generated Health Data (PGHD) enabled by Internet of Things (IoT) devices, community, clinical and public health data (Physical, Cyber Social and Clinical Data) 8
  9. 9. PGHD with Internet of Things a digital footprint representative of patient’s health ● By 2020, 40% of Internet of Things based devices will be related to health care [1] ● Monitors health indicators specific to patients disease ● Enables remote and continuous monitoring for long term chronic illness ● Multimodal data ● Integrated and standardized technology backbone enables disease surveillance and treatment support Medical Internet of Things [1] Dimitrov, Dimiter V. "Medical internet of things and big data in healthcare." Healthcare informatics research 22.3 (2016): 156-163. 9
  10. 10. Healthcare: Before and After IoT (+ mobile + social) ● Episodic to Continuous ● Clinic - centric to Patient - centric ● Clinician controlled to Patient empowered ● Disease Focused to beyond Medical intervention ● Limited data to 360 Multimodal data 10
  11. 11. Big Data • Diverse and multimodal data - Patients, Social and Clinical data • Eight weeks of data from seven sensors collected for 16 Parkinson’s patients is about 12 GB of data [kaggle] • According to expert forecasts, smart devices will produce one-tenth of the total amount of information on Earth, up to 44 ZB in 2020 [sam-solutions] 11
  12. 12. Big Data into Smart Data making sense of the data Massive amount multimodal data collected from various medical IoT sensors Turning Big Data into Insights into Actionable information Analyze the data to find out what the data tells about the patient Provide timely actionable information specific to patient disease 12 Smart data makes sense out of Big data : It provides value from harnessing the challenges posed by volume, velocity, variety and veracity of big data, in-turn providing actionable information and improve decision making.
  13. 13. Patient Generated Health Data to action Contextualization, Abstraction and Personalization SSN Ontology Interpreted Data (abductive) [OWL] e.g., diagnosis 3 Annotated Data [RDF] e.g., label Raw Data [Text] e.g., number Interpreted Data (deductive) [OWL] e.g., threshold 1 0 2 13
  14. 14. Augmented Personalised Health Big Data to Smart Data Augmented Personalised Health(APH) is a vision to enhance the healthcare by using AI techniques on semantically integrated PGHD, environmental data, clinical data, public health data & social data. Data Components PGHD, Clinical data, environmental data and Social Data Smart Data meaningful data obtained after contextualised processing Image Source - 14
  15. 15. Smart Data: Data with Knowledge Smarterdata Data Sophistication Smart data should answer: - What causes my disease severity? - How well am I doing with respect to prescribed care plan? - Am I deviating from the care plan? - I am following the care plan but my disease is not well controlled. Do I need treatment adjustments? - How well controlled is my disease over the time? 15
  16. 16. Health Management Strategies of APH 1. Self Monitoring Constant and remote monitoring of disease specific health indicators for any given patient 2. Self Appraisal Interpretation of the data collected with respect to disease context for the patient to evaluate themselves 3. Self Management Identify the deviation from normal and assist patients to get back to prescribed care plan 4. Intervention Change in the care plan - with the converted smart data by APH, provide decision support for treatment adjustments 5. Disease Progression and Tracking Longitudinal data collection and analysis to enhance patients health over the time Sheth, How will the Internet of Things enable Augmented Personalized Health? 16
  17. 17. APH Applications [@Kno.e.sis, currently involving patient evals] Asthma • 6.3 million children in USA are affected by Asthma; 300 million adults & children worldwide [CDC] • Non-adherence to medication makes it one of the poorly controlled disease • Multifactorial disease • Difficult to diagnose based on episodic visits and clinical records Bariatric Surgery ● 36% of the adults in the United States suffer from obesity [CDC] ● 65% of the world’s population lives in countries where the occurrence of death due to overweight and obesity is higher than being underweight [ASMBS] ● Bariatrics surgery is one of the efficient ways to reduce weight ● Weight recidivism: significant subset of patients regain weight due to non adherence to post surgical guidelines 17
  18. 18. APH for Asthma and Bariatrics: patient centric drivers This not only prevent the disease, but also enhances the patient’s health BariatricsAsthma 18
  19. 19. kHealth Asthma A multisensory approach for personalised asthma care in children kHealth Bariatrics A multisensory approach to support patient post surgical progress In order to enhance patient’s health care by providing answers to those questions... Also, kHealth for Dementia, (Asthma-Obesity), (ADHF), (GI-liver cirrhosis) 19
  20. 20. kHealth-Asthma 20
  21. 21. kHealth Asthma Overview Data Sources Heterogeneous data and collection method Smart Data Actionable meaningful information from the data collected 21
  22. 22. kHealthDash Knowledge-enabled Personalized DASHboard for Asthma Management 22
  23. 23. Duration Number of Patients Compliance One month 40 patients 30 patients - above 75% 3 Months 1 patient 100% Duration Number of Patients Compliance 3 Months 15 Patients 5 patients - 100% 10 patients - above 75% Completed Trials (45 patients) Total Data points - 1057 data points (42 patients) Ongoing Trial Data from Tablet Data from Tablet Sensors Data points / day For 1107 days Fitbit 8 8856 Peak Flow Meter 2 2214 Data Collection [ as of Jan 2018] 1325 days for 41 patients, average of compliance = 89% which is 1107 days 23
  24. 24. Data Collection Parameter # of data points/day Total Pollen 2 /day (Every 12 Hours) 854 PM 2.5 24/day (Hourly) 10248 Ozone 24/ day (Hourly) 10248 Temperature 24/day (Hourly) 10248 Humidity 24/day (Hourly) 10248 Outdoor Environmental data Dec 01, 2016 to present (as of Feb 2, 2018) = 427 days Parameters # data points /day Total CO2 288 (Every 5 minute, 24 x 12) 381600 (1325 days for 41 patients) Volatile Compounds Temperature Humidity PM 2.5 Global Pollution Index Indoor Environmental Parameters of Completed Trials collected using Foobot (41 patients) Total = 381600 x 6 (Parameters) = 2289600 data points and still collecting from current deployments Total - 41, 846 data points and still collecting... 24
  25. 25. Data Collection per day per patient Active sensing Tablet Symptom - 6 Short acting med - 1 Long acting med - 1 Total - 8 x 2(twice a day) = 16 Peak Flow meter = 2 (twice a day) Total = 16+2 = 18 data points/day Passive sensing Foobot CO2, VOC, Humidity, Temperature, PM2.5, Global Pollution Index Fitbit Sleep - 4 (REM,Light sleep,Deep sleep, # minutes active) Activity - 4 (minutes active, sedentary minutes, minutes lightly active, # steps) Subtotal - 8 Outdoor Parameters Ozone, PM2.5, Temperature, Humidity = 24 x 4 = 96 Pollen = 12 Subtotal = 108 Total = 1844 data points / day 288(every five minutes) x 6 = 1728 Total number of data points per patient per day = 18 + 1844 = 1862 data points/ day 25
  26. 26. 26 Abstraction Actionable Information Google Virely watch collects physiological data [] Modality of Data Measure relevant signals for studies spanning cardiovascular, movement disorders, and other areas. Examples include electrocardiogram (ECG), heart rate, electrodermal activity, and inertial movements. Cohort/ Size/ Status To be used in other clinical studies IBM and to track progression of Lung disease like COPD Modality of Data IoT device, to record symptoms and vital-signs, such as cough intensity, sputum (saliva and mucus) color, lung function, breath rate and heart rate, oxygen saturation, as well as activity. Cohort/ Size/ Status 100/ planned (2018) [] Stanford Wearable Study for early disease diagnoses (Lyme disease) Modality of Data Heart rate, skin temperature, and sleep data Cohort/ Size/ Status 60/ completed [] kHealth Active Decompensated Heart Failure (ADHF) to reduce readmission Kno.e.sis-Wexner-Ohio State U. Modality of Data Mobile app Q/A (tablet), BP monitor, scale Cohort/ Size/ Status Proof of concept/ on hold [6] kHealth Dementia Kno.e.sis-Wright State Physicians Modality of Data Mobile app Q/A (tablet), sleep sensor, number of steps, location, gait speed, gait parameters including speed and cadence. Cohort/ Size/ Status 40 (validation) + 20 (longitudinal)/ ongoing [8] Technology Integrated Health Management (TIHM) for Dementia Led by Surrey and Borders NHS Foundation Trust Modality of Data Survey questions, sensors to measure physiological symptoms (e.g. Blood pressure), physical activity, and environmental data; the target is to reduce the hospital admissions and provide more enhanced care and support. Cohort/ Size/ Status Planned 350 (with technology) + 350 (without technology)/ ongoing [] kHealth Asthma in Children for monitoring asthma control and predict vulnerability (in future, self management) Kno.e.sis-Dayton Children’s Hospital Modality of Data Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1), peak expiratory flow (PEF), indoor temperature, indoor humidity, particulate matter, volatile organic compound, carbon dioxide, air quality index, pollen level, outdoor temperature, outdoor humidity, number of steps, heart rate and number of hours of sleep. Also clinical notes. Cohort/ Size/ Status 200/ ongoing (~50% complete) kHealth Bariatrics Pre and Post Surgery monitoring and self adherence Kno.e.sis-Wright State Physicians Modality of Data Mobile app Q/A (tablet), weighing scale, pill bottle sensor, water bottle sensor for reminder to drink water, number of steps, heart rate and number of hours of sleep. Also clinical notes. Cohort/ Size/ Status TBD/ ongoing [5] Example efforts involving PGHD and other health relevant data Data
  27. 27. Anecdotal Evidence using kHealthDash 27
  28. 28. Anecdotal Evidence using kHealthDash Factors Frequency Probability Pollen 7/7 1 Ozone 6/7 0.857 PM2.5 1/7 0.14 Symptoms - 6, Medications - 3 Total events - 7 (2 medications have been taken on the same day as symptom occurrence) Patient gets symptom when the pollen is high and there is 87% and 10% chance that the patient will get symptom when ozone and PM2.5 are high respectively. To account for multiple factors, we will be using Regression in the future. 28
  29. 29. 29 Computing Predictors Medications Activity Temperature Humidity Pollen Air Quality Spirometry Outdoor, Indoor & Medical (Predictors) Logistic Regression Model [A x1+ B x2+ C x3…..] Weights Computed Cough Cough Symptoms Outcome
  30. 30. Patient Health Score How controlled is my asthma? How vulnerable am I today? Patient Generated Health Data (PGHD) Population Signals (Environmental Parameters) Public Signals (Social Media Data) Risk Assessment Model Domain Knowledge Well Controlled Moderately Controlled Poorly Controlled 30
  31. 31. Bariatrics Obesity • 65% of the world’s population lives in countries where the occurrence of death due to overweight and obesity is higher than being underweight Problem ● Chances of regaining weight as stomach can still expand after surgery ● Continuous monitoring of the patients by the surgeon is very essential Challenges Post Bariatric Surgery ● Patient acceptance and active participation involving continuous monitoring of the patient ● Cost and reimbursement models ● Challenging research in understanding of variety of data over long period 31
  32. 32. 32 A system capable of: ● remotely and continuously monitor patients ● identify non-compliance before and after surgery ● nudge/assist for better compliance for improved outcomes and reduce recidivism Solution: kHealth Bariatrics
  33. 33. kHealth Bariatrics Pill Bottle Sensor Reminds patients to take their pills and records it Fitbit (Activity, Sleep and Heart Rate) kHealth Bariatrics App Diet and Emotional well being through contextual questions Water Bottle Sensor Reminds patients to hydrate and records it Bluetooth Weighing Scale Records patients weight and send it to cloud 33
  34. 34. kHealth Post-Bariatric Surgery Proposed Method Aggregate the data collected from the sensors, questionnaires and use AI techniques to: ● analyse and predict the deviations that could cause the post surgical complications and, ● serve as an assistant leading to better patient-compliance and outcomes 34
  35. 35. How do we solve problems with real world complexity, gather vast amount of data, diverse knowledge…. and come up with intelligent decisions that works for an individual at a given time? next: a pedagogical take 35
  36. 36. Semantic Cognitive Perceptual computing 36
  37. 37. Interplay between Semantic, Cognitive and Perceptual Computing (SC, CC and PC) with examples More here- Video, Slides Semantic Cognitive Perceptual computing - use case: Asthma 37
  38. 38. Thank you Special Thanks kHealth Team Members Revathy Venkataramanan (Graduate Student) Utkarshani Jaimini (Graduate Student) Hong Yung Yip (Graduate Student) Vaikunth Sridharan (Graduate Student) Dipesh Kadaria (Graduate Student) Quintin Oliver (Undergraduate Student) Tanvi Banerjee (Faculty) Dr. KrishnaPrasad Thirunarayan (Faculty) Clinical Collaborators Dr. Maninder Kalra (Pulmonologist at Dayton Childrens Hospital) Dr. Joon Shim (Bariatric Surgeon at Miami Valley Hospital) The Project kHealth Asthma is funded by NIH 1 R01 HD087132- 01 38
  39. 39. 39