Visual Analytics for Healthcare - Panel at AMIA 2012 in Chicago


Published on

AMIA 2012 Panel on Visual Analytics for Healthcare


Adam Perer, PhD
Research Scientist
IBM T.J. Watson Research Center, Hawthorne, NY

Ben Shneiderman, PhD
Professor, Computer Science
University of Maryland, College Park, MD

Yuval Shahar, PhD
Professor, Head of the Medical Informatics Research Center
Ben Gurion University, Beer Sheva, Israel

Jeffrey Heer, PhD
Assistant Professor, Computer Science
Stanford University, Stanford, CA

David Gotz, PhD
Research Scientist
IBM T.J. Watson Research Center, Hawthorne, NY


With the proliferation of medical information technology, users at all levels of the healthcare system have access to more data than ever before6. This data can be of tremendous value but is often difficult to access and interpret. For example clinicians are often faced with the challenging task of analyzing large amounts of unstructured, multi-modal, and longitudinal data to effectively diagnose and monitor the progression of a patient’s disease4,5. Similarly, patients are confronted with the difficult task of understanding the trends and correlations within data related to their own health. At the institutional level, healthcare organizations are faced with the desire to use data to improve overall operational efficiency and performance, while continuing to maintain the quality of patient care and safety.

Recent advances in visualization and visual analytics have the potential to help each of the user groups listed above do more with the often overwhelming amount of data available to them 1,3,7,8. However, to be successful, visualization designers and clinicians must work together closely to ensure that the right technologies are used to help address the meaningful problems. Unfortunately, despite the continuous use of scientific visualization and visual analytics in medical applications, the lack of communication between engineers and physicians has meant that only basic visualization and analytics techniques are currently employed in clinical practice2,9.

The goal of this panel is to present state-of-the-art visualization applications for healthcare and engage the leading physicians and clinical researchers at AMIA to discuss the areas in healthcare where additional visualization techniques are most needed.

Published in: Health & Medicine
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Visual Analytics for Healthcare - Panel at AMIA 2012 in Chicago

  1. 1. Panel onVisual Analytics forHealthcare AMIA 2012 - November 5, 2012 Moderator: Adam Perer, IBM Research
  2. 2. Visual analytics combinesautomated analysis withinteractive visualizations to understand, reasonand make decisions from big data Definition adapted from Daniel Keim, Jörn Kohlhammer,Geoffrey Ellis and Florian Mansmann’s Mastering the Information Age Solving Problems with Visual Analytics
  3. 3. Panelists• Ben Shneiderman University of MarylandPattern Finding in Point & Interval Event Sequences
  4. 4. Panelists Yuval ShaharBen-Gurion UniversityVisual Analytics for Discovery of Time-Oriented Clinical Knowledge
  5. 5. Panelists David Gotz IBM ResearchVisual Analytics for Healthcare
  6. 6. Panelists Diana Maclean Stanford UniversityFinding What to Look For Exploratory VisualAnalytics for OnlineHealth Communities
  7. 7. Pattern Finding in Point & Interval Event Sequences ! Ben Shneiderman @benbendcFounding Director (1983-2000), Human-Computer Interaction Lab Professor, Department of Computer Science Member, Institute for Advanced Computer Studies University of Maryland College Park, MD 20742
  8. 8. Interdisciplinary research community - Computer Science & Info Studies - Psych, Socio, Poli Sci & MITH (
  9. 9. Patient Histories: Our Research Tool Event Records Display Types LifeLines Points, One Individual Intervals LifeLines2 Points Many Individual, Summary Similan Points Many Individual LifeFlow Points Many Individual, Aggregate EventFlow Points, Many Individual, Intervals Aggregate
  10. 10. LifeLines: Patient Histories
  11. 11. LifeLines2: Contrast+Creatine
  12. 12. LifeLines2: Align-Rank-Filter & Summarize
  13. 13. LifeLines2: Align-Rank-Filter & Summarize
  14. 14. Similan: Search
  15. 15. LifeFlow: Aggregation Strategy Temporal Categorical Data (4 records) LifeLines2 format Tree of Event Sequences LifeFlow Aggregation
  16. 16. LifeFlow: Interface with User Controls
  17. 17. EventFlow: Original Dataset
  18. 18. LABA_ICSs Merged
  19. 19. SABAs Merged
  20. 20. Align by First LABA_ICS
  21. 21. Reduce Window Size
  22. 22. Overview + Details
  23. 23. Current Directions - Motif Simplifications: Find & Replace - Query Features - Menus & Graphical Query - Absence for Points & Intervals - Flexible Temporal Patterns - Scalability - Event types, Events, Records, Query complexity - LP Algorithm - Long-term Case Studies
  24. 24. 30th Annual Symposium May 22-23, 2013 @benbendc
  25. 25. Visual Analytics for Discovery of Time-Oriented Clinical Knowledge or: You need to Know Something if You Want to Know More Yuval Shahar, M.D., Ph.D. Medical Informatics Research CenterDepartment of Information Systems Engineering Ben Gurion University, Beer Sheva, Israel
  26. 26. Declarative Knowledge in the Medical Domain•  Many medical tasks, especially those involving chronic patients, require extraction of clinically meaningful concepts from multiple sources of raw, longitudinal, time-oriented data•  Example: Modify the standard dose of the drug, if during treatment, the patient experiences a second episode of moderate anemia that has persisted for at least two weeks•  Examples of clinical tasks that require temporal reasoning: –  Therapy •  Following a treatment plan based on a clinical guideline –  Monitoring and Diagnosis •  Searching for a gradual increase of fasting blood-glucose level –  Quality assessment •  Comparing observed treatments with those recommended by a guideline –  Research •  Discovery of hidden dependencies over time between clinical parameters
  27. 27. The Need for Intelligent Mediation: The Gap Between Raw Data and Meaningful Concepts Clinical databases store raw, time-stamped data BUT: Clinicians and decision-support applications reason in terms of abstract, clinically meaningful concepts, typically over time periods, AKA Temporal abstractions! Automated computation of concepts or of temporal patterns derivable from raw data, using knowledge, supports monitoring, interactive data mining, and automated discovery of meaningful temporal patterns
  28. 28. Bridging the Gap: Knowledge-BasedMonitoring, Exploration, and Discovery •  A distributed architecture that caters for three needs: –  Automated means for monitoring and recognition of meaningful known patterns, in time-oriented data, by applying temporal-abstraction knowledge from multiple domain-specific knowledge sources to data from multiple data sources –  Interactive, human-operated means for dynamic visual exploration of a time-oriented data repository, using on-the- fly integration with domain-specific knowledge, to identify new meaningful patterns and add them to the knowledge base –  Automated analysis, enumeration, and discovery of new meaningful, significant temporal-abstraction patterns (relationships amongst temporal-abstraction intervals)
  29. 29. The Temporal-Abstraction Ontology (Shahar, Artif. Intell. 1997)• Used by the Knowledge-Based Temporal-Abstraction Method;Includes:• Events (interventions) (e.g., insulin therapy; surgery; irradiation) - part-of, is-a relations• Parameters (measured raw data and derived [abstract] concepts) (e.g., hemoglobin values; anemia levels; liver toxicity grade) - abstracted-into, is-a relations• Patterns (e.g., crescendo angina; paradoxical hyperglycemia) - component-of, is-a relations• Abstraction goals (user views)(e.g., diabetes therapy) - is-a relations• Interpretation contexts (effect of regular insulin; pregnancy; infant) - subcontext, is-a relations• Interpretation contexts are induced by all other entities
  30. 30. Temporal-Abstraction Output Types• State abstractions (LOW, HIGH)• Gradient abstractions (INC, DEC)• Rate Abstractions (SLOW, FAST)• Pattern Abstractions (CRESCENDO) - Linear [one-time] patterns - Periodic [repeating] patterns - Fuzzy patterns (partial match)
  31. 31. Temporal-Abstraction Knowledge Types • Structural (e.g., part-of, is-a relations) - mainly declarative/relational (BMI = ƒ(Wt,Ht)) • Classification (e.g., value ranges; patterns) - mainly functional (BMI = Wt/Ht^2) • Temporal-semantic (e.g., concatenable property) - mainly logical (anemia is concatenable; pregnancy is not) • Temporal-dynamic (e.g., interpolation functions) - mainly probabilistic (anemia periods can be bridged)
  32. 32. The IDAN Temporal-Abstraction Mediator [Boaz and Shahar, Artif. Intell. Med 2005] Medical knowledge Knowledge- Medical service acquisition expert tool Temporal- Decision Standard medical abstraction support vocabularies service controller system End user (care Medical data Access Temporal- service provider) abstraction service
  33. 33. The GESHER Knowledge Structuring and Maintenance Tool:Creating a Declarative Knowledge Map from Medical Concepts [Hatsek et al., OMIJ 2010] A knowledge mapConstraints onconcept values Structured text description
  34. 34. The KNAVE-II Single-Subject Browsing and Exploration Interactive Interface [Shahar et al., Artif. Intell. Med 2006] Overall patternMedical knowledge Intermediate browser interpretationsConcept search Raw clinical data
  35. 35. Evaluation of KNAVE-II (Palo Alto Veterans Administration Health Care System) (Martins , Shahar, et al., Artif. Intell. Med. 2008)!  14 clinicians with varying medical/computer use backgrounds!  Each user was given a brief demonstration of the interface!  DB: more than 1000 bone-marrow transplantation patients (2-4yrs)!  Each user asked to answer 10 queries common in oncology protocols!  A cross-over study design compared the KNAVE-II module versus two existing methods: paper charts and an electronic spreadsheet (ESS) !  (The 2nd phase, using more difficult queries, compared only versus the ESS)•  Direct Ranking comparison: KNAVE-II ranked first in preference by all users•  SUS Usability Scores: KNAVE-II 69, ESS 48, Paper 46 (P=0.006)•  Time: In the first evaluation: Users were significantly faster using KNAVE-II, up to a mean of 93 seconds difference versus paper, and 27 seconds versus the ESS, for the hardest query (p = 0.0006); In the second evaluation: The comparison with the ESS showed a similar trend for moderately difficult queries (P=0.007) and for hard queries (p=0.002); the two hardest queries were answered a mean of 277 seconds faster when using KNAVE-II rather than ESS•  Correctness: for KNAVE-II 92% [110/120]; for ESS 57% [69/120], in the second study; scores were significantly higher for all queries (p<0.0001)
  36. 36. Exploration of Subject Populations: The VISITORS System [Klimov and Shahar, Artif. Intell. Med. 2010; J. Intell. Info. Sys. 2010]•  VISualizatIon and exploration of Time-Oriented raw data and abstracted concepts for multiple RecordS –  Knowledge-based time-oriented interpretations of the raw data –  Graphical construction of subject-selection query expressions –  Visual display and interactive exploration –  Use of absolute time as well as relative time (from some event) –  Multiple-record aggregation and association•  Evaluated for functionality and usability by clinicians and knowledge engineers, with encouraging results
  37. 37. A VISITORS Select Subject Query (1)•  Demographic Constraints: –  Male subjects, who are Young (age≤20) or Old (age≥70) OR relation Query is automatically and incrementally being created from the user s graphical specification
  38. 38. A VISITORS Select Subjects Query (2) •  Knowledge based constraints Hemoglobin state was abstracted as less than Normal, for at least seven days, starting at a time point that is at least two weeks after the allogenic BMT WBC count was increasing during the same period
  39. 39. A VISITORS Select Time Intervals Query •  Find time intervals (in a monthly resolution) during which the HGB value state was considered lower than “normal” for more than 50% of the subjects
  40. 40. The VISITORS Multiple-Records Main Interactive-Display Interface Subject groupsMedical knowledge browser Multiple-subjects raw data Distribution of derived patterns over timeConcept search
  41. 41. Temporal Association ChartsAbstractions for the same [Klimov and Shahar, Meth. Info. Med. 2010]subject group areconnected; support andconfidence are indicatedby width and hue The data of each subject are connected by a line
  42. 42. A Temporal-Mediation Application Example: The MobiGuide Project •  Coordinated by Mor Peleg, Haifa University, Israel •  Funded by the EU; an FP7 Integrated Project •  13 partners from 5 countries •  Monitoring of chronic patients through bodily sensors and a smart phone –  Cardiac arrhythmia patients in Italy –  Diabetes and high blood pressure in high-risk pregnancy in Spain •  Provision of alerts to the patients through the mobile phone, and guideline- based decision support to their care providers through the Web •  Abstraction of raw time-oriented monitored and historical patient data, to support interpretation, alerting, decision support, quality assessment, and mining performed by a temporal mediator
  43. 43. Temporal Data Mining:Mining Temporal Interval Related PatternsA Temporal Interval Related Pattern (TIRP) is a conjunction of temporal relations among symbolic time intervals (i.e., abstractions){A1 o B, A1 o D, A1 m C1, A1 b C2, A1 b A2, B o D, B c C1, B b C2, B b A2, C1 b C2, C1 b A2, C2 o A1 , D c C1,D o C2}
  44. 44. KarmaLego – Fast TIRP Mining [Moskovitch & Shahar, IDAMAP 2009, AMIA 2009]!*Ri = {Before, After, During, Overlaps…}
  45. 45. A KarmaLego Example: Looking at a Diabetes Dataset [Moskovitch & Shahar, AMIA 2009]!•  Contains 2038 diabetic patients data accumulating over five years (2002-2007) , monitored by a large HMO•  Includes monthly measurements such as of HbA1c, Glucose, and Cholesterol values, and medications purchased, including diabetic (insulin-based) medications, statins, and beta- blockers, normalized by the Defined Daily Dose (DDD)•  The laboratory-test values were abstracted using the KBTA method, based on domain expert specifications•  The medication doses were abstracted, using the Equal- Width Discretization method, into three states!
  46. 46. Exploration of Diabetes TIRPs: An Example of discovered Patterns [Moskovitch & Shahar, AMIA 2009] 0.26 0.18 0.22 0.280.25 0.23 0.33 0.42 0.29 Shown : Levels of [vertical] support; [No. cases/Horizontal support], D.dec, De.stab: drug dose gradient; H.dec,, H.stab: HbA1C gradient F = Finishes; M = Meets; S = Starts (temporal relations)
  47. 47. The KarmaLego Visualization Tool (I)!
  48. 48. The KarmaLego Visualization Tool (II)!
  49. 49. Automated Classification: Using TIRPs as Features [Moskovitch & Shahar, IDAMAP 2009]•  The TIRPs discovered by KarmaLego can be used as features for classification•  Classification was rigorously evaluated in several medical domains •  Example: An ICU dataset of patients who underwent cardiac surgery at the Academic Medical Center in Amsterdam during April 2002-May 2004 •  Static data include details such as age, gender, surgery type •  Temporal data (HR, BP, FiO2…) measured each minute during first 12 hours •  Classification task: Determine whether the patient was mechanically ventilated more than 24 hours during her postoperative ICU stay •  664 patients; 196 patients were mechanically ventilated for more than 24hrs (29.5%) •  Multiple aspects were investigated: The temporal-relations fuzziness factor value, the discretization method, the feature selection method… •  Overall accuracy: 79.6% for most combinations involving 5 discrete states using a very simple equal-width discretization method
  50. 50. Summary: Intelligent Abstraction, Exploration, and Discovery of Time-Oriented Data and Their Abstractions•  It takes knowledge to obtain even more knowledge!•  Distributed integration of time-oriented clinical data and knowledge•  Faster identification of new patterns •  Goal-directed: By supporting intelligent, interactive visual exploration, by a domain expert, of the contents of the accumulating time-oriented database •  Data-driven: by automated discovery of frequent temporal patterns•  Quick adaptation to new patterns, by enabling human experts to easily modify the knowledge base•  Visualization provides concise, meaningful summaries of large amounts of time-oriented data in terms familiar to the clinicians •  Temporal abstractions can also be used for generation of natural language summaries•  Suggests an iterative process in which new discovered and validated knowledge is added to the knowledge base and is exploited for the discovery of further medical knowledge
  51. 51. David&Gotz&Healthcare&Analy4cs&Research&Group&IBM&T.J.&Watson&Research&Center&VISUALANALYTICSFORHEALTHCARE
  52. 52. Makinghealthcaresmarter nMy&Team&at&IBM&Research& HealthcareAnalyBcsResearchGroup IBMT.J.WatsonResearchCenter YorktownHeights,NewYork hMp:// Data StaBsBcs Mining Visual Clinical AnalyBcs Medicine
  53. 53. Makinghealthcaresmarter nPersonalized&EvidenceDBased&Medicine& Pa4ent& Clinician& Searchand Analysis TensofThousandsto10+MillionPa9ents •  SeveralYearsofDataPerPa/ent •  ThousandsofFeatures •  Demographics •  Diagnoses •  Labs •  Procedures •  Claims •  UnstructuredPhysicianNotes
  54. 54. Makinghealthcaresmarter nPersonalized&EvidenceDBased&Medicine& Pa4ent& Clinician&
  55. 55. Makinghealthcaresmarter n Many&Opportuni4es&for&Visualiza4on& Similarity Analysis ? Clinically similar to x1 ? Q x2 Q … 3 xN Q Query patient Patient similarity assessment in clinical 1 Visual cohort refinement factor/feature space x x1 x 1 2 K 1 x 1 , x2 2 ,… , 1 x K 2 2 … … … x x x ? 1 2 N N N KPatient population 2 Visual outcome analysis
  56. 56. Makinghealthcaresmarter nCohort&Refinement&and&Manipula4on& SolarMapFacetAtlas
  57. 57. Makinghealthcaresmarter nDICON&•  Iconic&mul4dimensional&visualiza4on&of&cohorts&&&&•  Icons&are&interac4ve& –  Compare& –  Merge& –  Split&
  58. 58. Makinghealthcaresmarter nDICON&Demonstra4on&
  59. 59. Makinghealthcaresmarter nVisual&Outcome&Analysis&
  60. 60. Makinghealthcaresmarter nOuTlow:A&Temporal&Pathway&Visualiza4on&Patient Outcome Time-stamped Events Aggregate Alignment&Point& [A]& [A,B]& [A,B,C,D]& [&]& [B]& [A,C]& [A,B,C]& Average outcome = 0.4 [A,B,C,E]& Average time = 10 days [C]& [B,C]& Number of patients = 10
  61. 61. Makinghealthcaresmarter n OuTlow:&Visual&Encoding& Past NOW Future Horizontal& posi4on& shows& Dsequence&of& states.& A Height&is& number&of& C people& E B Color&is& outcome& Width&is&dura4on&of& measure& transi4on&
  62. 62. Makinghealthcaresmarter nOuRlow&Demonstra4on&
  63. 63. Makinghealthcaresmarter nInterac4ve&Cohort&Refinement&•  So&far…& AnalyBcs VisualizaBon•  However,&cohort&analysis&is&not&always&a&oneDway& process& Persisted Cohorts Cohort Cohort Cohort Cohort Cohort Cohort Views Cohort Analytics
  64. 64. Makinghealthcaresmarter nItera4ve&Visual&Cohort&Analysis&
  65. 65. Makinghealthcaresmarter n Conclusion& Similarity Analysis ? Clinically similar to x1 ? Q x2 Q … 3 xN Q Query patient Patient similarity assessment in clinical 1 Visual cohort refinement factor/feature space x x1 x 1 2 K 1 x 1 , x2 2 ,… , 1 x K 2 2 … … … x x x ? 1 2 N N N KPatient population 2 Visual outcome analysis
  66. 66. Finding What to Look For Exploratory Visual Analytics for Online Health Communities Diana MacLean, PhD Candidate Advised by Jeffrey Heer Computer Science Dept. Stanford University
  67. 67. Exploratory Visual Analytics•  Start of research cycle (rinse, repeat)•  Goals • Create a mental map of the data • Drive hypotheses generation•  Useful for • Big(ish) data • Researchers with partial/full domain expertise
  68. 68. What do users talk about? Do forums contain novel, useful information?Can online health forum participation help patients?FORUM CONTENT
  69. 69. The Raw Data
  70. 70. Creating a Co-Occurrence GraphWgat allergy that might trigger asthma?And she has had allergy and asthma problems since birth.It could be asthma, or you could have a heart condition.
  71. 71. Creating a Co-Occurrence GraphWgat allergy that might trigger asthma?And she has had allergy and asthma problems since birth.It could be asthma, or you could have a heart condition. asthma allergy
  72. 72. Creating a Co-Occurrence GraphWgat allergy that might trigger asthma?And she has had allergy and asthma problems since birth.It could be asthma, or you could have a heart condition. asthma allergy
  73. 73. Creating a Co-Occurrence GraphWgat allergy that might trigger asthma?And she has had allergy and asthma problems since birth.It could be asthma, or you could have a heart condition. asthma allergy
  74. 74. Creating a Co-Occurrence GraphWgat allergy that might trigger asthma?And she has had allergy and asthma problems since birth.It could be asthma, or you could have a heart condition. asthma allergy
  75. 75. Creating a Co-Occurrence GraphWgat allergy that might trigger asthma?And she has had allergy and asthma problems since birth.It could be asthma, or you could have a heart condition. heart condition asthma allergy
  76. 76. Pain Forum
  77. 77. Pain Forum
  78. 78. Pain Forum
  79. 79. Pain ForumHypothesis: A common discussion pattern in the PainManagement forum is, “I took [DRUG X] after havingsurgery on [BODY PART Y].”
  80. 80. Pain Forum : Body Parts
  81. 81. Pain Forum : Body Parts
  82. 82. Hypothesis 1: A data-driven derivation of “pain types”from health forum data would closely mirror anexpert-derived categorization.Hypothesis 2: We can use data-driven categorizationto map out symptom types for conditions that are lessunderstood (e.g. Lyme Disease).
  83. 83. Asthma Forum
  84. 84. Asthma Forum
  85. 85. Asthma Forum Hypothesis: Forums related to specific conditions have smaller vocabularies of medically- relevant terms.
  86. 86. Hypothesis: Drugs aregrouped by function/application. We can minethis data to determinewhich drugs people areusing to treat certainconditions.
  87. 87. When%you%go%outside,%try%wearing%a%scarf%over%your%nose%and%mouth%to%see%if%it%quells%the%reac8on%
  88. 88. When you go outside, trywearing a scarf over your noseand mouth to see if it quells thereactionAfter attack I got an enroumousamount of mucs (half of trashbas of napkins and more)especially after attack.
  89. 89. Hypothesis: This is spam.
  90. 90. Allergies Forum
  91. 91. Allergies Forum
  92. 92. Summary•  First: figure out what to look for•  Exploratory visual analytics can help us marshal hypotheses –  Quickly –  Even with big (ish) data –  But without accuracy /completeness guarantees•  Visualizations can be playful –  Fun/accuracy trade-off –  Can we engage non-experts (users), too?