Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Getting to the Wrong Answer Faster with Your Analytics: Shifting to a Better Use of AI in Healthcare

304 views

Published on

Wrong conclusions in your analytics can cause waste and disillusionment, not to mention suboptimal outcomes that may take months or even years to recover from. But analytic analysis isn’t about perfection—it’s about getting to the right answer by quickly getting to the wrong one.

In this interactive webinar, Jason Jones, chief data scientist at Health Catalyst, walks through scenarios that illustrate how commonly used analytic methods can lead analysts and leaders to the wrong conclusions, and shares how to course correct if this happens to you. In health and healthcare, leaders drive change by understanding and supporting better approaches, and analytics provide the best foundation for informed change management. Let’s work together to shift towards a better use of AI in healthcare.

View this webinar to learn:
- How analysis of the same data set can result in different conclusions.
- Tools and techniques to get your organization back on track after a misstep.
- Lessons from two case studies that will help you drive better analytics in your own organization.

Published in: Healthcare
  • Be the first to comment

  • Be the first to like this

Getting to the Wrong Answer Faster with Your Analytics: Shifting to a Better Use of AI in Healthcare

  1. 1. Getting to the Wrong Answer Faster with Your Analytics: Shifting to a Better Use of AI in Healthcare August 14, 2019 Jason Jones, PhD Chief Data Scientist, Health Catalyst
  2. 2. Learning Objectives • Describe how the same data can result in different conclusions. • Identify tools and techniques to put your organization back on track. • Describe two cases to drive better analytics.
  3. 3. Agenda • Buying a House • Rigor Sans Mortis • Review and What’s Next
  4. 4. © 2018 Health Catalyst • Moving to a new state (or small country). • Would like to buy a house. • Reports indicate some regions have more cancer than others. • You would like to figure out if there are sectors where you should (not) buy a house. • You will never need any more data than are exemplified here. You Want to Buy a House… 4
  5. 5. © 2018 Health Catalyst • Exactly the same data as before. • There were 400 cancers reported, now we can see where. • Chart shows the different geographic regions. • Does anything leap out? Improvement Through Visualization 5
  6. 6. © 2018 Health Catalyst • Adding the grid helps identify the regions more clearly. • There are 100 regions to choose from. • What pops out now? • Anywhere you can see to buy or not buy? Improvement Through Visualization 6
  7. 7. © 2018 Health Catalyst • Triangles highlight regions that have at least 2x the typical rate of cancer. • Green circles highlight regions with no cancers. Improvement Through Visualization 7
  8. 8. © 2018 Health Catalyst • Go to the URL below to make your selections (accessible on computer and most phone browsers). • Click the region where you would like to buy your home, then you’ll be asked where you would not want to buy. Improvement Through Visualization 8 https://buyingahouse.azurewebsites.net/
  9. 9. © 2018 Health Catalyst Results 9
  10. 10. © 2018 Health Catalyst • Exactly the same data, now in a bar chart. • Displays how many regions had each count of cancers. • Note the 2 regions with no cancers. • 2 regions with 9 cancers. • 1 region with 10 cancers. 10 Applying State/ML
  11. 11. © 2018 Health Catalyst • The blue line can help us see that this looks a little like a bell-shaped curve. 11 Applying State/ML
  12. 12. © 2018 Health Catalyst • The purple line is what we would expect to see with 100 regions and 400 cancers. • …If there was no relationship between region and cancer. 12 Applying State/ML
  13. 13. © 2018 Health Catalyst • The statistical test gives strong evidence that there is no relationship between region and cancer. 13 Applying State/ML
  14. 14. © 2018 Health Catalyst • So…where would you (not) buy a house? • Does cancer by region matter? Getting to the Wrong Answer Faster! 14
  15. 15. © 2018 Health Catalyst On a scale of 1-5, how useful did you find this exercise? • 1 – Not useful: It’s a silly example or I knew this already – 2% • 2 – 4% • 3 – Ambivalent: This might help someone somewhere, but not me now – 25% • 4 – 47% • 5 – Extremely: I learned something that I can internalize and leverage – 22% Poll Question #1 15
  16. 16. Agenda • Buying a House • Rigor Sans Mortis • Review and What’s Next
  17. 17. © 2018 Health Catalyst Two Examples • ”Scared Straight” started in 1978. • Teenage delinquents spend 3 hours in prison. • Stories of how delinquents changed course because of the intervention. • Cochrane review of 9 rigorous studies: 2 show no impact; 7 show negative impact. • To improve academic attendance and test scores in Kenyan kids, ICS provided: • More books (was 1 per class) • Flipcharts (to tailor lessons–E3L) • More teachers • Each intervention showed no impact upon careful study. • Deworming: • 25% decreased absenteeism • 20% increased income 10 years later 17 Nothing to Fear Books and Worms https://www.effectivealtruism.org/doing-good-better/ https://blogs.sciencemag.org/books/2017/04/03/a-journalist-shines-a-harsh-spotlight-on-biomedicines-reproducibility-crisis
  18. 18. © 2018 Health Catalyst What Should the Market Buy? 18 What is a layout? Slide layouts contain formatting, positioning, and placeholders for all of the content that appears on a slide. Placeholders are the containers in layouts that hold such content as text (including body text, bulleted lists, and titles), tables, charts, SmartArt graphics, movies, and pictures. A layout contains the theme (colors, fonts, effects, and the background) of a slide as well. https://www.aetv.com/shows/beyond-scared-straight (Jul 2019) https://www.cochrane.org/CD002796/BEHAV_scared-straight-and-other-juvenile-awareness-programs-for-preventing-juvenile-delinquency https://www.aetv.com/shows/beyond-scared-straight (Jul 2019) https://www.cochrane.org/CD002796/BEHAV_scared-straight-and-other-juvenile-awareness-programs-for-preventing-juvenile-delinquency
  19. 19. © 2018 Health Catalyst Two Examples • ”Scared Straight” started in 1978 • Teenage delinquents spend 3 hours in prison • Stories of how delinquents changed course because of the intervention • Cochrane review of 9 rigorous studies: 2 show no impact; 7 show negative impact • To improve academic attendance and test scores in Kenyan kids, ICS provided: • More books (was 1 per class) • Flipcharts (to tailor lessons–E3L) • More teachers • Each intervention showed no impact upon careful study. • Deworming: • 25% decreased absenteeism • 20% increased income 10 years later 19 Nothing to Fear Books and Worms https://www.effectivealtruism.org/doing-good-better/ https://blogs.sciencemag.org/books/2017/04/03/a-journalist-shines-a-harsh-spotlight-on-biomedicines-reproducibility-crisis
  20. 20. © 2018 Health Catalyst When was the last time you suspect your organization drew the wrong conclusion in a preventable way (bad data or methods)? Note: Hold yourself to the level you would expect in a master’s-level stats/methods class. • Within the last week – 31% • Within the last month – 25% • Within the last quarter – 25% • Within the last year – 17% • It’s never happened – 2% Poll Question #2 20
  21. 21. © 2018 Health Catalyst An Unfortunate Obstacle 21 Big Data Strategy Components: IT Essentials, published October 15, 2012 and refreshed December 9, 2014. Not Solved Wrong Order Change Management This image has caused harm in our market. Awesome!
  22. 22. © 2018 Health Catalyst What is your “chief complaint”? • Low analytic throughput – 18% • Low “data literacy” – 34% • Lack agreement on definitions – 31% • Don’t believe results – 8% • Pilot-itis (too many or too long) – 10% Poll Question #3 22
  23. 23. © 2018 Health Catalyst Low Analytic Throughput Symptom of problematic data, direction. 23 https://en.wikipedia.org/wiki/Control_chart Accelerates predictive, diagnostic & prescriptive efforts. Visualizing early & often can speed analysis 10x. Using SPC focuses the discussion.
  24. 24. © 2018 Health Catalyst Low Analytic Throughput Symptom of problematic data, direction, or interpretation. 24 Accelerates predictive, diagnostic & prescriptive efforts. Visualizing early & often can speed analysis 10x. Using SPC focuses the discussion. Add automated projection. Description of method: https://www.jstatsoft.org/article/view/v027i03 Robust evaluation of method: https://tinyurl.com/yadjh2u4 Return to “chief complaint”
  25. 25. © 2018 Health Catalyst Low Data Literacy • Which geographies currently have different performance? • Which geographies will be better, worse, or about the same in a year? • Are we becoming more similar or different as a system? Symptom of insufficient guidance. 25 Leverage stats/ML in “standard” reporting to facilitate interpretation and make conclusions explicit. Return to “chief complaint”
  26. 26. © 2018 Health Catalyst Lack Agreement on Definitions • Which subgroups or outcomes favor older or newer blood? • What’s your prescription for using older versus newer blood? Symptom of pursuing single version of truth. 26 “Single version of truth” is illusory. Pursue convergence of evidence and curiosity about differences. Cooper, NEJM, 2017 Nov. https://www.nejm.org/doi/full/10.1056/NEJMoa1707572 It is uncertain whether the duration of red-cell storage affects mortality after transfusion among critically ill adults. The age of transfused red cells did not affect 90-day mortality among critically ill adults. Older BetterFresh Better Return to “chief complaint”
  27. 27. © 2018 Health Catalyst Confounding…Easy as ABC • You’re interested in the relationship between A&B. • You worry something else, C, might be clouding the relationship. • What is an example of confounding enhancing or obscuring a relationship? • C cannot be a confounder unless it is related to both A and B–interrogate and deal with potential confounders. • How did the last picture deal with confounding? • What questions remain for you? Often asymptomatic 27 A B C https://www.healthknowledge.org.uk/public-health-textbook/research-methods/1a-epidemiology/biases Return to “chief complaint”
  28. 28. © 2018 Health Catalyst How Would You Improve This Information? 28 https://www.nejm.org/doi/full/10.1056/NEJMoa1707572 Return to “chief complaint”
  29. 29. Agenda 29 • Buying a House • Rigor Sans Mortis • Review and What’s Next
  30. 30. © 2018 Health Catalyst • Humans are great at detecting certain kinds of patterns. • We need to guide the analytic process to support drawing the right conclusions from patterns. • We have found that specific tools and techniques are both useful and attainable. • The tools will not be enough: change management is key! • Analytic and technical staff and vendors need to close the gap between tools (“BI” and “ML”). • Leaders need to encourage these attempts and remove fear. Review and What’s Next 30
  31. 31. Q&A
  32. 32. Thank You!

×