Wrong conclusions in your analytics can cause waste and disillusionment, not to mention suboptimal outcomes that may take months or even years to recover from. But analytic analysis isn’t about perfection—it’s about getting to the right answer by quickly getting to the wrong one. In this interactive webinar, Jason Jones, chief data scientist at Health Catalyst, walks through scenarios that illustrate how commonly used analytic methods can lead analysts and leaders to the wrong conclusions, and shares how to course correct if this happens to you. In health and healthcare, leaders drive change by understanding and supporting better approaches, and analytics provide the best foundation for informed change management. Let’s work together to shift towards a better use of AI in healthcare. View this webinar to learn: - How analysis of the same data set can result in different conclusions. - Tools and techniques to get your organization back on track after a misstep. - Lessons from two case studies that will help you drive better analytics in your own organization.

1. Getting to the Wrong Answer Faster
with Your Analytics: Shifting to a Better
Use of AI in Healthcare
August 14, 2019
Jason Jones, PhD
Chief Data Scientist, Health Catalyst

2. Learning
Objectives
• Describe how the same data can
result in different conclusions.
• Identify tools and techniques to put
your organization back on track.
• Describe two cases to drive better
analytics.

3. Agenda
• Buying a House
• Rigor Sans Mortis
• Review and What’s Next

4. © 2018
Health
Catalyst
• Moving to a new state (or small
country).
• Would like to buy a house.
• Reports indicate some regions have
more cancer than others.
• You would like to figure out if there are
sectors where you should (not) buy a
house.
• You will never need any more data
than are exemplified here.
You Want to Buy a House…
4

5. © 2018
Health
Catalyst
• Exactly the same data as
before.
• There were 400 cancers
reported, now we can see
where.
• Chart shows the different
geographic regions.
• Does anything leap out?
Improvement
Through
Visualization
5

6. © 2018
Health
Catalyst
• Adding the grid helps
identify the regions more
clearly.
• There are 100 regions to
choose from.
• What pops out now?
• Anywhere you can see to
buy or not buy?
Improvement
Through
Visualization
6

7. © 2018
Health
Catalyst
• Triangles highlight regions
that have at least 2x the
typical rate of cancer.
• Green circles highlight
regions with no cancers.
Improvement
Through
Visualization
7

8. © 2018
Health
Catalyst
• Go to the URL below to make your
selections (accessible on computer
and most phone browsers).
• Click the region where you would
like to buy your home, then you’ll be
asked where you would not want to
buy.
Improvement
Through
Visualization
8
https://buyingahouse.azurewebsites.net/

9. © 2018
Health
Catalyst
Results
9

10. © 2018
Health
Catalyst
• Exactly the same data,
now in a bar chart.
• Displays how many regions
had each count of cancers.
• Note the 2 regions with no
cancers.
• 2 regions with 9 cancers.
• 1 region with 10 cancers.
10
Applying State/ML

11. © 2018
Health
Catalyst
• The blue line can help us
see that this looks a little
like a bell-shaped curve.
11
Applying State/ML

12. © 2018
Health
Catalyst
• The purple line is what we
would expect to see with
100 regions and 400
cancers.
• …If there was no
relationship between
region and cancer.
12
Applying State/ML

13. © 2018
Health
Catalyst
• The statistical test gives
strong evidence that there
is no relationship between
region and cancer.
13
Applying State/ML

14. © 2018
Health
Catalyst
• So…where would you (not)
buy a house?
• Does cancer by region
matter?
Getting to the Wrong
Answer Faster!
14

15. © 2018
Health
Catalyst
On a scale of 1-5, how useful did you find this exercise?
• 1 – Not useful: It’s a silly example or I knew this already – 2%
• 2 – 4%
• 3 – Ambivalent: This might help someone somewhere, but not me now – 25%
• 4 – 47%
• 5 – Extremely: I learned something that I can internalize and leverage – 22%
Poll Question #1
15

16. Agenda
• Buying a House
• Rigor Sans Mortis
• Review and What’s Next

17. © 2018
Health
Catalyst
Two Examples
• ”Scared Straight” started in 1978.
• Teenage delinquents spend 3
hours in prison.
• Stories of how delinquents
changed course because of the
intervention.
• Cochrane review of 9 rigorous
studies: 2 show no impact; 7
show negative impact.
• To improve academic attendance
and test scores in Kenyan kids,
ICS provided:
• More books (was 1 per class)
• Flipcharts (to tailor lessons–E3L)
• More teachers
• Each intervention showed no
impact upon careful study.
• Deworming:
• 25% decreased absenteeism
• 20% increased income 10 years
later
17
Nothing to Fear Books and Worms
https://www.effectivealtruism.org/doing-good-better/
https://blogs.sciencemag.org/books/2017/04/03/a-journalist-shines-a-harsh-spotlight-on-biomedicines-reproducibility-crisis

18. © 2018
Health
Catalyst
What Should the Market Buy?
18
What is a layout?
Slide layouts contain formatting,
positioning, and placeholders for
all of the content that appears on
a slide. Placeholders are the
containers in layouts that hold
such content as text (including
body text, bulleted lists, and
titles), tables, charts, SmartArt
graphics, movies, and pictures.
A layout contains the theme
(colors, fonts, effects, and the
background) of a slide as well.
https://www.aetv.com/shows/beyond-scared-straight (Jul 2019)
https://www.cochrane.org/CD002796/BEHAV_scared-straight-and-other-juvenile-awareness-programs-for-preventing-juvenile-delinquency
https://www.aetv.com/shows/beyond-scared-straight (Jul 2019)
https://www.cochrane.org/CD002796/BEHAV_scared-straight-and-other-juvenile-awareness-programs-for-preventing-juvenile-delinquency

19. © 2018
Health
Catalyst
Two Examples
• ”Scared Straight” started in 1978
• Teenage delinquents spend 3
hours in prison
• Stories of how delinquents
changed course because of the
intervention
• Cochrane review of 9 rigorous
studies: 2 show no impact; 7
show negative impact
• To improve academic attendance
and test scores in Kenyan kids,
ICS provided:
• More books (was 1 per class)
• Flipcharts (to tailor lessons–E3L)
• More teachers
• Each intervention showed no
impact upon careful study.
• Deworming:
• 25% decreased absenteeism
• 20% increased income 10 years
later
19
Nothing to Fear Books and Worms
https://www.effectivealtruism.org/doing-good-better/
https://blogs.sciencemag.org/books/2017/04/03/a-journalist-shines-a-harsh-spotlight-on-biomedicines-reproducibility-crisis

20. © 2018
Health
Catalyst
When was the last time you suspect your organization drew the wrong
conclusion in a preventable way (bad data or methods)? Note: Hold
yourself to the level you would expect in a master’s-level stats/methods
class.
• Within the last week – 31%
• Within the last month – 25%
• Within the last quarter – 25%
• Within the last year – 17%
• It’s never happened – 2%
Poll Question #2
20

21. © 2018
Health
Catalyst
An Unfortunate Obstacle
21
Big Data Strategy Components: IT Essentials, published October 15, 2012 and refreshed December 9, 2014.
Not
Solved
Wrong
Order Change
Management
This image has
caused harm in
our market.
Awesome!

22. © 2018
Health
Catalyst
What is your “chief complaint”?
• Low analytic throughput – 18%
• Low “data literacy” – 34%
• Lack agreement on definitions – 31%
• Don’t believe results – 8%
• Pilot-itis (too many or too long) – 10%
Poll Question #3
22

23. © 2018
Health
Catalyst
Low Analytic Throughput
Symptom of problematic data, direction.
23
https://en.wikipedia.org/wiki/Control_chart
Accelerates
predictive,
diagnostic &
prescriptive efforts.
Visualizing
early & often
can speed
analysis 10x.
Using SPC
focuses the
discussion.

24. © 2018
Health
Catalyst
Low Analytic Throughput
Symptom of problematic data, direction, or interpretation.
24
Accelerates
predictive,
diagnostic &
prescriptive efforts.
Visualizing
early & often
can speed
analysis 10x.
Using SPC
focuses the
discussion. Add
automated
projection.
Description of method: https://www.jstatsoft.org/article/view/v027i03
Robust evaluation of method: https://tinyurl.com/yadjh2u4 Return to “chief complaint”

25. © 2018
Health
Catalyst
Low Data Literacy
• Which geographies currently
have different performance?
• Which geographies will be
better, worse, or about the
same in a year?
• Are we becoming more similar
or different as a system?
Symptom of insufficient guidance.
25
Leverage stats/ML in “standard”
reporting to facilitate interpretation and
make conclusions explicit.
Return to “chief complaint”

26. © 2018
Health
Catalyst
Lack Agreement on Definitions
• Which subgroups or
outcomes favor older or
newer blood?
• What’s your prescription for
using older versus newer
blood?
Symptom of pursuing single version of truth.
26
“Single version of truth”
is illusory.
Pursue convergence of evidence and
curiosity about differences.
Cooper, NEJM, 2017 Nov. https://www.nejm.org/doi/full/10.1056/NEJMoa1707572
It is uncertain whether the duration of red-cell storage affects
mortality after transfusion among critically ill adults.
The age of transfused red cells did not affect 90-day mortality
among critically ill adults.
Older BetterFresh Better
Return to “chief complaint”

27. © 2018
Health
Catalyst
Confounding…Easy as ABC
• You’re interested in the relationship
between A&B.
• You worry something else, C, might be
clouding the relationship.
• What is an example of confounding
enhancing or obscuring a relationship?
• C cannot be a confounder unless it is
related to both A and B–interrogate and
deal with potential confounders.
• How did the last picture deal with
confounding?
• What questions remain for you?
Often asymptomatic
27
A B
C
https://www.healthknowledge.org.uk/public-health-textbook/research-methods/1a-epidemiology/biases
Return to “chief complaint”

28. © 2018
Health
Catalyst
How Would You Improve This Information?
28
https://www.nejm.org/doi/full/10.1056/NEJMoa1707572
Return to “chief complaint”

29. Agenda
29
• Buying a House
• Rigor Sans Mortis
• Review and What’s Next

30. © 2018
Health
Catalyst
• Humans are great at detecting certain kinds of patterns.
• We need to guide the analytic process to support drawing the right conclusions
from patterns.
• We have found that specific tools and techniques are both useful and attainable.
• The tools will not be enough: change management is key!
• Analytic and technical staff and vendors need to close the gap between tools
(“BI” and “ML”).
• Leaders need to encourage these attempts and remove fear.
Review and What’s Next
30

31. Q&A

32. Thank You!