Self-driving cars have become the most visible form of computer-aided decision support in society. What can we learn from these innovations—both good and bad, technically and culturally—about computer-aided decision support for clinicians? The adoption of EHRs provided a foundation; what and how do we build on that foundation to help clinicians, and patients, benefit from meaningful, precise decision support? Scott Weingarten, MD, MPH, and Dale Sanders explore clinical decision support in a joint webinar. Dr. Weingarten is recognized throughout the U.S. and international healthcare space as a physician and for his contributions to decision support, including his role in founding Zynx and Stanson Health. Dale brings a technologist’s viewpoint to the conversation, informed by his background in computer-aided decision support in the healthcare, military, and national intelligence sectors. During this webinar, learn more about the following topics: -How clinical decision support can improve the quality, safety, and value of care. -How developments in the field of artificial intelligence will impact clinical decision support. -The conceptual framework for digitizing an industry.Tradeoffs in artificial intelligence models between data volume and algorithm complexity. -The approach to digitization in the automobile and aerospace industries. -Shortcomings in current healthcare data.Future aspirations and plans for further digitization of healthcare.

1. Clinical Decision Support:
Driving the Last Mile
January 16, 2020
Dale Sanders
Chief Technology Officer, Health Catalyst
Scott Weingarten, MD, MPH
Chief Executive Officer, Stanson Health

2. © 2020
Health
Catalyst
On a scale of 1-5, rate your organization’s clinical decision support
effectiveness.
Poll Question #1
2
1. Not effective at all – 12%
2. Somewhat effective – 29%
3. Moderately effective – 38%
4. Very effective – 14%
5. Extremely effective – 7%

3. © 2020
Health
Catalyst
In your opinion, what is the greatest barrier to better clinical decision
support?
• Technology of EHRs – 20%
• Uncertainties in evidence-based medicine – 13%
• Clinician cultural resistance – 30%
• Fundamentally poor data quality in healthcare – 30%
• Other – 7%
Poll Question #2
3

4. © 2020
Health
Catalyst
• Spend lots of time getting the Concepts right, then explore options
for Implementation
• I’m not talking about requirements… I’m talking about Concepts
• There are only a few good Concepts for solving a problem, but there are
usually lots of options for Implementation
• If you don’t nail the Concepts, your Implementation will forever underperform
or fail
• Example
• Historically, the conceptual center of EHR design was the Encounter, but it
should have been the Patient
• That conceptual miss has dogged all of us for years, forcing all sorts of
workarounds in software, data, and workflow
Solving Problems, Building Systems

5. © 2020
Health
Catalyst
• The human mind works like a filing system
• Give it the General file folders, then fill those file folders with Specifics
• The military called this “Gen-Spec” learning
Teaching and Informing the Human Mind
5

6. • Dale: 20 minutes
– Concepts and frameworks for
decision support in healthcare
– Current and future state of data in
US healthcare
• Scott: 20 minutes
– The opportunities and potential for
better decision support
– Examples of clinical decision
support in the real world of EHRs
• Q&A: 20 minutes
Today’s Agenda

7. © 2020
Health
Catalyst
Underground
Command Center
for US Nuclear Forces
7

8. © 2020
Health
Catalyst
US National
Emergency
Airborne Command
Post System
“Doomsday Planes”
8

9. © 2020
Health
Catalyst
Splashdown of inert
warheads in Kwajalein
Atoll from Peacekeeper
Intercontinental Ballistic
Missile, ~1995
Telemetry data, galore
9

10. © 2020
Health
Catalyst
• Our data quality
(Completeness x Validity) in
healthcare is not that great
• The data is still useful, but
beware of these current data
quality limitations in decision
support
Employ Decision Support, Cautiously
10
Time
Data
Quality
2008

11. © 2020
Health
Catalyst
Airplane Pilot Decision Support
“The airframe, the
hardware, should get it
right the first time and not
need a lot of added bells
and whistles to fly
predictably.”
“Boeing’s solution to its
hardware problem was
software.”
Choose your use cases carefully…
11

12. Decision Support
Concepts and Frameworks in Healthcare

13. © 2020
Health
Catalyst
1313

14. © 2020
Health
Catalyst
Closed Loop Analytics
Loop C: Populations
• MTTI: Years, decades
• SPA: Millions, several hundred thousand
• Analytic consumers: Board of Directors, executive leadership team, Strategic plans and policy
Loop B: Protocols
• MTTI: Weeks, months
• SPA: Subsets of patients– hundreds, thousands
• Analytic consumers: Care improvement teams, clinical service lines
Loop A: Patients
• MTTI: Minutes, hours
• SPA: Individual patients
• Analytic consumers: Physicians and patients at the point of care
MTTI: Mean Time To Improvement, SPA: Span of Population Affected
14

15. © 2020
Health
Catalyst
Improve
Health
Level 9 Direct-to-Member Analytics & Artificial Intelligence
Level 8 Personalized Medicine & Prescriptive Analytics
Level 7 Clinical Risk Intervention & Predictive Analytics
Reduce
Variation
Level 6 Population Health Management & Suggestive Analytics
Level 5 Waste & Care Variability Reduction
Improve
Efficiency
Level 4 Automated External Reporting
Level 3 Automated Internal Reporting
Level 2 Standardized Vocabulary & Member Registries
Level 1 Enterprise Data Operating System
Level 0 Fragmented Point Solutions
The Healthcare Analytics Adoption Model
15

16. © 2020
Health
Catalyst
Creating the Patient’s Digital Twin
Developing three fundamental AI pattern recognitions in healthcare
16
Patients like
this
[pattern]
Who were
treated like
this
[pattern]
Had these
outcomes
and costs
[pattern]
Less about predictions, more about patterns
hpcwire.com

17. © 2020
Health
Catalyst
Sanders’ Predictive Analytics Postulate
17
Predictions without interventions are a
liability to the decision maker, not an asset.

18. Digitizing an Industry for
Decision Support
Aerospace and Automotive Role Models

19. © 2020
Health
Catalyst
What’s Required to become “Digitized?”
1. Digitize the assets you
are trying to manage
and optimize
Airplanes
Air traffic control,
baggage handling,
ticketing,
maintenance,
manufacturing
2. Digitize your
operations for
managing the
assets you are
trying to understand
and optimize
19
Patients
Registration,
scheduling, encounters,
diagnosis, orders,
billing, claims

20. © 2020
Health
Catalyst
Data Volume is Key to AI
“The Unreasonable Effectiveness
of Data”, March 2009, IEEE
Computer Society; Alon Halevy,
Peter Norvig, and Fernando
Pereira, of Google
“Invariably, simple models and a
lot of data trump more elaborate
models based on less data.”
20

21. © 2020
Health
Catalyst
• Every 10 hours, Tesla collects
1 million miles of driving data
• 25Gbytes per car per hour
• “We can fix problems in your
car and make it safer, long
before you know you need it.”
• “10,000 fatalities and 500,000
injuries per year will be
prevented.”
– Ram Ramachander, Chief
Commercial Officer, Social
Innovation Business at Hitachi
Vehicle Health Monitoring – Human
Health Monitoring
21

22. © 2020
Health
Catalyst
Properties of Satellite (and Human) Telemetry Data
• High Dimensionality: Hundreds to thousands of data variables
• Multimodality: Day and night modes; pediatric & adult
• Heterogeneity: Continuous, real values; discreet, categorical values
• Temporal Dependence: At what time you collect the data matters;
the temporal dimension between heterogeneous data also matters
• Missing Data: Is the missing data expected to be missing, or not?
Spacecraft Health Monitoring – Human
Health Monitoring
22
TRW/Northrup Grumman
DSP Satellite

23. © 2020
Health
Catalyst
23
“…newest generation aircraft…
five-to-eight terabytes per flight”
“Airplanes like the 787 and A350 collect 10,000
times more data than 1990s or early 2000s-era
aircraft. That is because more parameters are
being measured at higher frequencies, using
broader transmission pipelines.”
– Joel Reuter, Vice President of Public Affairs, Rolls-Royce North America

24. Current State
The Data for Decision Support in Healthcare

25. © 2020
Health
Catalyst
Our Digital Understanding of Patients is Poor
This is my life.
This is healthcare’s
digital view of my life.
25

26. © 2020
Health
Catalyst
We Are Not “Big Data” in Healthcare, Yet
26
Citation: Dale Sanders, CIO, Northwestern
Medicine. Calculating annual storage
requirements for the Northwestern electronic
health record, 2011
5-8TB per 4 hrs.
30TB in 8 hrs.
26
100 MB per year

27. © 2020
Health
Catalyst
Turn this into your strategic
data acquisition roadmap
• In the US, our digital view of
the patient is stuck in the lower
left quadrant
• On average, we collect data on
patients about 3x per year in
the US, during visits to the
clinic or hospital
• We collect almost no data on
healthy patients, who rarely
visit the healthcare system
The Human Health Data Ecosystem
27

28. © 2020
Health
Catalyst
• July 2019
• U of Toronto, Microsoft,
Johns Hopkins, Harvard,
MIT, New York University
28
“…diseases in EHRs are poorly labeled,
conditions can encompass multiple underlying
endotypes, and healthy individuals are
underrepresented. This article serves as a primer
to illuminate these challenges and highlights
opportunities for members of the machine
learning community to contribute to healthcare.”

29. © 2020
Health
Catalyst
Clinical Text Data: Questionable Quality
29
In a typical note, 18% of the text
was manually entered; 46% copied;
and 36%, imported

30. © 2020
Health
Catalyst
EHR Documentation = Observed
Physician Behavior
30
• 38.5% of review of systems (ROS)
were confirmed (61.5% of the time, the
EHR data did not reflect reality)
• 53% of physical exams (PE) were
confirmed (47% of the time, the EHR
data did not reflect reality)
• Sept 2019
• UCLA, Stanford, UC Santa Cruz
Perception
Reality

31. © 2020
Health
Catalyst
49% of randomized clinical trails were
deemed high risk for wrong conclusions
because of missing or poor measurement
of outcomes data
The Importance of Outcomes Data
31
18 Sep 2019

32. Future State
What’s a better state look like?
How do we get there?

33. © 2020
Health
Catalyst
Enabling the Digital Healthcare Conversation
33
"I can make a health optimization recommendation for you, informed not
only by the latest clinical trials, but also by local and regional data about
patients like you; the real-world health outcomes over time of every
patient like you; and the level of your interest and ability to engage in
your own care. In turn, I can tell you within a specified range of
confidence, which treatment or health management plan is best suited
for a patient specifically like you and how much that will cost.”*
Between a physician and their patient… or patient and their avatar
*—Inspired by the Learning Health Community
We are parsing this statement for outcomes and cost data,
predictive analytics, machine learning, social determinants of
health data, recommendation engines

34. © 2020
Health
Catalyst
A National Healthcare Goal
34
By 2030, every citizen will possess
at least 10,000x more data, coupled with
analytics and AI, to support their health
optimization, than exists in 2020
In the US, that means going from 100MB to 1TB per year

35. © 2020
Health
Catalyst
35
Feb 2019

36. © 2020
Health
Catalyst
Microns-thin, one-inch skin-
pliable sensors with integrated
Bluetooth antenna, CPU,
physiologic monitors, and
wireless power
36

37. © 2020
Health
Catalyst
Rise of The Digitician and Patient Data Profiles
37
• Different patient types have
different data profiles required for
the active management of their
outcomes and health
• I’m not talking about quality
measures
• I’m talking about telemetry,
diagnostics and functional status
about the state of the patient, not
the state of healthcare processes
• It’s the Digitician’s job to
prescribe the right sensors and
proactively collect this data for
patients in their panel, and feed
the analytics of that to the care
team and patient

38. © 2020
Health
Catalyst
In Closing…
38
• Be humble healthcare… look for
role models, borrow concepts
and hire engineers from military,
aerospace, and automotive
• The volume and quality of
healthcare data is lower than the
hype would lead you to believe
• The good news: Much is left to
achieve, and transformation is
truly ahead in front of us

39. 39
Scott Weingarten, MD, MPH
Chief Executive Officer, Stanson Health

40. © 2020
Health
Catalyst
• IOM/NAM – 17-year gap
• Evidence-based care 50% of the time
– Female physicians have lower patient mortality
rates than male physicians
• 1/3rd of health care costs = waste
• In 1 hour….
– There may be approximately 28 deaths in the
United States because of medical errors
– There may be $22 million spent on medical
over-treatment
Opportunity
40

41. © 2020
Health
Catalyst
41
Medical Education: Information
Acquisition, Application
NIH research:
$39 billion in FY 2019
US medical research
budget 2016, $172
billion
Medical Research
Funding
20,000 biomedical journals
6,000 articles per day
1 article every 30 seconds
75,000 lab tests
878% health care data
growth since 2016
Doubling time of medical
information 73 days in
2020
Output
Brain
The Cloud
Point of Care

42. © 2020
Health
Catalyst
Changing Care
Predictors of Success Adjusted OR
Automatic provision of decision support as part
of workflow
112
Provision of decision support at the time and location
of decision making
15
Provision of recommendation rather than just an
assessment
7
Computer-based generation of decision support 6
Source: Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical
decision support systems: a systematic review of trials to identify features critical to success. BMJ.
2005 Apr 2;330(7494):765. PMID: 15767266
42

43. © 2020
Health
Catalyst
CDS 1.0
43
Note: CDS alert displays using EHR’s native best practice alerts; EHR’s do not allow use of actual screenshots
Physician
starts order
in EHR
Likely
appropriate
Order
placed
Inappropriate
order cancelled
Likely
unnecessary
Logic
evaluates
30+
elements
Choosing Wisely: Don't perform population based
screening for 25-OH-Vitamin D deficiency. 1, 2, 3
(American Society for Clinical Pathology)
Reasons for override:
malabsorption syndromes
meds (glucocorticoids,
antifungals, etc.)
diet excludes dairy
products
monitoring of known
vitamin D deficiency
dark skin complexion
Hyperlink: Choosing Wisely – American Society for Clinical Pathology
Information for Patients: Vitamin D Tests (ASCP)
Comments:
see comments
remove order keep order

44. © 2020
Health
Catalyst
CDS 1.0
44
>$400,000
savings/yr.
Analytics July 6 – Aug 3, 2019
High Cost Lab Reminder >$500

45. © 2020
Health
Catalyst
“Making it easier to do the
right thing, harder to do the
wrong thing”
45

46. © 2020
Health
Catalyst
Comprehensive CDS
46
Update to
Practice
Standards for
Electrocardiograp
hic Monitoring in
Hospital Settings:
A Scientific
Statement From
the American
Heart
Association.
Circulation 2017;
Oct 3:[Epub
ahead of print].
Standards for Inpatient
Electrocardiographic
Monitoring
Oct. 04,
2017
$3.7M
Annualized Savings
“Hard Green”
1. Identified cardiac
monitoring/telemetry
in order sets &
preference lists
2. Utilized BPAs to
guide practice
3. No change in
mortality, rapid
response times or
code blues

47. © 2020
Health
Catalyst
Order Sets and Preference Lists
47

48. © 2020
Health
Catalyst
Order Sets and Preference Lists
There was nearly a 50% increased odds of
dementia associated with total anticholinergic
exposure of more than 1095 TSDDs within a
10-year period, which is equivalent to 3 years’
daily use of a single strong anticholinergic
medication at the minimum effective dose
recommended for older people
Analytics, 1/1/2019-7/28/201948

49. © 2020
Health
Catalyst
References:
1. Arch Intern Med. 2009 Nov 23;169(21):1952-60. doi: 10.1001/archinternmed.2009.357.
2. Neuroepidemiology. 2016;47(3-4):181-191. doi: 10.1159/000454881. Epub 2016 Dec 24.
3. Pharmacoepidemiol Drug Saf. 2010 Dec;19(12):1248-55. doi: 10.1002/pds.2031. Epub 2010 Oct 7.
Additional sources: https://www.hcup-us.ahrq.gov/db/vars/totchg/nisnote.jsp
*Cancellation of inappropriate benzo or sedative-hypnotic order
Choosing Wisely (American Geriatrics Society):
“Don’t use benzodiazepines or other sedative-hypnotics in older
adults as 1st choice for insomnia, agitation or delirium.”
CDS Intervention #92 AMB live, #119 INPT live
Prevented
35 falls1
63 dementia2
10 hip fractures3
723
follows*/year
Based on HFHS data:
CDS analytics 07/01/18 – 06/30/19
Current Benefits: Harm Avoided
49

50. © 2020
Health
Catalyst
Peer-comparison Feedback
Randomized controlled trial
Low value care - Antibiotics for URIs
• 248 providers
• 14,753 patient visits
Results
• Control (24.1% to 13.1%)
• Peer-comparison feedback (19.9% to 3.7%,p<0.001)
1 year later
• Drift
• Peer-comparison feedback still has some impact
Reinforcing Medical Education
50

51. © 2020
Health
Catalyst
5151

52. © 2020
Health
Catalyst
Affiliation % of potentially
low-value care (n=47)
Physician 1 62%
Physician 2 5%
52

53. © 2020
Health
Catalyst
53
CDS 2.0 Avoiding
Alert Fatigue and
Increasing Impact

54. © 2020
Health
Catalyst
54
Decision Support 2.0

55. © 2020
Health
Catalyst
5555
Decision Support 2.0

56. © 2020
Health
Catalyst
Decision Support 2.0
56

57. © 2020
Health
Catalyst
• Patient EHR structured data often
incomplete, fragmented, and sometimes
erroneous1
• Recent survey of U.S. hospitals with
advanced EHRs, about 35 % of clinical
data was captured in structured format,
65% in unstructured text1
• Use of 1/3 of available clinical data can
cause false-positives and missed
opportunities
• More effective CDS will require NLP, ML,
and AI
CDS 2.0
1 Meystre SM, Lovis C, Bürkle T, Tognola G, Budrionis A, Lehmann CU. Clinical Data Reuse
or Secondary Use: Current Status and Potential Future Progress. Yearb Med Inform.
2017;26(1):38–52. doi:10.15265/IY-2017-007
57

58. © 2020
Health
Catalyst
Version 2.0, CDS Requires NLP, ML, and AI
58

59. © 2020
Health
Catalyst
NLP, AI
59
Fact extraction
Extract clinical facts
from unstructured EHR
elements
Fact inference
Infer additional clinical
facts from the entire
EHR record
Return CDS
Recommend
ation
✓
CDS scoring
Determine the
appropriateness of a
provider’s action
codified
facts
“… to r/o pulmonary embolism …”
pregnancy status
yes
no
patient record
✓
✓
✓
Physician
signs order

60. © 2020
Health
Catalyst
Need NLP, AI
60

61. © 2020
Health
Catalyst
Trajectory
61

62. © 2020
Health
Catalyst
62

63. © 2020
Health
Catalyst
• CDS that includes interpretation of free text should provide more
effective CDS with less fatigue
• Evolution…
– 2019 - Clinical, physiological, lab, images, patient preferences, etc.
– 2020 - Clinical, physiological, lab, images, patient preferences, social
determinants, genetics, proteomics, microbiome, etc.
CDS
63

64. © 2020
Health
Catalyst
Based upon Mr. Jones’ genetic profile, microbiome information,
symptoms, vital signs, laboratory values, personal preferences,
social determinants…
The Future
What tests and treatments
are appropriate for Mr.
Jones?After review of…
Mr. Jones, I would
recommend the
following….
64

65. © 2020
Health
Catalyst
• Evidence-based care 50% of the time
– Female physicians have lower patient mortality
rates than male physicians
• 1/3rd of health care costs = waste
• During the hour that we are spending
together today
– There may be approximately 28 deaths in the
United States because of medical errors
– There may be $22 million spent on medical
over-treatment
Opportunity
65

66. © 2020
Health
Catalyst
66

67. Q&A
67
Scott Weingarten, MD, MPH
Chief Executive Officer, Stanson Health
Dale Sanders
Chief Technology Officer, Health Catalyst

68. Thank You!