Expanding AI in Healthcare: Introducing the New Healthcare.AI™ by Health Catalyst

© 2021 Health Catalyst
© 2021 Health Catalyst
Jason Jones, PhD, Chief Analytic and Data Science Officer
Introducing the New
Healthcare.AI™ Offering

Agenda
• Why Healthcare.AI now?
• How can you use this framework?
• What can you do by next Tuesday?
© Health Catalyst. Confidential and proprietary.

Why Now?
• Increasing business critical decisions and stakeholders needing faster turnaround and
smaller margins for error—data and compute should help
• Application of AI often less than successful
• Struggled to integrate AI into current tools, change workflows, and demonstrate a
positive impact
• First release of healthcare.ai focused on transactional predictive models, pleased many,
lacked guidance to be effective, and too narrow in scope
• New Healthcare.AI™ offering dramatically broadens the use and use cases of AI within
health and healthcare organizations

The New Healthcare.AI™ Framework

The New Healthcare.AI™ Framework
The New Healthcare.AI™
Framework expands the use of
augmented intelligence (AI)
healthcare. The framework:
• Helps you choose/build
transactional predictive
models (Level 2)
• Expands the utility of AI
through optimizing selection
and application to complex,
system-wide, business-critical
issues (Level 3-5)
• Expands the number of
potential AI users by applying
AI to the most common
analytics uses cases (Level 1)
+ indicates augmented/enhanced standard analytic approaches

New Healthcare.AI™ helps you get better answers to your common
analytics questions
How might we improve in the future?
• How can I find the best way to follow up with patients
after discharge?
How did we improve over time or against peers?
• Did we reduce readmissions over the time?
• How does our readmission rate compare to Hospital B?
When and how should we deploy a predictive
model?
• Does this readmission prediction model make sense?
• When should a case manager reach out to a patient?
Help me choose, tailor, or build a predictive
model to answer…
• Which patients are at greatest risk for readmission?
• Which patients are more likely not to complete their
post-discharge follow-up visit?
Where are we and what is our trend and
timeframe to show improvement?
• What is our past readmission rate and where is it going?
• How long might it take to demonstrate improvement?

How Our New Healthcare.AI™ Offering
supports The Healthcare.AI™ Framework
New Healthcare.AI Expert Services
A flexible set of expert data science
services that can be tailored to fit the
amount of guidance—and level of advanced
AI capabilities—you need.
Our team of data science experts can help
your organization find and build your own
data science expertise, provide guidance on
a short-term project basis, or embed a data
scientist to help with longer-term efforts.
New Healthcare.AI Product Suite
(Level 1)
An integrated suite of self-service tools that
expand the use of AI with easy one-click
access from within existing BI tools and
Health Catalyst accelerators and
applications.

Learn About Healthcare.AI and Our Framework
https://healthcare.ai/

after discharge?
model?
model to answer…
Level 1: Analytics Integration

Poll #1
Which of the following techniques has your
organization used to guide decisions in the last
month? (Select all that apply):
• Cluster Analysis – 30%
• Forecasting – 70%
• Predictive Models – 60%
• Statistical Process Control (SPC) – 30%
• Statistical Tests of Group Differences – 20%

Poll #2
Which of the following techniques would you
expect your BI developers and analysts to be
able to apply? (Select all that apply):
• Cluster Analysis – 52.87%
• Forecasting – 64.37%
• Predictive Models – 73.56%
• Statistical Process Control (SPC) – 41.38%
• Statistical Tests of Group Differences –
41.38%

New Healthcare.AI™ Analytics Integration
Problems to solve:
• Business: How do we achieve correct, consistent, and
transparent insight from data?
• Technical: Visualization tools are aesthetically pleasing but
lack interpretive guidance
Solution:
• Healthcare.AI™ Product Suite
• Provides easy, embedded, one-click AI via API calls across
BI tools, analytics accelerators, and Health Catalyst apps
• Pre-introduction…7 clients in PROD, 14 exploring, and >5K
hits per day
Capabilities:
• Statistical Process Control (SPC+): identify outliers, trends, and
level changes
• Time Series Outlier Detection: identify outlying points despite
changes in levels, trends, and seasonality
• Forecasting: project future performance using econometric
time series and regression techniques
• Time Series+: combines SPC, time series outliers, forecasting,
and power analysis in a single capability
• Forest Plots+: understand and manage performance, clusters,
and future direction across the system
• COVID-19: situational awareness based on infection counts
and growth rates
Support:
• Health Catalyst Analytic Accelerators and applications
• BI Tools: PowerBI, Qlik Sense, QlikView*, Tableau
* Not all capabilities will be available in QlikView

Where You Had A Line Chart…
• See variation in AMI performance
over time
• How does past performance
compare to risk adjusted norms?
• Which points are outliers?
• Is there a decrease in performance
recently?

One-Click Gives You Access To New Insight (Time Series+)
• Clearly see increase in observed to
expected mortality in July 2020
• Associated with 3rd COVID wave
• Is there any evidence performance
will return on its own?
• What if we knew there was
seasonality or other level/trend
shifts?
• How long would it take to
demonstrate a statistically
significant improvement?

Where You Had a Dot or Bar Chart…
• See variation across providers
• Immediately focus on lowest and
highest performers
• Which providers really have different
performance?
• What is their trajectory?
• Where can we learn and what can
we change?

One-Click Gives You Access To New Insight (Forest Plots+)
• Top performer is a super star
• Bottom performer is low volume and
tied with 3 other “low performers”
• Providers “O” and “V” are moving in
opposite directions
• Key driver was EHR documentation
training

after discharge?
model?
model to answer…
Level 2: Choosing/Building
Predictive Models

Poll #3
How many predictive models have been
published for COVID-19:
• 1-5 – 14.47%
• 6-20 – 9.21%
• 21-50 – 17.11%
• 51-100 – 6.58%
• 100+ – 52.63%

Predictive Models: Building Them Is Not the Problem
• Process of building predictive
models is largely automated
• Defining the problem to solve,
collecting the data, and
calibration to the population and
organization remain challenges
https://www.bmj.com/content/369/bmj.m1328

Many Curated Predictive & Prescriptive Models Are Available
Borrow Before You Build:
• Literature
• Sites/apps like https://www.mdcalc.com/
• Well regarded by clinicians
• Often come with specific guidance
• Sometimes come with society backing and/or
evidence of utility (e.g., pneumonia guidelines)
• Choosing a simple model like LACE for
readmission can help you leverage what others
have learned

Poll #4
How many curated calculators/models are
publicly available and on our site?
• <100 – 31.37%
• 100-199 – 15.69%
• 200-500 – 19.61%
• 501-1000 – 15.69%
• 1000+ – 17.65%

Optimizing Predictive Model Utility
Building/Tailoring Predictive Models
• Achieve: Cycle of
improvement,
adaptation, and
success.
• Levers: Failure ID,
Telemetry, Research
methods.
• Achieve: Usable tool to
support change.
• Levers: DOS, Closed
Loop, Care
Management, Patient
Safety Monitor, client
tools, etc.
• Achieve: Clarity and
confidence to lead
change.
• Levers: Model
understanding.
• Achieve: Most efficient
machine to do the job.
• Levers: Feature
importance, selection,
and training.
• Bonus: Enhance data
quality.
Optimize:
Technical and
Substantive
{PDSAs}
Deploy:
Workflow
centered
{UI/UX}
Decide:
Functional,
Operational, &
Contextual
{Judgment}
Build:
Exploration,
Simplification,
and Selection
{Algorithms}
Define:
Populations,
Outcomes, and
Features
{Engineering}
• Achieve: Agreement on
the problem(s) and
success statements.
• Levers: Statistical
process control,
variation analysis,
forecasting, and
SMART goals.

Building Predictive Models
Goal: Identify members at greatest risk for
frequent ED visits in the next 90 days for
care management outreach
• Routinely have 100s of possible features
• Recent ED utilization, comorbidities,
utilization scores, demographics, social
determinants, labs, … many overlap
• Engage leaders in users in identifying
anything they think might matter
• Make it easy to consider more
• Then build the most efficient model they
will buy into
Members Likely To Have Frequent Emergency Department Visits in 90 Days

We Recommend Tailoring or Building Predictive Models To Fit You
Care Management Use Case…
• Features: Similar but different (e.g., claims vs social determinants vs labs)
• Population Differences: Medicaid in the Southwest vs mixed payers in Northeast
• Technical Differences: “Ranger” is the worst and the best
• Calibration and testing are good ideas—the models almost come for free!

after discharge?
model?
model to answer…
Level 3: Optimizing
Predictive Models

Optimizing for Predictive Model Utility
Original Ask: Help us reduce 90-day inpatient
admissions.
What We Saw: Care managers didn’t trust the
existing model and felt it wasn’t capturing some
critical components.
Recommendation: Rebuild the model with user and
leader trust as a key design principle and formally
evaluate additional data sources.
Tools: Should be readily available to analysts to
leverage with initiative leaders:
• Explainable AI.
Processes: Scriptable.
Population Health Management Example
Miliard, M. (2020). Christiana Care offers tips to ‘personalize the black box’ of machine learning. Healthcare IT News. Retrieved June 19, 2020,
from https://www.healthcareitnews.com/news/christiana-care-offers-tips-personalize-black-box-machine-learning

Optimizing Predictive Model Utility
3 Levels of Predictive Model Understanding—Whether You Have Built or Borrowed
• Achieve: Cycle of
improvement,
adaptation, and
success.
• Levers: Failure ID,
Telemetry, Research
methods.
• Achieve: Usable tool to
support change.
• Levers: DOS, Closed
Loop, Care
Management, Patient
Safety Monitor, client
tools, etc.
• Achieve: Clarity and
confidence to lead
change.
• Levers: Model
understanding.
• Achieve: Most efficient
machine to do the job.
• Levers: Feature
importance, selection,
and training.
• Bonus: Enhance data
quality.
Optimize:
Technical and
Substantive
{PDSAs}
Deploy:
Workflow
centered
{UI/UX}
Decide:
Functional,
Operational, &
Contextual
{Judgment}
Build:
Exploration,
Simplification,
and Selection
{Algorithms}
Define:
Populations,
Outcomes, and
Features
{Engineering}
• Achieve: Agreement on
the problem(s) and
success statements.
• Levers: Statistical
process control,
variation analysis,
forecasting, and
SMART goals.

after discharge?
model?
model to answer…
Level 4: Retrospective
Comparison

A Tale of Two Examples
• “Scared Straight” started in 1978.
• Teenage delinquents spend 3 hours
in prison.
• Stories of how delinquents changed
course because of the intervention.
• Cochrane review of 9 rigorous
studies: 2 show no impact; 7 show
negative impact.
• To improve academic attendance and
test scores in Kenyan kids, ICS
provided:
o More books (was 1 per class)
o Flipcharts (to tailor lessons–E3L)
o More teachers
• Each intervention showed no impact
upon careful study.
• Deworming:
o 25% decreased absenteeism
o 20% increased income 10 years later
Nothing To Fear Books and Worms
https://www.effectivealtruism.org/doing-good-better/
https://blogs.sciencemag.org/books/2017/04/03/a-journalist-shines-a-harsh-spotlight-on-biomedicines-reproducibility-crisis

What Should the Market Buy?
https://www.aetv.com/shows/beyond-scared-straight (Mar 2021)
https://www.cochrane.org/CD002796/BEHAV_scared-straight-and-other-juvenile-awareness-programs-for-preventing-
juvenile-delinquency

Retrospective Comparison
What We Wondered: Do higher volume providers
deliver better outcomes?
What We Worried: Higher volume providers often
are referral centers and get more difficult cases. It’s
not feasible to randomize patients to providers.
Conclusion: In the absence of other information
about the quality of surgery at the hospitals near
them, Medicare patients undergoing selected
cardiovascular or cancer procedures can significantly
reduce their risk of operative death by selecting a
high-volume hospital.
What We Needed: Careful study design and risk
adjustment to conduct the best possible
retrospective analysis.
Risk adjustment often can be done via predictive
modeling.
Consolidating Services
Figure 8: Adjusted In-Hospital or 30-Day Mortality among
Medicare Patients (1994 through 1999), According to Quintile
of Total Hospital Volume.10
Birkmeyer, J. D., Siewers, A. E., Finlayson, E. V., Stukel, T. A., Lucas, F. L., Batista, I., Welch, H. G., & Wennberg, D. E. (2002). Hospital volume and
surgical mortality in the United States. The New England Journal of Medicine, 346(15), 1128–1137. https://doi.org/10.1056/NEJMsa012337

Prescriptive Optimization
We Knew (2011)11: Dealing with social
determinants of health for high-risk patients
could reduce hospitalizations.
We Learned (2020)12: High-risk patients did
experience decreased hospitalization—but
there was no difference between treatment
and control groups…and now we’re trying
refined approaches.
We were able to learn only because we had
the right study design up front.
The same has happened with intensive
monitoring in severe sepsis or septic shock and
tight glucose control.
Hot Spotting
Gawande A. (2011). The hot spotters: Can we lower medical costs by giving the neediest patients better care? New Yorker (New York, N.Y.),
40–51. Finkelstein, A., Zhou, A., Taubman, S., & Doyle, J. (2020). Health care hotspotting – A randomized, controlled trial. The New England
Journal of Medicine, 382(2), 152–162. https://doi.org/10.1056/NEJMsa1906848

Achievable and Applicable
Original Ask: Help us show identify patient at risk for a bad outcome if they acquire COVID-19.
What We Saw: Prediction was easy, but much uncertainty about what to do with strained resources.
Recommendation: Members randomized to care manager and phase of outreach.
Tools: Randomization can be simple and simplifies analysis.
Needed: Clarity, curiosity, and leadership.
COVID-19 Vulnerability

Poll #5
Would you like to learn more about Health
Catalyst’s products and services?
• Yes
• No

The New Healthcare AI™ Framework
The New Healthcare.AI™
Framework expands the use of
augmented intelligence (AI)
healthcare. The framework:
• Helps you choose/build
transactional predictive
models (Level 2)
• Expands the number of
potential AI users by applying
AI to the most common
analytics uses cases (Level 1)
• Expands the utility of AI
through optimizing selection
and application to complex,
system-wide, business-critical
issues (Level 3-5)

Expanding AI in Healthcare: Introducing the New Healthcare.AI™ by Health Catalyst

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Expanding AI in Healthcare: Introducing the New Healthcare.AI™ by Health Catalyst

Similar to Expanding AI in Healthcare: Introducing the New Healthcare.AI™ by Health Catalyst (20)

More from Health Catalyst

More from Health Catalyst (20)

Recently uploaded

Recently uploaded (20)

Expanding AI in Healthcare: Introducing the New Healthcare.AI™ by Health Catalyst

Editor's Notes