The Why And How Of Machine Learning And AI: An Implementation Guide For Healthcare Leaders

© 2017 Health Catalyst
Proprietary and Confidential
An Implementation Guide for
Healthcare Leaders
The Why and How of
Machine Learning and AI

Eric Just
• SVP at Health Catalyst in Product Development
• Started career in healthcare at Northwestern University
• Genomics
• Clinical Data Warehouse
• Health Catalyst
• Early platform vision
• Sales
• Client Operations
• Product

Ken Kleinberg
• VP of Research at Chilmark Research, covering
• Analytics, AI, machine learning, mobile/wireless, IoT, EHRs, medical
devices…
• Healthcare transformation, PHM, consumerism, payer-provider
convergence…
• 38 years in IT – last 20 in healthcare, including
• Managing Director, Advisory Board
• VP of Bus Dev, Health Language/Wolters Kluwer
• VP and Hospital Strategist, Allscripts
• Senior Director of Global Health, Symbol/Motorola/Zebra
• VP and Editor-in-Chief of Healthcare, Gartner

Learning Objectives and Agenda
1) What Are Machine Learning (ML) and
Artificial Intelligence (AI) and Why
Should I Care
• Getting past the hype to real definitions
• Understanding use cases and benefits
2) How Do I Implement ML and AI in My
Organization
• Practical steps, resources, costs,
timeframes, skills, and vendors
• The importance of data management
and an analytics platform
• Real-time data and moving targets
3) Dealing with Challenges
• Change management, regulation,
liability, human touch
4) What Does the Future Hold for ML
and AI (and for My Strategy)
• Job augmentation and replacement,
taking on greater challenges, scary (yet
bright) futures
What You Need to Know and Do

Executive Summary
• Machine learning and the broader area of AI, inspired by biology and evolution to learn and
adapt, have long histories.
• Use cases in healthcare include improved predictions, patient clustering and classification, early
detection/diagnosis, scheduling, medical image recognition, and natural language processing
(NLP)/text mining.
• Practical steps for getting started with ML and AI include: goal setting, allocating resources,
establishing realistic timelines, costs, and benefits; managing complex data sources; running models;
measuring results and taking action/operationalizing.
• Existing investments in data management and analytics, as well as processes to identify needs and
create operational decision models, give organizations an advantage.
• Typical challenges include change management, regulation, liability, and concerns about human
“touch.”
• The future of ML and AI include increased job augmentation and replacement, tackling more complex
problems (e.g., genomics), and the need to ensure these technologies are applied with the proper rigor
and ethical oversight.
5

What Are Machine
Learning and AI and
Why Should I Care?

Understanding the Hype in Machine Learning and AI
• World-leading companies have invested billions into voice assistants, facial recognition, shopping
assistants, and semi-autonomous vehicles.
• There has been a media frenzy over AI and job displacements, disruptions, and dangers – the rise of
the intelligent machines!
• It is difficult to discern the degree of healthcare vendor investments in these technologies and the
sophistication of their solutions.
• To more fully appreciate the opportunities and challenges requires some understanding of what
machine learning and AI are.
7

Machine Learning and AI Definitions
* ML and AI draw and
advance approaches and
principles from traditional
statistics and mathematics,
rule-based systems, and
biological systems to create
solutions that can learn and
adapt.
* They reduce the effort
required by humans to
manually choose algorithms,
data sources, model
parameters, and steps in
order to test hypotheses,
discern knowledge, and
achieve goals.
8
Rules,
inference
engines,
symbolic
reasoning,
programming
Biology, parallel
processing,
genetics, evolution
Machine
Learning
and AI
Math,
statistics,
probability
theory,
analytics

Clarification of Definitions
Artificial intelligence refers to systems or machines that can adapt and learn from data and
their environments to exhibit the intelligent behavior found in living things.
Machine learning is a subset of AI with its origins in math, statistics, computer science, and
biology (e.g., neural networks, which simulate networks of neurons in the brain). Its models
adjust themselves when presented with training data. This can be:
• Unsupervised (meet an algorithmic goal, such as clustering data into discernible groups)
• Supervised (the system is given labeled training data and adjusts itself to provide the right
answer)
Deep learning is a more sophisticated form of ML that incorporates multiple layers (e.g., of
neurons) capable of pre-processing and filtering, to model more complex input and
relationships.
Cognitive computing attempts to model human thought processes, including generating
ranked hypothesis.
9

Some Additional Definitions
Expert systems refer to rule-based reasoning systems that either attempt to determine
which possible answers are supported by data (backward chaining) or start with known
information and work towards reaching a viable answer (forward chaining) – rules can be
used to apply known knowledge and context to ML and AI.
Descriptive, predictive, and prescriptive analytics are increasingly valuable approaches
to determine what has happened (descriptive), what will or might happen (predictive), and
what to do about it (prescriptive).
Natural language processing uses a variety of ML and AI techniques to understand
human text or speech.
10

Tackling (Big) Data and Problem Complexity
11
The volume, variety, and velocity of information can far exceed a care professional’s abilities
Social
determinants
of health
Microbiome
Environmental
data
Genomic data
Medical and
family history
Patient-
generated
health data
Population
health data
Resource
availability (care
facilities, clinicians,
medications)
Policy and
regulation
Insurance
coverage
Medical
knowledge
Patient Journey
Digital images,
lab tests

What to Apply Where and Why
Approach Best For Positives Risks
Statistics
• Good assumptions to start with
• Standard models sufficient
• Clean (structured) data
• Skills and mature tools often
already exist in the org
• Acceptance by org
• Techniques may not be powerful enough
for difficult problems
• Models may not transfer well to real and
changing world
• Lots of manual effort required
Machine
Learning
• A hypothesis to test or pattern to
recognize
• Some assumptions to start with
• Many models to try
• May be able to utilize some
existing organizational skills
• Removes some of the manual
burden of model development
• Can identify most impactful
input features
• Models may appear to do well but be
simply “memorizing” training data
• Models may not be sophisticated enough
for more difficult problems
Deep
Learning
• A difficult problem to solve
• Complex interactions
• Messy and/or huge amounts of
data
• A “black-box” answer will do
• May be able to provide answers
or predictions difficult to
discern by any other method
• Skills and tools may not exist in the org
• Users may require more specific
explanation for model results
• Even large datasets may not carry
enough info for the most sophisticated
systems to find insight
Expert
System
• Extensive domain knowledge
available
• Processing order not predefined
• Reasons for answers must be
apparent
• Answers can be traced to
which rules have “fired”
• Organizations can agree on
rules or customize locally
• Too many rules (over a few thousand) can
be difficult to manage/curate
• Has challenges with conflicting data
12

ML and AI Use Cases
• Pattern recognition
• Clustering
• Prediction
• Diagnosis
• Scheduling
• NLP
• Smart network
• Robotics
• Facial recognition
• Consumer buying patterns
• Financial market trends
• Car problems
• Train schedules
• Virtual assistant
• Internet of things (home)
• Assembly line
• Tumor type
• Disease variations
• Length of stay, readmissions
• Cancer care
• Surgery and rehab
• Chart abstraction
• Internet of things (hospital room)
• Medication delivery
13
Type of Goal Cross-Industry Example Healthcare Example

Setting Expectations While Accelerating Exponentially
14
Useful
Too good?
Promising
Essential
No way Geeky/Niche
Advances have a way of creeping up
on us – one day a joke, another day
useful, and all of a sudden, Wow!
We are here
with AI
Smart
Phones
Self-driving cars
Virtual Assistants
GPS
“Doc in a box”
Human brain
interface
Internet

How Do I Implement
Machine Learning and
AI in My Organization?

Practical Steps
Major Considerations include:
16
Strategy – Ensuring
that ML and AI are
applied to the rights
kinds of problems for
the right
organizational goals
Resources &
Infrastructure –
Determining and
implementing analytics
platforms, data
warehouse capabilities,
tools to access data,
and security required
to succeed
Funding –
Distributing
reasonable costs for
ML and AI
infrastructure and
initiatives across
relevant
stakeholders
Timeframes and
Benefits –
Determining how
long ML and AI
initiatives will take
and what are the
expected benefits
Skills – Ensuring
the rights kinds of
skills for project
scoping, sourcing
decisions, model
development, and
operational changes
Vendors, Partners,
and Tools –
Choosing which
vendors and
partners will be able
to provide the tools
and skills to help

Approach to Strategy for ML and AI
• A balanced approach supported and visible to executives
and the board, desired or driven by (or they are at least
willing to give it a chance) the departmental level (clinical,
financial), and enabled by IT
• Incorporated into the overall organizational strategy
and initiatives (PHM, ACO, Quality, Consumerism)
• Pursued within the process of operational
governance to ensure priority, funding, ownership,
results, P&L
• Nurtured with a staged and natural progression
growth that considers start-up costs, sourcing
of skills and resources, knowledge transfer,
risk, and scale
17
• Driven totally from the top or
totally bottom up
Good Worrisome
• Considered an IT project with no specific
links to business or clinical initiatives
• Politics, lack of transparency, lack of
accountability
• Big bang, shoot for the moon, high expense

Flexible and Scalable Infrastructure
Matching to an Organization’s Needs and Capabilities
18
Turnkey application
(simpler use, more
reproducible)
Configurable application
(varied needs, controlled
by business user)
Customizable application
(specific needs, pilots)
Typical Adoption Progression
these can be
incrementally added
multiple approaches
– mix and match
full leveraged
environment to support
many needs
web-based decision
support
ALGORITHM TOOLKIT BI PLATFORM
WORKFLOW
INTEGRATION

Real-Time Data and Moving Targets
• One of the challenges with any prediction, decision support or analytics solution is its applicability/accuracy as
input sources and situations change.
• Should the system see something it was not trained on, depending on the influence/combinations of the inputs, the
system might be able to come up with an accurate answer, semi-accurate, or it could be completely wrong.
• ML and AI systems, by definition, should be readily adaptive to new training data and new situations.
• It may be possible to monitor/identify different “alert levels” – that is, operating within a data stream thoroughly
trained on, working with a different yet similar set of data, or working with data outside the training range.
• Use of real-time data platforms can help ensure that systems are continually retrained on relevant data for
successful operational deployment.
19
Retrospective
training data
Good for initial
model development
Real-time
training data
Needed for operational
model deployment

The Cost Realities of Machine Learning and AI
20
Organizations
should consider
change
management and
cultural costs
associated with
implementing ML
and AI
Costs are mostly
incremental for
organizations that
have already
invested heavily in
BI/Analytics and are
using that data to
affect improvements
Organizations that
have not gone this
route will have much
higher hills to climb
The most important
first step is to use
data effectively, even
at a basic level, to
begin making
decisions

Timeframes and Benefits
21
What will you do if benefits are not (completely or substantially) met??
Are you making
interim
measures at key
points?
Do you have
buy-in from
stakeholders
about the
benefits?
Are you
providing a
range of
(compelling)
expected
benefits?
Do you have
baseline
measurements
to compare
results?
Are you setting
short, medium,
and long-term
goals?
Are you
measuring
benefits in terms
of costs, times,
quality, or other
measures
(precision vs
recall)?

Skillset Sourcing for Implementing ML and AI
Job Function Description Internal External
Clinical Identifying uses cases, discerning valuable outcomes, dismissing wrong/dangerous results, rallying support **** *
Financial Recognizing opportunity costs, spreading funding across relevant stakeholders, evaluating ROI **** *
IT Integrating data sources and data warehouse systems, monitoring performance, sourcing/hosting, upgrades *****
Security Ensuring that data sources are properly secured *****
Project Mgmt. Setting realistic timeframes, meeting resource requirements, and conveying progress to stakeholders **** *
Bus Dev Identifying and contracting with suitable partners, data sharing, consortiums, co-development *** **
Marketing Ensuring visibility to achievements for purposes of organizational reputation and patient recruitment ** ***
ML/AI Choosing the right algorithms and modeling approaches, running the models * ****
BI/Data Science Ensuring value add, integration with existing techniques, dashboard design ** ***
User Support Education, training, mentoring, answering questions, forwarding on more difficult problems, tracking concerns *** **
Legal Minimizing regulatory risks, meeting FDA requirements, ensuring HIPAA security, vetting business partners **** *
Operations Minimizing disruption, scheduling, meeting reporting period requirements *****
22

Considerations – Does Your Team Know the Answers?
AI and ML Experience
• Which ML algorithms and AI approaches (or combinations) are you going to pick?
• Do you have experience with descriptive, predictive, and prescriptive analytics?
• Has your team or your vendor any experience (and success) with competitions such as Kaggle?
• Does your team have access to online courses to increase their expertise?
Hardware Resources
• Do you have the processing power (GPUs) and memory to run the models with the amount of data necessary?
• Have you considered cloud and virtualization?
Data Sets
• Are you able to identify and have access to relevant data sources?
• Do you have access to adequate sample datasets?
• To what degree will you be attempting to transform/analyze unstructured data?
Vendors and Products
• Will you be using a turnkey solution, a configurable application system, or an application you have to customize or develop
(and later support)?
• Will you be using a proprietary commercial solution, or open source? If using open source, what support is available from who?
• To what degree do you expect to use or need consulting services?
23

Questions to Ask Vendors of ML and AI
Issue Description Concern
ML and AI
Experience
How long has the vendor been using ML and AI?
Do they know how to use it? Can they explain
how it works?
BI Platform Does the vendor have a platform for BI and analytics?
Is it a modern architecture built with software
engineering principles?
Healthcare
Experience
Does the vendor have relevant/dedicated healthcare experience (with similar
organizations)?
Will you be teaching them about healthcare?
ML and AI Source
Are the ML and AI capabilities developed by the vendor, commercially available, or
open source?
Proprietary lock-in?
Compelling
Difference
Can the vendor demonstrate a compelling difference ML and AI make to its solution?
Is it worth it?
Flexibility
To what degree is the vendor able to switch in new ML and AI capabilities or
customize their solution?
In love with their hammer? – what does their
roadmap look like?
Integration with
EHRs
What approach and success does the vendor have with integrating (or competing)
with major EHR vendors?
Will there be cooperation?
Scalability
Are there any scalability issues regarding the amount of data, processing times,
database structure (e.g., Hadoop), hardware requirements?
Will you hit these limits?
Business Model
How does the vendor charge for their software, hosting, and services capabilities
(over time)?
Do they price the way you want?
Self-Sufficiency Does the vendor offer tools or education to help make you self-sufficient?
What skill levels will be required to use this
solution?
24

Dealing with Challenges

Typical Challenges
It would seem that after identifying the use cases and technologies, and then sourcing, developing
and deploying the models, an organization would be home free. As hard as it might be to succeed on
those fronts, there are additional challenges that are likely to be faced that could be more difficult
to address and risk success of the system and worse.
These include dealing with:
26
Change
Management
AI and ML systems may
be discovering results,
exposing weakness,
taking away people’s jobs
and more – often at a
pace faster expected – as
in any transformative
project, a focus on change
management is key.
Regulation
As ML and AI take on
more complex situations,
the may come under
increasing regulatory
scrutiny.
Liability
As ML and AI become
more autonomous, a
question arises as to who
is liable when things go
wrong.
Human Touch
As ML and AI systems and
results touch more of our
lives, how can we ensure
a respectful and
compassionate interaction
while also balancing the
needs of systems and
society?

Change (Disruption/Transformative) Management
• People generally don’t like change – particularly if
their job, career, or ego is on the line.
• The bar is not always high enough to overcome
inertia – good enough my prevail.
• Intelligent automation often unveils inefficiency,
waste, fraud, abuse, bureaucracy.
• Once projects are underway, results may come
much faster than previously experienced.
• Fear over job loss can lead to lack of cooperation,
feet-dragging, passive aggressive behavior,
sabotage, or open hate (Johnson and Johnson’s
Sedasys for anesthesiology).
• AI can be risky since the mechanisms are often
not well understood, the skills and tools may be
unfamiliar, and the costs and benefits can be
uncertain.
27
How To Succeed
• Successful change management requires open and transparent communication, a clear vision of the future, instilling
belief that people will succeed under the new system, that there is no path backwards, and that everyone will benefit.
• Successful projects require good planning, adequate resources, leadership, governance, accountability, realistic
milestones, and the right tools and knowledge of how to use them (example, the right partners).
What to Expect

Regulation
• Privacy/Security - Because AI, BI, and analytics
systems are increasingly being used to deal with
complex data with many additional input sources,
security and privacy concerns may be higher
than with other systems.
• Safety – While it is clear that medical devices are
subject to a risk-based framework of regulation by
the FDA, clinical software applications (e.g.,
EHRs), analytic models, and clinical decision
support remain in a gray area.
• Oversight - As systems move from being
reference systems, to providing advice, to taking
(autonomous) action, the reasons for them to
come under regulatory control increase.
28
How To Succeed
What to Expect
• Ensure data operating systems are especially secure and able to manage multiple data sources.
• Understand who will be responsible for regulatory approvals (if any) from the start – recognize that some vendors
may tell you a system does not require approvals when it may.

How To Succeed
What to Expect
Liability
• Crossing Boundaries – It’s one thing for a machine to
carry out a mundane repetitive task, and another for it to
start making decisions that affect health and lives.
• With rule-based systems, the decisions can be traced
to specific rules that “fire,” and the responsible parties
for developing those rules are accountable.
• With ML and AI, it’s possible for predictions, answers,
and decisions to be made without a complete (or even
substantial) understanding of how the result was
reached.
• As ML and AI take on more complex tasks, its only
natural for people to be concerned about “who’s calling
the shots” and to what degree ethics or compassion have
been incorporated in the solution.
• Media attention to such issues as which obstruction
should a self-driving car hit if a crash is inevitable (school
bus or wheelchair).
29
• Health systems should ensure, to the degree possible, AI systems can be transparent to the decisions they make,
traceable to training sets, and accountable to human experts who deploy them – black boxes are a last resort.
• Consider initially picking use cases that provide advisory results or are non-clinical – they may be less “sexy,” but will
present fewer barriers and risks.

Human Touch – Moving Beyond Informing Decisions
One final (future) challenge concern about AI systems as they start to play an increasing role in people’s lives is to what
degree they can exhibit human touch vs. being (perceived as) cold and inhuman
Fortunately, its possible to have them exhibit compassion and understanding, perhaps with more consistency
than some humans, and to be attuned (via facial recognition, voice processing, historical data, etc.) to the emotional/behavioral
state of the user.
Given the nuance and complexity of human behavior, it’s more than easy to get this wrong, especially
if the subject is intentionally trying to derail or game the solution, or exhibits conflicting or new behaviors outside the
experience of the system.
On the scale of population health, care rationing, costs, etc., these become more organizational, cultural/societal issues. We’re
still some time away from machines being on equal footing with humans in deciding courses of action and the future.
30
How To Succeed
• Recognize that how users interact with a system and to what degree that system design shows a concern for
its results will become increasingly important as humans rely on these systems to a greater degree.
What to Expect

What does the Future
Hold for Machine
Learning and AI (and for
My Strategy)?

What Role Will AI Play in the Future?
32
Will depend on your type of job (tech, care manager, nurse, PA, physician) and your training
3-5 yearsToday Future
Minimal
experience
or skills
Most
experience
or skills
Medium
experience
or skills
Assist
Suggest
Annoy?
Super-charge
Assist
AssistSuggest
Super-charge
Replace

Dangers of AI
33
I think someone (or
something) is trying
to target our
genetics
I don’t know how to
do that anymore
There’s no future in
that career
anymore
Which of these two
systems should I
trust?
Over-reliance and
trust
Fewer new
professionals
enter the field
Competing
medical
knowledge
Loss of privacy,
exposure of
vulnerabilities

Advice
• Include machine learning and AI in short and longer-term strategic planning – look to apply these
approaches to gain tactical (labor saving), strategic (new markets), and transformational
(breakthrough) effects.
• Evaluate strategic options for introducing and expanding these technologies into the organization
• Determine to what degree and pace to build internal teams and skills (especially quants who
can identify and translate technology advances to clinical and business use cases), and to
what degree to enlist vendors and external experts as partners.
• Place increasing emphasis on first ensuring a solid BI platform to deal with more extensive and
complex data sourcing and management.
• Address change management – expect to deal with it at an increasing pace – especially in terms
of changing job functions and staffing requirements across the continuum.
• Continue to look for signposts in other industries and healthcare where advances are making a
difference and disrupting existing business models.
34

The Bright Future
35
Awesome collaboration of human and machine
Engineered health – genetics (remove disease)
Significantly reduced healthcare costs
More preventive care and treatment (nanobot repair)
Cyborg-capabilities -– implants (senses, motor function,
organs)
Limitless life – brain scans
Greater time for leisure, exploration

Thank You
Q&A with Ken and Eric

Healthcare Analytics Summit 17
Summit highlights
Industry Leading Keynote Speakers
We’ll hear from well-known healthcare visionaries. We’ll also
hear from two C-level executives leading large healthcare
organizations.
CME Accreditation For Clinicians
HAS 17 will again qualify as a continuing medical education
(CME) activity.
30 Educational, Case Study, and Technical
Sessions
We have the most comprehensive set of breakout sessions of
any analytics summit. Our primary breakout session focus is
giving you detailed, practical “how to” learning examples
combined with question and opportunities.
The Analytics Walkabout
Back by popular demand, the Analytics Walkabout will feature
24 new projects highlighting a variety of additional clinical,
financial, operational, and workflow analytics and outcomes
improvement successes.
Analytics-driven, Hands-on Engagement for
Teams and Individuals
Analytics will continue to flow through the three-day summit
touching every aspect of the agenda.
Networking and Fun
We’ll provide some new innovative analytics-driven
opportunities to network while keeping our popular fun run and
walk opportunities and dinner on the down.
Sept. 12-14, 2017
Grand America Hotel
Salt Lake City, UT
ERIC J. TOPOL
Author, The Patient Will
See You Now and The
Creative Destruction of
Medicine. Director,
Scripps Translational
Science Institute
DAVID B. NASH,
MD. MBA
Dean, Jefferson
School of
Population
Health
JOHN MOORE
Founder and Managing
Partner, Chilmark Research
ROBERT A. DEMICHIEI
Executive Vice President and
Chief Financial Officer, University
of Pittsburgh Medical Center
THOMAS D.
BURTON
Co-Founder, Chief
Improvement Officer,
and Chief Fun Officer,
Health Catalyst
DALE SANDERS
Executive Vice
President, Product
Development,
Health Catalyst
THOMAS DAVENPORT
Author , Consultant
Competing on Analytics*, ,
Analyitcs at Work, Big Data at
Work, Only Humans Need
Apply:Winners and Losers in the
Age of Smart Machines.
*Recognized by Harvard
Business Review editors as one
the most important management
ideas of the past decade, one of
HBR’s ten must-read articles in
that magazine’s 90-year history.

The Why And How Of Machine Learning And AI: An Implementation Guide For Healthcare Leaders

More Related Content

What's hot

Similar to The Why And How Of Machine Learning And AI: An Implementation Guide For Healthcare Leaders

More from Health Catalyst

Recently uploaded

The Why And How Of Machine Learning And AI: An Implementation Guide For Healthcare Leaders