The document discusses the benefits and challenges of building an in-house data science team at OSF Healthcare. It provides examples of projects the team has worked on, including readmission models, cost and utilization models, staffing simulation, and natural language processing. The benefits of an in-house team include higher performance, lower costs, full transparency and IP ownership. Challenges include internal resistance, implementation risks, and defining rules for model use. The document outlines the team's structure, basic project approach, and strategies for ongoing success, such as project intake processes and focus on foundational elements.

1. OSFHealthcareAdvancedAnalytics
The Benefits and Challenges of Building an
In-House Data Science Team
October 31, 2017

2. Agenda
• Introduction
• Getting Started with a Data Science Team
• Project Highlights
WHY

3. Introduction
Advanced Analytics at OSF Healthcare

8. • The dilemma is not about understanding the
population’s risk, but identifying WHO is at high risk
• Reduce internal costs through appropriate
intervention targeting
• Reduce non-value added work by appropriately
leveraging normally collected clinical data
• Drive additional value from our EPIC investment
Data Science and Predictive Modeling in Healthcare

9. • Business Understanding:
• Generate Charter and Project Plan (Goals, Team, Business Leader,
Data Sources Identified)
• Data Understanding:
• Collect, Describe, Explore and Verify Quality of Initial Data
• Data Preparation:
• Data Cleaning & Extensions
• Feature Engineering
• Model Set Formatting
• Modeling:
• Determine Appropriate Modeling Options for Desired End State
• Construct and Cross Validate Models
• Determine Best fit Solution
• Evaluation:
• Evaluate Results
• Determine Acceptance or Rejection for Production
• Deployment:
• Develop Implementation Project and Team
• Deploy Solution
• Support and Update Solution as Appropriate
15%
85%
Separate
Project
Representation of CRISP-DM Project Phases
Basic Project Approach

10. Key Success Factors

11. • Focus: Operational/Clinical Research (Predictive)
• Avg. Duration: 2-6 months in Dev.
• Output: Models for Workflow Incorporation &
Data Products for Proactive Planning
• Project Types: Predictive Models, Natural
Language Processing, Image Processing,
Simulation & Optimization, Custom Algorithm
Development
• Focus: Operational/Clinical Research (Retrospective)
• Avg. Duration: 2-6 weeks
• Output: Analytical Reports and Intervention
Effectiveness Determinations
• Project Types: Intervention Effectiveness
Analysis, Descriptive Analytics/Modeling,
Publication and Clinical Research Support,
Sample Size Calculations
Team Roles

12. Team Description

13. Skills and Methods Description

14. GettingStarted

15. Foundational Elements

16. Pipeline Based Project Intake

17. Ongoing Fundamentals

18. • Higher Performance
• Better Alignment
• Lower Cost (Build &
Maintenance)
• Full Transparency
• Develop Skills as Assets
• Full IP Ownership
• Potential Revenue Source
• Increased Flexibility
(Build & Ongoing)
• Develop Competitive
Advantage
• Talent (Acquisition &
Retention)
• Internal Resistance
• Easier to de-prioritize due
to decreased costs
• Benchmarking
• Access to External Data
• Liability
• Market Awareness
• Market Place Access
• Staff Skill Dev.
• Staff Augmentation on
lower priorities
• Potential ease of
implementation (EPIC)
• Additional Expert
Knowledge
• Reputation Sharing
• Potential Experience with
additional External Data
Sources and Technologies
• Build Costs
• Solution Flexibility
• Ongoing Costs (Staff and
Dollars)
• IP Sharing
• Purpose alignment
(business vs. academia)
• Data Breach Risk
• Implementation Risk
• Cost of Failure (Staff,
Opportunity and Dollars)
• Misalignment of
expectations and reality
• Political Objectivity
• Market Awareness
• Potential Time-To-
Value Improvements
• Potential Liability
Sharing
• Additional Expert
Knowledge
• Potential Experience with
additional External Data
Sources and Technologies
• Initial Costs
• Solution Flexibility
• Ongoing Costs (Staff and
Dollars)
• IP Ownership
• Data Breach Risk
• Cost of Failure
• No Internal Asset
development
• No Sustainable
Competitive Advantage
Available
BENEFITS RISKS
BuildPartnerBuy
Project Resourcing

19. ProjectHighlights
Project wins and lessons learned

20. 30 Day All Cause Readmissions
• Proved internal development can work, and
produce higher performance
• Reallocating >$2M in Nurse Time back to
patient care annually
• Embedded into multiple workflows through
multiple business units
• Lack of organizational knowledge for
deployment
• Concerns regarding ability to internally
develop
• Acquiring feedback from end users
Wins Challenges

21. Cost and Utilization Models
• >$1-Million per year in cost avoidance through
internal development of Utilization Model
• Care Management Model provides > $23-
Million per year in improved identification of
high risk patients
• Able to reduce complexity for clinicians while
improving performance
• Both models provide substantial performance
improvements over alternative options
• Lack of clarity around model purpose
generates perception of risk in use
• Models deployment can be impacted by much
more than performance concerns
• The larger the involved leadership team, the
more potential decision delays
Wins Challenges

22. Staffing Simulation/Optimization
• Optimized Call Center Staffing to ensure
improved performance metrics
• Helped right size ePharmacy staffing prior to
service implementation
• Added a high impact, and easily understood,
method to the Advanced Analytics toolbox
• More direct interaction with study subjects
than typical
• Requires workflow mapping, which isn’t a
commonly used approach on the team
• Working through IT processes to provide
business leader Python
Wins Challenges

23. Risk of Hospitalization Models
• Built models with substantially improved
performance for COPD and Heart Failure
• Created adaptive learning approaches for
production
• While clinical interest in the models is high,
defined processes/expectations for model use
is lacking
Wins Challenges

24. ED Sepsis Risk Models
• Provides alert of high risk ~68 minutes faster
than current solution
• Improved precision and recall, compared to
current solution
• Built strong collaborative relationships across
multiple teams
• Concerns regarding custom solution vs. better
known, but less performant approaches
• First real-time implementation inside EPIC for
point-of-care modeling
• Defining rules for when and where to trigger
best practice alert (BPA)
Wins Challenges

25. Natural Language Processing
• Achieving strong results with Ejection Fraction
extraction
• Building research relationships with
Homelessness identification
• Empowering de-identified research database
through PHI replacements in notes
• Identifying clinical resources for annotation
assistance
• Obtaining required annotation software
through IT processes
Wins Challenges

26. Contact
Juli Plack
Vice President, Information Delivery
Juliann.B.Plack@osfhealthcare.org
Chris Franciskovich
Manager, Advanced Analytics
Chris.M.Franciskovich@osfhealthcare.org