The 10th Annual Utah Health Services Research Conference: Iterative Development of Sepsis Detection Algorithms for the Emergency Department. By: Peter Haug - Intermountain Healthcare Health Services Research Conference: March 16, 2015 Patient Centered Research Methods Core, University of Utah, CCTS

1. Iterative Development of Sepsis Detection
Algorithms for the Emergency Department
Developing Diagnostic Systems for Clinical Care
Peter Haug, MD.
Homer Warner Center for Informatics Research
Intermountain Healthcare

2. Diagnostic Modeling and Sepsis
Subjects for todays presentation
• Electronic Data and Diagnostic Models
• Two Active Sepsis Systems for the ED
– Supporting the Sepsis Bundle
– Research on Septic Shock
• A Development Environment to Train Diagnostic
Algorithms
– A Semi-Automated Development Environment
– Using “Ontologies” to Describe Diagnostic Relationships
– Supporting Iterative Development of Diagnostic Algorithms
– Built-In Evaluation Tools
• Example: Developing a Comprehensive Sepsis Model?

3. Multiple Episodes of Care
Research and
Analysis
Data
Warehouse:
Available for
Analysis
Electronic Health Record
Clinical Data Serves Us Twice
Medical
Decision
Support
Supporting
Care/Collecting
Data

4. Page 4
Introducing Bayesian Networks
A Graphical Diagnostic Modeling Tool
• Development: Good Authoring Tools
– Models Trained from data
– Or Created Manually
– Easily Displayed and Inspected
– (Partially) consistent with Intuition
• Processing: Good Runtime
Characteristics
– Models Work well with Missing Data
– Resilient with Redundant Data
– Can Generate Multiple Outputs
– Produces Probabilistic Outputs

5. Case 1: Detecting Sepsis for the Sepsis Bundle
• The “Sepsis Bundle” is a set of 11 clinical
interventions necessary for quality care of the septic
patient.
• Early detection is required to support the initial
interventions in the Sepsis Bundle.
• Need to identify “Possible Sepsis” patients within 2
hours of admission.
• Use the earliest available data that can discriminate
Septic patients.
• Data used: Heart Rate, Age, Temperature, WBC,
MeanBP

6. Detecting Sepsis for the Sepsis Bundle
Goal: Identify Sepsis Patients within 2 hours of admission.
Simple Sepsis Model: Designed for Early Detection

7. Effectiveness of the Sepsis Detector
• Developed on ED patients from 2006-2008
• Validated on ED patients from 2009-2011.
• Five features chosen from 100+.
• Naïve Bayesian Model.
• Compared to:
• SIRS Criteria at 1 hour.
• SIRS Criteria at 12 hours.
• Manual Triage Nurse alert.
Table 2: Comparative Performance
Thrshld Sens. FPR PPV AUC
Deterministic Alerts
Triage Alert N/A 0.556 0.033 0.064 0.761
SIRS 2+ by 1hr N/A 0.719 0.147 0.020 0.786
SIRS 2+ by 12hr N/A 0.858 0.186 0.019 0.836
Probabilistic Model
As Deployed 0.050 0.781 0.049 0.062 0.866
Probabilistic Model at Alternate Alert Thresholds
…Match Triage Alert Sensitivity 0.329 0.556 0.017 0.119 0.769
…Match Triage Alert False Pos. 0.096 0.716 0.033 0.082 0.841

8. A Second Sepsis Model
Identifying Septic Shock Patients for a Research Study
• Developed to Identify Patients for a
Clinical Trial
• Goal: Identify Sicker Septic Patients
– Patients with Shock
– Patients Who Will Require Resuscitation
• A Rule-Based Approach
– Implemented Using Bayesian Network
Technologies
Page 8
Heart_Rate (HeartRate) = if(HeartRate >= 90, Met, Unmet)

9. Ontology-Driven Modeling System: Input and Output
Uses an Ontology to Semi-automatically Create a Diagnostic System
• Inputs:
– Analytic Health Repository/Enterprise Data Warehouse
– A Diagnostic Ontology.
– Diagnostic Scenario
• Outputs:
– A diagnostic predictive model produced by the system.
– A list of the features found to be relevant to the
diagnostic problem addressed.
– An analysis of the quality of the predictive model.
– The raw data generated by the system and used to
develop the diagnostic model.
Data Warehouse/
Analytic Health
Repository
Disease
Ontology
Concept Retrieval (from
Ontology
Concept Translation to
EDW Representation
Output
20%
20%
20%
20%
20%


i
ii dfPdP
dfPdP
fdP
)|()(
)|()(
)|(
Prediction
Algorithm
Analysis Results
Analytic Data
Relevant Ontologic
Concepts
Analysis
Design
Utility
Analytic Workbench
· Diagnostic Models
· Model Comparisons
· Model Explanation
(by reference to the
Ontology)
Natural
Language
Processing
Subsystem
Structural Knowledge Retrieval
from the Ontology
Data Retrieval from the Analytic
Health Repository

10. Ontology-Driven Model Discovery
Using knowledge embedded in ontologies to automate research?
• Ontologies capture medical knowledge in a computable
form
• They capture hierarchical relationships (staph pneumonia
is a bacterial pneumonia”)
• They capture clinical relationships (“disease causes
finding”
• They capture synonyms (pneumococcal pneumonia =
ICD9: 481)
• An Research Automation Tool using an Ontology Can:
• Help select research patients
• Identify and extract relevant data
• Condition preliminary analysis of the data
• Support visualization of this data
• Return Data and results to the user for further study

11. Ontologies Describe How Diseases Are Related
(according to ICD9)
Pneumococcal pneumonia
Pneumococcal pneumonia
ICD9: 481
Other Bacterial Pneumonia
Other bacterial pneumonia
ICD9: 482
Streptococal Pneumonia
Pneumonia due to Other
Streptococcus
ICD9: 482.3
Bronchopneumonia
Bronchopneumonia,
organism unspecified
ICD9: 485
Viral Pneumonia
Viral pneumonia
ICD9: 480
Staphlococcal Pneumonia
Pneumonia due to
Staphylococcus
ICD9: 482.4
Hemophilus Pneumonia
Pneumonia due to
Hemophilus influenzae
ICD9: 482.2
Pseudomonas Pneumonia
Pneumonia due to
Pseudomona
ICD9: 482.1
Pneumonia
More Bactierial
Pneumonias
Staph Aureus Pneumonia
Pneumonia due to
Staphylococcus, unspecified
ICD9: 482.40
MSSA Staph Pneumonia
Methicillin Susceptable
Staph Aureus (MSSA)
Pneumonia
ICD9: 482.41
MRSA Staph Pneumonia
Methicillin Resistant Staph
Aureus (MRSA) Pneumonia
ICD9: 482.42
Other Staph Pneumonia
Other Staphylococcus
pneumonia
ICD9: 482.49
Bacterial
Pneumonia
More Pneumonias

12. Ontologies can Also Describe How Clinical Data
are Related to Diseases
has_X-ray_Manifestation
Pneumonia
Pneumonia, Organism
unspecified
ICD9: 486
Pneumococcal pneumonia
Pneumococcal pneumonia
ICD9: 481
Pneumonia
More Bacterial
Pneumonias
Bacterial
Pneumonia
More Pneumonias
has_Sign
White Blood Count
Hematology: White
Blood Count
LOINC: 62239-9
has_Altered_Lab_Value
Pulmonary Rales
Signs: Chest
Auscultation-Rales
PTXT:
28.1.3.22.34.2.1.32
Temperature
Vital Signs:
Temperature
LOINC: 8310-5
has_Altered_VS
Localize Infitrate
X-ray Finding:
Localized Infiltrate
SNOMED: 128309002
has_Micro_Manifestation
Other Bacterial Pneumonia
Other bacterial pneumonia
ICD9: 482
More
Manifestations
has_??_Manifestation
Sputum Culture:
Positive
SNOMED: 442773002
+ Sputum Culture

13. Building a New Sepsis Detector
Using the Diagnostic Modeling Tools
• Example: a simple Sepsis model
– Built from 186,725 patients and 91 features
– Model restricted to 15 features
– Uses the simplest Bayesian Model (Naïve Bayes)
• A completely automated process.

14. Using Modeling Tools to Refine a Sepsis Detector
• The Diagnostic Modeling System
Provides Tools to Modify Models
• Two Pathways to Improve Accuracy
– Refine Model
– Extend Data
• Refining a Model
– Changing Feature Relationships
– Adding Feature Aggregation Constructs
• Extending Training Data
– Deriving New Features from Existing
Data (Summary Features)
ODMS
Extracted Data
Analytic
Health
Repository
Diagnostic Model


i
ii dfPdP
dfPdP
fdP
)|()(
)|()(
)|(
5 4 3 2 1
Analytic Results
Modified Diagnostic
Model


i
ii dfPdP
dfPdP
fdP
)|()(
)|()(
)|(
Modified Data
Analytic Workbench
· Process Data from AHR
· Produce Predictive Model
· Test Predictive Model
Ontology
Submit New
Model for
Evaluation
Submit Modified
Data for
Evaluation

15. Adding Semantics to the Model
Saying “Findings are consistent with Diagnosis”
Orientation_Consistent
True
False
11.3
88.7
Oriented_x_3
Absent
Present
11.3
88.7
Alert_and_oriented_x_3
Absent
Present
37.2
62.8
Oriented_to_situation
Absent
Present
99.9
.060
Not_oriented_to_time
Absent
Present
94.1
5.92
Not_oriented_to_situation
Absent
Present
99.9
.052
Not_oriented_to_place
Absent
Present
97.0
3.04
Not_oriented_to_person
Absent
Present
99.6
0.44
Not_oriented_to_S_O
Absent
Present
100
.003
Not_Oriented_x_3
Present
Absent
0.49
99.5
Sepsis
Present
Absent
1.75
98.2
Pneumonia
Present
Absent
0.49
99.5
CXR_Consistent
False
True
94.5
5.47
Multi_Lobar_Infiltrates
Present
Absent
Unknown
33.3
33.3
33.3
CXR_Ordered
Present
Absent
0
100
Single_Lobe_Infiltrate
Absent
Present
Unknown
33.3
33.3
33.3
“Chest X-ray Results consistent with Pneumonia” “Mental Status consistent with Sepsis”

16. The Results of Diagnostic Model Development
Providing tools to critique model accuracy
• Inspecting model accuracy can be done Graphically.
– ROC Curves
– Precision/Recall Tradeoff Graphs
• Relative value of features can also be critiqued.
– Graphical and Tabular views.
Comparing Two
Models Using the ROC
Curves
Inspecting a Model’s
Tradeoffs in Accuracy

17. Feature Extraction/Selection
• Sepsis Features returned by the Analytic Workbench
Page 17
• Relevant features identified using links in the
Ontology
– Links from Diagnoses to Observations
– Standard naming conventions
• Features are evaluated and ranked
• The most informative subsets are used for
analysis.

18. Sepsis Model: Extending Its Coverage
• Initial Model: Designed to function within the first 2 hours
• Revised Model: Designed to function anytime within the
first 24 hours
Simple Sepsis Model: Designed for Early DetectionComplex Sepsis Model: Effective at Multiple Times?

19. Results of Analysis of New Sepsis Models Accuracy
• Data: 280,143 patients of whom 4976 had in ICD‐9 diagnosis of sepsis.
• Divided into a training set (186,725 patients) and a test set (93,418 patients).
• 91 features associated with sepsis which were available during the initial day of care.
• 118 features were available after feature expansion.
• Built 12 different Sepsis Diagnostic Models
0.95 0.97 0.95 0.97 0.97 0.97
0.30
0.38
0.27
0.37 0.39 0.40
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Naïve
Bayes-OS
Naïve
Bayes-ES
TAN Bayes-
OL
TAN Bayes-
EM
Custom
Bayes-Naïve
Custom
Bayes-TAN
AROC
Precision at 60%

20. Conclusion
“All models are wrong; some models are useful.”
Attributed to statistician George Box
• Diagnostic Models have a role in clinical care
• Enterprise Data Warehouses can contribute to
Model development
• Stored Medical Knowledge (in the form of
Ontologies) can accelerate Modeling
• Bayesian Models can flexibly represent a variety of
clinical conditions.

21. Questions???
Comments and Questions