Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OSFair2017 Training | Reproducibility in critical care research


Published on

Tom Pollard talks about reproducibility in critical care research & makes an introduction to MIMIC, the eICU Collaborative Research Database and datathons

Workshop title: Datathons in Evidence-Based Medicine: Applying Open Science Principles to Support Cross-Disciplinary Education and Research

Workshop abstract:
In this interactive workshop, we explore how open science enables “datathons”, events that bring together teams of researchers to work together on unanswered clinical questions. We begin by outlining the datathon model and describe our experiences in holding these events internationally. We then offer an opportunity to participate in an interactice exercise, working together to analyse highly detailed information collected from patients admitted to critical care units at a large tertiary care hospital. Participants will learn about open science in clinical research and gain an overview of MIMIC-III, a freely-available critical care dataset collected from over >50,000 hospital stays.


Published in: Science
  • Login to see the comments

  • Be the first to like this

OSFair2017 Training | Reproducibility in critical care research

  1. 1. Reproducibility in critical care research Dr Tom Pollard Laboratory for Computational Physiology, Massachusetts Institute of Technology (MIT), USA @tompollard
  2. 2. Critical 
  3. 3. Heart rate O2 saturation NIBP, mean Respiratory rate Intake volume, dL Output volume, dL 0 0 10 30 504020 60 120 100 80 60 40 20 Time after admission to the intensive care unit, hours Measurement,absolutevalue Code status Full code Comfort measures Incomprehensiblesounds Flex-withdraws None Oriented Obeys commands Spontaneously Oriented Obeyscommands Spontaneously Oriented Obeys commands Spontaneously Confused Obeyscommands To speech Confused Obeyscommands To speech GCS: Verbal GCS: Moto GCS: Eye Platelet, K/uL Creatinine, mg/dL Whitebloodcell,K/uL Neutrophil, % Morphine Sulfate Vancomycin(1dose) Piperacillin(1dose) NaCl 0.9% Amiodarone Dextrose 5% 48 0.7 9.1 37 53 12.4 46 0.7 16.8 45 0.8 23.2 10.0mL/hour 10.0mL/hour 10.0mL/hour 1mg/min 0.5mg/min 0.5mg/min 50mL/hour 25mL/hour 25mL/hour
  4. 4. 4 Title 4
  5. 5. Research opportunity ■ …data that could be used to discover new knowledge for the benefit of patients 5
  6. 6. 6 ■ but, this data is inaccessible to researchers
  7. 7. 8
  8. 8. 9 Two key steps to gaining access: • complete a recognized course in protecting human research participants that covers Health Insurance Portability and Accountability Act (HIPAA) requirements • sign a data use agreement, which outlines appropriate data usage and security standards, and forbids efforts to identify individual patients. Accessing the data 9
  9. 9. eICU Collaborative Research Database
  10. 10.
  11. 11. Relational database (a collection of linked spreadsheets) 1313
  12. 12. 28 courses worldwide, and counting…
  13. 13. Reproducibility is the ability to reproduce the results of a given study Note the distinction between this and *replication* (whether the results hold up in different experimental conditions). Reproducibility
  14. 14. Nature 533, 452–454 (26 May 2016) doi: 10.1038/533452a
  15. 15. Why should I care about reproducibility?
  16. 16. Reproducibility leads to incremental progress
  17. 17. Incremental progress on ImageNet A. Canziani et al, “An Analysis of Deep Neural Network Models for Practical Applications”, CoRR, 2016.
  18. 18. MIMIC is freely available critical care database 
 So papers using the MIMIC are reproducible. Right?
  19. 19. Reproducibility in critical care: 
 a mortality prediction case study ● Collect all studies which attempted to predict mortality in recent history ● Attempt to regenerate the cohort ● Compare our reproduced study cohort with the published cohort Reproducibility in critical care: a mortality prediction case study. Proceedings of Machine Learning for Healthcare (2017).
  20. 20. e.g. presence of bilateral infiltrates on chest radiograph 38 distinct evals.
  21. 21. Methods 1. Define a base cohort 2. For each study, add in the additional exclusion criteria specified 3. Compare the published sample size and mortality rate to ours 4. **Bonus** compare simple logistic regression AUROC to their AUROC
  22. 22. Base cohort ● Start with a “base” cohort with the minimum required exclusions ● Exclude ○ Patients < 15 years old ○ Invalid admissions (no charted obs, no heart rate obs, no admit/disch time) ○ Organ donor accounts ○ Stays less than 4 hours ● Outcomes of interest ○ In-hospital mortality ○ Post ICU discharge mortality ■ 48-hour, 30-day ○ Post hospital discharge mortality ■ 30-day, 6-month, 1-year, 2-year
  23. 23. Additional cohort criteria ● Hug et al. 2009 ○ >1 obs. for HR/GCS/Hct/BUN, Not NSICU/CSICU, first ICU stay, full code, no eventual brain death ● Lee et al. 2015, Lee and Maslove 2017 ○ Only ICU stays with complete SAPS data ● Ghassemi et al. 2014 ○ Age>18, >100 words across all notes ● Grnarova et al. 2016 ○ Age > 18, stays with only one hospital admission ● Che et al. 2016 ○ None described
  24. 24. Results - sample size
  25. 25. Results - sample size 75% have 1000+ more patien ts
  26. 26. Results - % mortality
  27. 27. Results - AUROC 71% of logistic regression models outperformed published paper
  28. 28. Discussion ● The majority of studies were not reproducible (>2/3rds)! ● How can we make it easier for others to reproduce our papers?
  29. 29. Discussion ● State all restrictions to cohort ○ Age, length of stay, certain care units, certain diagnoses ● Be explicit in criteria description ○ Specify MICU service or MICU physical location ○ “Removed patients with missing data” -> BAD!!! ○ “Removed patients with fewer than 1 heart rate measurement” -> GOOD!!! ● Detail data abstraction steps ○ 64% of studies were outperformed by logistic regression on simple features ○ Clearly data abstraction matters - give it more space in the paper!
  30. 30. Reproducibility Reusable data
  31. 31. Progress on ImageNet not just thanks to data… A. Canziani, A. Paszke and E. Culurciello, “An Analysis of Deep Neural Network Models for Practical Applications”, CoRR, 2016.
  32. 32. Reproducibility ● Reproducibility has two components Reusable data Reusable code Please do this more!
  33. 33. Thank you for your attention!