Download as PDF, PPTX
![2014/15 - 4,207 surgical episodes with 97 deaths (mortality rate 2.3%) – 89 deaths predicted
Area under ROC curve 0.86 [0.82, 0.89]
PRAiS2 performance in test data](https://image.slidesharecdn.com/martinutley-161107144250/75/Engaging-with-data-23-2048.jpg)
Uploaded byNuffield Trust
Engaging with data
The document discusses the development and use of the PRAIS software tool for monitoring 30-day survival rates following pediatric cardiac surgery, emphasizing its implementation in UK surgical centers since 2013. It highlights challenges in data quality and the importance of clinician engagement in improving risk-adjusted outcome monitoring. The document also outlines the validation of various diagnoses and risk models used to enhance understanding of surgical performance.
More Related Content
Engaging with data
- 1. Engaging with data MartinUtley, Director of the Clinical Operational Research Unit, University College London Chair: Chris Sherlaw-Johnson, Senior Research Analyst, Nuffield Trust
- 2. Building tools withclinical teams for use in monitoring outcomes. Martin Utley UCL Clinical Operational Research Unit
- 3. I received royaltiesfrom UCL Business in relation to co-authorship of the PRAiS software tool for monitoring 30-day survival following paediatric cardiac surgery. Declaration
- 4.
- 5. ? One can monitoroutcomes to identify problems One has absolute standards (some components to ¼ inch, others to a thousandth) If there is a problem, the machine can be fixed. To diagnose and fix problem, one has to stop production and open the box. Stopping production comes at a cost, so one has to have a degree of confidence that there is a genuine problem. To what extent do these conditions apply to health care? An insentient machine
- 6. Survival to 30days following paediatric cardiac surgery
- 7. Variable Life AdjustedDisplay (VLAD) with Partial Risk Adjustment in Surgery (PRAiS)
- 8. Variable Life AdjustedDisplay (VLAD) with Partial Risk Adjustment in Surgery (PRAiS) Pagel C, Utley M, Crowe S, Witter T, Anderson D, Samson R, McLean A, Banks V, Tsang V, Brown K, Real time monitoring of risk-adjusted paediatric cardiac surgery outcomes using variable life-adjusted display: implementation in three UK centres, Heart (2013) 99:1445-50. In use for local monitoring of outcomes at UK surgical centres since 2013.
- 9. In use forlocal monitoring of outcomes at UK surgical centres since 2013. Dissemination to wider clinical community Clinical consultation / contribution / leadership Development of risk model Design of data displays
- 10. In use forlocal monitoring of outcomes at UK surgical centres since 2013. Dissemination to wider clinical community Clinicians led engagement with surgical units, National audit body & other stakeholders
- 11. All episodes ina 30% validation sample within development set: Mortality: 3.5%, N=7890 High risk diagnoses Mortality: 8.8%, N=667 HLHS Interrupted aortic arch Pulmonary atresia (inc PA+IVS) Comorbidity Low risk diagnoses Mortality: 1.4%, N=3196 Fallot/DORV-Fallot type VSD Pulmonary stenosis Aortic arch obst +/- VSD/ASD ASD Subaortic stenosis (isolated) Aortic regurgitation Procedure Medium risk diagnoses Mortality: 4.2%, N=4027 PDA Acquired EMPTY/Unknown Functionally uni-ventricular heart TGA+VSD/DORV-TGA type Pulmonary atresia+VSD (inc fallot type) Tricuspid valve abnormality (inc Ebstein) Mitral valve abnormality (inc supra, sub) TAPVC AVSD Aortic valve stenosis Misc Cong TGA & IVS Common arterial truncus Normal All episodes in a 30% validation sample within development set: Mortality: 3.5%, N=7890 High risk diagnoses Mortality: 8.8%, N=667 HLHS Interrupted aortic arch Pulmonary atresia (inc PA+IVS) Comorbidity Low risk diagnoses Mortality: 1.4%, N=3196 Fallot/DORV-Fallot type VSD Pulmonary stenosis Aortic arch obst +/- VSD/ASD ASD Subaortic stenosis (isolated) Aortic regurgitation Procedure Medium risk diagnoses Mortality: 4.2%, N=4027 PDA Acquired EMPTY/Unknown Functionally uni-ventricular heart TGA+VSD/DORV-TGA type Pulmonary atresia+VSD (inc fallot type) Tricuspid valve abnormality (inc Ebstein) Mitral valve abnormality (inc supra, sub) TAPVC AVSD Aortic valve stenosis Misc Cong TGA & IVS Common arterial truncus Normal 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Year Percentage episodes with empty co-morbidity fields New ways of using diagnostic codes clinically led / meaningful then simplified for purposes of risk model Any old way of including comorbidity knowingly included poor quality data acknowledging clinical nous commitment to long game Development of risk model Old ways of using operation codes accepted existing schemes impact on model “performance” impact of model use Year % missing comorbidity data
- 12. PRAIS 1: • 29Specific procedures • 3 diagnosis groups (low, medium, high) • UVH • Yes/no comorbidity • Bypass/non-bypass • Continuous weight • Continuous age • Age band (neonate/infant/child) • Epoch effect 38 parameters PRAIS 2: • 16 spec proc groupings • 11 diagnosis groupings • UVH • 4 “other illness” indicators • Bypass/non-bypass • Continuous nonlinear weight • Continuous nonlinear age • Epoch effect 36 parameters PRAiS2 – developed on 21838 surgical episodes 2009-2014 from NCHDA data. Recent update
- 13. VLADs do notgive an evaluation of the effectiveness / benefit of a surgical programme to the patient population. VLADs are not evidence of good or bad performance. DATA patient case-mix structure / resources care processes professionals Queensland pyramid model of investigation Worked with 3 sites on design of software We improved ease of data entry / use, shared strengths and limitations, listened to concerns.
- 14. Consultation on designof data displays Added after consultation with units Secondary outcomes we would have discarded if not for discussions with clinical team
- 15. Weeks following releaseof PRAiS 2013
- 16. Christina Pagel -How can we better support appropriate interpretation and use of risk adjusted survival data? PRAiS 1 launch
- 17. What? How? Why? PRAiS2 Research team workedwith clinicians, family representatives, press officers and journalists on resources to support understanding and use of risk-adjusted mortality data.
- 18.
- 19.
- 20.
- 21. What? How? Why? Funding NIHR HSDRPI Victor Tsang Health Foundation PI MU NICOR Funding NIHR HSDR PI Christina Pagel Funding NIHR HSDR PI Victor Tsang & Kate Brown Funding GOSH Charity PI Kate Brown None of the views expressed are those of DH, NIHR, Health Foundation, GOSH Charity etc.
- 23. 2014/15 - 4,207surgical episodes with 97 deaths (mortality rate 2.3%) – 89 deaths predicted Area under ROC curve 0.86 [0.82, 0.89] PRAiS2 performance in test data
- 24. Brown KL, CroweS, Franklin R, McLean A, Cunningham D, Barron D, Tsang V, Pagel C, Utley M, Trends in 30-day mortality rate and case mix for paediatric cardiac surgery in the UK between 2000 and 2010, Open Heart. 2015 Feb 14;2(1):e000157. doi: 10.1136/openhrt-2014-000157. Absolute survival rates completely absent from debate in 2013 Kate Brown
- 25. PRAIS 2: • 16spec proc groupings • 11 diagnosis groupings • UVH • 4 “other illness” indicators • Bypass/non-bypass • Continuous nonlinear weight • Continuous nonlinear age • Epoch effect 36 parameters Performance assessed using cross-validation in main data set and prospective use in 2014/15 national data set. PRAiS risk model recently updated Close collaboration with an expert panel including of intensive care consultants, surgeons, cardiologists, data experts, analysts