• Like
Can't wait for the weekend to begin.
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Can't wait for the weekend to begin.



  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • How many are structure process outcomes
  • CUSUMs based on log-likelihood ratios are preferable to methods such as VLADs (variable life-adjusted displays), which have been used previously in analyses on St George’s data, because they have better a detection rate for a given false positive rate. v8 will add admission source and ethnic category
  • The data used to estimate the pre-op risk may be historical, e.g. national data for the previous year(s) If no date of operation is given, just use the date of admission. Thresholds are based on simulation, giving false positive rates after e.g. 50 patients, true detection rates within e.g. 10 patients, % of alarms that are true positives etc.
  • CUSUMs based on log-likelihood ratios are preferable to methods such as VLADs (variable life-adjusted displays), which have been used previously in analyses on St George’s data, because they have better a detection rate for a given false positive rate.


  • 1. Measuring the Quality of Hospital Care Dr Paul Aylin Professor Sir Brian Jarman Dr Alex Bottle [email_address]
  • 2. Contents
    • Background
    • English Hospital Statistics
    • Case-mix adjustment
    • Presentation of performance data
      • League tables
      • Bayesian ranking
      • Statistical process Control Charts
  • 3. Florence Nightingale
  • 4. Florence Nightingale
    • Uniform hospital statistics would:
      • “ Enable us to ascertain the relative mortality of different hospitals as well as of different diseases and injuries at the same and at different ages, the relative frequency of different diseases and injuries among the classes which enter hospitals in different countries, and in different districts of the same country”
      • Nightingale 1863
  • 5.
    • Heart operations at the BRI
    • “ Inadequate care for one third of children”
    • Harold Shipman
    • Murdered more than 200 patients
    Key events
  • 6. Mortality from open procedures in children aged under one year for 11 centres in three epochs; data derived from Hospital Episode Statistics (HES)
  • 7. Following the Bristol Royal Infirmary Inquiry
    • Commission for Health Improvement (now Healthcare Commission) - regularly inspect Britain's hospitals and publish some limited performance figures.
    • National Clinical Assessment Authority – investigates any brewing crisis.
    • National Patient Safety Agency collates information on medical errors.
    • Annual appraisals for hospital consultants
    • Revalidation, a system in which doctors have to prove they are still fit to practice every five years
  • 8.  
  • 9.  
  • 10.  
  • 11.  
  • 12.  
  • 13.  
  • 14.  
  • 15.  
  • 16. Hospital Episode Statistics
    • Electronic record of every inpatient or day case episode of patient care in every NHS (public) hospital
    • 14 million records a year
    • 300 fields of information including
      • Patient details such as age, sex, address
      • Diagnosis using ICD10
      • Procedures using OPCS4
      • Admission method
      • Discharge method
  • 17. Why use Hospital Episode Statistics
      • Comprehensive – collected by all NHS trusts across country on all patients
      • Coding of data separate from clinician
      • Access
      • Updated monthly from SUS (previously NHS Wide Clearing Service)
  • 18. Case mix adjustment
    • Limited within HES?
      • Age
      • Sex
      • Emergency/Elective
  • 19.  
  • 20. Risk adjustment models using HES on 3 index procedures
      • CABG
      • AAA
      • Bowel resection for colorectal cancer
  • 21. Risk factors Age Recent MI admission Sex Charlson comorbidity score (capped at 6) Method of admission Number of arteries replaced Revision of CABG Part of aorta repaired Year Part of colon/rectum removed Deprivation quintile Previous heart operation Previous emergency admissions Previous abdominal surgery Previous IHD admissions
  • 22. ROC curve areas comparing ‘simple’, ‘intermediate’ and ‘complex’ models derived from HES with models derived from clinical databases for four index procedures Aylin P; Bottle A; Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044
  • 23. Calibration plots for ‘complex’ HES-based risk prediction models for four index procedures showing observed number of deaths against predicted based on validation set Aylin P; Bottle A; Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044
  • 24. Current casemix adjustment model for each diagnosis and procedure group
    • Adjusts for
      • age
      • sex
      • elective status
      • socio-economic deprivation
      • Diagnosis subgroups (3 digit ICD10) or procedure subgroups
      • co-morbidity – Charlson index
      • number of prior emergency admissions
      • palliative care
      • year
      • month of admission
  • 25. Current performance of risk models ROC (based on 1996/7-2007/8 HES data) for in-hospital mortality
    • 56 Clinical Classification System diagnostic groups leading to 80% of all in-hospital deaths
    • 7 CCS groups 0.90 or above
      • Includes cancer of breast (0.94) and biliary tract disease (0.91)
    • 28 CCS groups 0.80 to 0.89
      • Includes aortic, peripheral and visceral anuerysms (0.87) and cancer of colon (0.83)
    • 18 CCS groups 0.7 to 0.79
      • Includes septicaemia (0.77) and acute myocardial infarction (0.74)
    • 3 CCS groups 0.60 to 0.69
      • Includes COPD (0.69) and congestive heart failure (0.65)
  • 26. Presentation of clinical outcomes
    • “ Even if all surgeons are equally good, about half will have below average results, one will have the worst results, and the worst results will be a long way below average”
      • Poloniecki J. BMJ 1998;316:1734-1736
  • 27.  
  • 28. Criticisms of ‘league tables’
      • Spurious ranking – ‘someone’s got to be bottom’
      • Encourages comparison when perhaps not justified
      • 95% intervals arbitrary
      • No consideration of multiple comparisons
      • Single-year cross-section – what about change? 
  • 29.  
  • 30. Bayesian ranking
    • Bayesian approach using Monte Carlo simulations can provide confidence intervals around ranks
    • Can also provide probability that a unit is in top 10%, 5% or even is at the top of the table
      • See Marshall et al. (1998). League tables of in vitro fertilisation clinics: how confident can we be about the rankings? British Medical Journal, 316, 1701-4.
  • 31.  
  • 32. Statistical Process Control (SPC) charts
    • Shipman:
      • Aylin et al, Lancet (2003)
      • Mohammed et al, Lancet (2001)
      • Spiegelhalter et al, J Qual Health Care (2003)
    • Surgical mortality:
      • Poloniecki et al, BMJ (1998)
      • Lovegrove et al, CHI report into St George’s
      • Steiner et al, Biostatistics (2000)
    • Public health:
      • Terje et al, Stats in Med (1993)
      • Vanbrackle & Williamson, Stats in Med (1999)
      • Rossi et al, Stats in Med (1999)
      • Williamson & Weatherby-Hudson, Stats in Med (1999)
  • 33. Common features of SPC charts
    • Need to define:
      • in-control process (acceptable/benchmark performance)
      • out-of-control process (that is cause for concern)
    • Test statistic
      • Function of the difference between observed and benchmark performance
      • calculated for each unit of analysis
  • 34. HSMR 2007/8 with 99.8% control limits
  • 35. Funnel plots
    • No ranking
    • Visual relationship with volume
    • Takes account of increased variability of smaller centres
  • 36. Risk-adjusted Log-likelihood CUSUM charts
      • STEP 1: estimate pre-op risk for each patient, given their age, sex etc. This may be national average or other benchmark
      • STEP 2: Order patients chronologically by date of operation
      • STEP 3: Choose chart threshold(s) of acceptable “sensitivity” and “specificity” (via simulation)
      • STEP 4: Plot function of patient’s actual outcome v pre-op risk for every patient, and see if – and why – threshold(s) is crossed
  • 37. More details
      • Based on log-likelihood CUSUM to detect a predetermined increase in risk of interest
      • Taken from Steiner et al (2000); pre-op risks derived from logistic regression of national data
      • The CUSUM statistic is the log-likelihood test statistic for binomial data based on the predicted risk of outcome and the actual outcome
      • Model uses administrative data and adjusts for age, sex, emergency status, socio-economic deprivation etc.
    Bottle A, Aylin P. Intelligent Information: a national system for monitoring clinical performance. Health Services Research (in press).
  • 38.  
  • 39. Currently monitoring
    • 78 diagnoses
    • 128 procedures
    • 90% deaths
    • Outcomes
      • Mortality
      • Emergency readmissions
      • Day case rates
      • Length of Stay
  • 40.  
  • 41.  
  • 42. How do you investigate a signal?
  • 43.  
  • 44.  
  • 45. Factors affecting hospital statistics Aetiology Basic morbidity Observed morbidity Admission Hospital statistics Information system Diagnostic coding Diagnostic fashion Readmissions Medical care Medical practice Illness behaviour Organisation of care Admission criteria
  • 46. What to do with a signal
      • Check the data
      • Difference in casemix
      • Examine organisational or procedural differences
      • Only then consider quality of care
  • 47. Future
    • Patient Reported Outcomes (PROMs)
    • Patient satisfaction/experience
    • Safety/adverse events
    • Pay for performance and quality
  • 48.  
  • 49.  
  • 50.  
  • 51.  
  • 52. Comparison of HES vs clinical databases
    • Isolated CABG
      • HES around 10% fewer cases compared to National Cardiac Surgical Database
      • Fifth National Adult Cardiac Surgical Database Report 2003. The Society of Cardiothoracic Surgeons of Great Britain and Ireland. Dendrite Clinical Systems Ltd. Henley-Upon-Thames. 2004.
    • Vascular surgery
      • HES = 32,242
      • National Vascular Database = 8,462
      • Aylin P; Lees T; Baker S; Prytherch D; Ashley S. (2007) Descriptive study comparing routine hospital administrative data with the Vascular Society of Great Britain and Ireland's National Vascular Database. Eur J Vasc Endovasc Surg 2007; 33 :461-465
    • Bowel resection for colorectal cancer
      • HES 2001/2 = 16,346
      • ACPGBI 2001/2 = 7,635
      • ACPGBI database, 39% of patients had missing data for the risk factors
      • Garout M, Tilney H, Aylin, P. Comparison of administrative data with the Association of Coloproctology of Great Britain and Ireland (ACPGBI) colorectal cancer database. International Journal of Colorectal Disease (in press)
  • 53. Why is it important to take into account time trends
    • UK Adult Cardiac Surgery
    • Mortality rates halved in last 10 years
    • Use if out of date risk models gives impression of all units performing better than expected.
  • 54.  
  • 55.  
  • 56. Adjusted (EuroSCORE) mortality rates for primary isolated CABGs by centre (3 years data up to March 2005) using SCTS data with 95% and 99.8% control limits based on EuroSCORE expected mortality.
  • 57. Adjusted (EuroSCORE) mortality rates for primary isolated CABGs by centre (3 years data up to March 2005) using SCTS data with 95% and 99.8% control limits based on mean national mortality rates
  • 58. Other considerations
    • Transfers
      • Transfers linked. All spells (admissions) linked into superspells
      • For diagnosis, outcome based on discharge method at end of superspell
    • Diagnosis on admission
      • No diagnosis on admission exists within HES/SUS
      • We use primary diagnosis given on completion of first episode, unless a “vague symptoms and signs” diagnosis, in which case we examine subsequent episode
    • Palliative care
      • If treatment specialty in any episode in the admission coded to palliative care or includes ICD10 code Z515, accounted for in risk model