Martin Bardsley & Adam Steventon: Stemming demand: how best to track the impact of interventions
Stemming demand: how best to track the impact of interventions? Martin Bardsley Adam Steventon Nuffield Trust Health Strategy Summit March 24th 2010
Monthly number of emergencyadmissions in England
Approaches to managingdemand...• self-management education, •targeting people at high risk,• self-monitoring, •multidisciplinary teams after discharge,• group visits to primary care, •nurse-led clinics and nurse-led follow-• broad managed care up, programmes, •assertive case management, home• integrating social and health visits. care, • nurse-led clinics,• multidisciplinary teams in hospital, • telecare,• discharge planning, • telemonitoring.• multidisciplinary teams after discharge, But do they work?• care from specialist nurses, In your patch?
Challenges of evaluation....• Difficult to randomise a distinctive treatment and control group within the same organisations or service.• Service delivery patterns may change incrementally over time.• The client/patient group may change over time.• Randomised trials can be costly and sometimes out of proportion to the investment in the change).• Can be slow – changes need to be made embedded and cases followed up for a long time.• Results may only reflect experiences of a subset of users.
Alternative approaches.......• Exploits existing data sets – as much as possible. This makes it cheaper and easier to set up though it does create its own challenges.• Is continuous and timely. Aiming to provide interim results and feedback during throughout the evaluation period. This can potentially help fine tune the service – and the measurement process.• Aim to capture events and experiences for as broad a group of users and potential users as possible. So looks, to some degree at the majority of service users.• Develops accurate comparative tools – using the right methods to identify pseudo control groups as the basis for judging changes over time.• Exploits linked data sets to construct individual patient histories.
Why use routine information?Advantages Disadvantages• Relatively inexpensive • May not include the right• Comprehensive information• Person and event level • Rely on prior classifications• Accessible • Quality and completeness• Can be linked into routine of recording management reporting • Limited range of outcomes processes
Two methodological problems• Regression to the mean: if you select people with high service use, their service use will probably reduce anyway.• Cost are highly skewed: a relatively small change in very high costs users can have an impact.
Average number of emergency bed daysEmerging risk 50 45 40 35 30 25 20 15 10 5 0 -5 -4 -3 -2 -1 Intens +1 +2 +3 +4 e year
Will the next card be higher orlower? HIGHER LOWER ERRRR?? = Regression to the mean in the style of Brucey
The distribution of futureutilisation is exponential £4,500Actual Average cost per patient £4,000 £3,500 £3,000 £2,500 £2,000 £1,500 £1,000 £500 £0 0 10 20 30 40 50 60 70 80 90 Predicted Risk (centile rank)
Approach 1 WSD trial. A randomised trial.• Study started in 3 sites in 2007. Aim to recruit 6000 patients to the trial.• Recruitment to the study ended in Autumn 2009. Last trial participant reach 12mnths in 2010.• Final analyses early 2011.
Are telecare and telehealth part of the solution?“For every pound spent on telecare, five pounds could be saved on expensive hospital and residential care” Counsel and Care, 2009
Five evaluation themes Theme 1 Theme 2 Theme 3 Theme 4 Theme 5 (Nuffield (UCL) (LSE) (Manchester) (Imperial) Trust) Impact of Participant- Costs and Experiences of Organisational service use reported cost- service users, factors and and outcomes and effectiveness informal sustainable associated clinical carers and adoption and costs for the effectiveness professionals integration NHS andsocial services Subset of 2,750 people plus 660 of their informal Subset of Qualitative Qualitative All 6,000 carers 2,750 people interviews interviews people Universities of Oxford and Birmingham
Information Flows Encrypted subset HES/SUS Client-event based Linked Data Subsets Encrypted subset GP Client-event based Client BasedLocal Needs variables (Risk Groups)Operational Community Encrypted subset Hospital Use Nursing Activity Client-event based GP & Community UseSystems Encrypted subset Social Care Use Social care Client-event based Client event data Person level records Demographics Batch Service
Ensuring even mix of patients Analysis by risk subgroup
Approach 2. Using case controlsderived from routine data. 1 Access routine data at person level 2 Construct control groups to overcome regression to the mean 3 Regular monitoring and updates to influence policy development
Number of people receivingintervention per month (4 sites)
Linking participants to HES (1) IC collates and adds (if required) NHS numbers using batchParticipating sites tracing Information Centre Sites collate patient lists IC derives Nuffield Trust extra identifiers Patient identifiers Trial information (e.g. Non-patient identifiable keys (e.g. (e.g. NHS number) start and end date) HES ID, pseudonymised NHS #)
Profiles of emergencyhospital admissions (1) Start of intervention
Profiles of emergencyhospital admissions (2) Start of intervention
Regression to the mean? 50Average number of emergency 45 40 35 30 25 20bed days 15 10 5 0 -5 -4 -3 -2 -1 Intense + 1 +2 +3 +4 year
Choices about multivariatematching• Draw controls from local area, similar areas or nationally?• Which variables to include?• What weight to attach to each variables (distance measure)?• With or without replacement?• 1-1 matching or 1-many matching?• Caliper matching on certain variables?
Building models every month Predictor variables taken from two ... To predict 12 previous years.... months ahead
Prevalence of health diagnosescategories in intervention andcontrol groups
Comparison of intervention and control group Intervention Control Standardised (N=378) (N=378) differenceProportion aged 85+ 47% 47% 0.0%Proportion female 68% 68% 0.0%Mean area-level deprivation score 16.6 16.2 4.8%Mean number of emergency admissions in 1.0 0.9 3.0%previous yearMean number of emergency admissions in 0.3 0.3 4.0%previous 30 daysMean emergency length of stay in previous 8.6 8.7 0.7%yearMean number of chronic conditions 1.6 1.5 4.3%Mean predictive risk score 0.25 0.25 0.2%
Overcoming regression to themean using a control group (1) Start of intervention
Overcoming regression to themean using a control group (2) Start of intervention
Overcoming regression to themean using a control group (3) Start of intervention
Overcoming regression to themean using a control group (4) Start of intervention
Almost real-timetracking of intervention PARR score Impact on emergency admissions (number per head over 3mths)
Discussion points• What are the rate limiting steps? – Data being available? – The right data to measure what you want? – Skills to analyse data locally? – Analytical resources locally?• What are the priority interventions for routine tracking?• How should feedback be organised and delivered?• What should only be assessed with randomisation?