Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Economic evaluation of changes to the organisation and delivery of health services

131 views

Published on

CHE Seminar 6 July 2017, Speaker: Rachel Meacock, Research Fellow in Health Economics, University of Manchester,

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Economic evaluation of changes to the organisation and delivery of health services

  1. 1. METHODS FOR THE ECONOMIC EVALUATION OF CHANGES TO THE ORGANISATION AND DELIVERY OF HEALTH SERVICES Rachel Meacock Centre for Health Economics, University of York 6th July 2017
  2. 2. Background • Established methods exist for the economic evaluation of new health technologies seeking NHS funding in England • Treatments go through mandatory NICE appraisal process • Changes to the organisation and delivery of health services, including policy changes, (service interventions) are funded from the same NHS budget, but not covered by this process • Often rolled out without supportive evidence or evaluation
  3. 3. Background • Inconsistent approaches - differing levels of scrutiny likely to result in allocative inefficiency in the health system • Resulted in a lack of methodological development and cost- effectiveness evidence • Whilst principle of assessing cost-effectiveness should apply to all NHS spending, methods will need adapting in places to enable evaluation of service interventions
  4. 4. Aim To contribute to the development of methods for the economic evaluation of service interventions 1. Method to quantify effects of service interventions in terms of QALYs in absence of primary data collection on HRQoL 2. Demonstrate how survival analysis can be used to improve treatment effect estimates associated with service interventions (length of life component of QALY)
  5. 5. Part 1: Quantifying the effects of service interventions in terms of QALYs in the absence of primary data collection on HRQoL
  6. 6. Introduction • Often estimate the impact of service changes on mortality • Useful indicator of the impact of a programme, but tells us nothing about the intervention’s value • To assess cost-effectiveness, we need to estimate the impact of a programme in terms of QALYs • Problem = usually evaluate using administrative data which does not contain information on HRQoL
  7. 7. Proposed method • A QALY ‘tariff’ applied to mortality outcomes to estimate the QALY gains associated with detected mortality reductions • Discounted and quality-adjusted life expectancy (DANQALE) tariff • Stratified by single year of age (18 - 100) and sex • Represents the average stream of remaining QALYs for an individual i in each age-sex group a from general population
  8. 8. DANQALE Two components of the QALY: • Length of life component: Sex-specific life expectancy estimates at each single year of age taken from 2008-10 ONS interim life tables • QoL adjustment: Age-sex specific mean EQ-5D values from 2006 wave of the Health Survey for England
  9. 9. We calculate the DANQALE (𝑄𝑖𝑎) for each individual i in each age-sex group a as: 𝑄𝑖𝑎 = 1 − 𝑚𝑖 𝑘=𝑎 𝐿 𝑎 𝑞 𝑘 1 + 𝑟 − 𝑘−𝑎 • 𝑚𝑖 equals 1 if the individual dies within 30days and 0 otherwise • k indexes ages from a to the life expectancy of an individual currently aged a(𝐿 𝑎) • 𝑞 𝑘 is HRQoL at age k • r is the discount rate (3.5%)
  10. 10. Analysis • Attach DANQALE to each individual in the data • Perform analysis as normal, but on DANQALE variable • Can then compare to costs of the programme, either at the individual or total programme level Mortality Discounted QALYs (DANQALE) Total QALYs AQ -0.9** [-1.4, -0.4] 0.07** [0.04, 0.11] 5,227
  11. 11. Limitations • Likely to over-estimate health gains enjoyed by additional survivors as assumes those surviving past 30days experience: – Average life expectancy of the general population – Average QoL of the general population BUT • Only captures QALYs gained due to mortality reductions i.e. deaths averted • Does not capture pure QoL improvements
  12. 12. Potential extensions • Might want to update QoL values – later waves of HSE, other data sources (e.g. SF-6D from Understanding Society) • Could use condition-specific QoL estimates from audits if available Reference Meacock R, Kristensen S, Sutton M. (2014). The cost- effectiveness of using financial incentives to improve provider quality: a framework and application. Health Economics, 23, 1- 13.
  13. 13. Part 2: Using survival analysis to improve estimates of life year gains in policy evaluations
  14. 14. Introduction • Focus on methodology for estimating the impact of service interventions on length of life • Length of life is a key outcome in cost-effectiveness analysis: – Cost per life year gained – Cost per QALY • Evaluations attempting to take a lifetime horizon can use admin data to estimate changes in short-term mortality • Convert these to projected life year gains using published estimates of life expectancy from the general population
  15. 15. Previous approach • Estimate the impact of a programme in terms of 30 day mortality (binary outcome) • Estimated reductions in this mortality are then translated into life years gained – Patients dying within 30 days are attributed no survival days (effectively assumed to die instantly) – Patients surviving past 30 days are assigned the remaining age/gender-specific life expectancy of the general population
  16. 16. Limitations to previous approach • Length of life of patients affected by interventions is likely to differ from the general population – may lead to incorrect estimations of the impact on life years gained • True impact of policies on survival may be more complex, potentially impacting survival over the whole life course • Such longer-term effects not captured by evaluations focusing solely on mortality rates within e.g. 30 days
  17. 17. Proposed solution • Even with minimal data of 1 financial year available in many administrative data sets, possible to observe most patients for longer than 30 days • Prolonged follow-up often ignored in policy evaluations • Survival analysis is commonly used in clinical trials to extrapolate gains in life expectancy from observed trial data • Utilises all available follow-up information on patients rather than applying an arbitrary cut-off window
  18. 18. Aim • Examine whether the additional information available within admin data sets on survival beyond usual 30 days considered, albeit censored, can be used to improve accuracy of estimated life year gains • Demonstrate the feasibility and materiality of using parametric survival models commonly employed in clinical trials analysis to extrapolate future survival in policy evals
  19. 19. Motivating example • Previous CEA of Advancing Quality (AQ) P4P programme • Examine pneumonia patients only • Consider a typical situation – data on dates of admission and death are available for 1 financial year pre and post AQ • Parametric survival models to estimate the effect of AQ on survival over lifetime horizon • Compare to results obtained using previous method
  20. 20. Data • Hospital Episode Statistics linked to ONS death records • Patients admitted to hospital in England for pneumonia: – 2007/08 (pre-AQ) – 2009/10 (post-AQ) • Data period: 1st April 2007 – 31st March 2011 • Risk-adjustment: primary & secondary diagnoses (ICD-10), age, gender, financial quarter of admission, provider, admitted from own home vs institution, emergency vs transfer
  21. 21. Methods 1. Comparison of methods on a development cohort • Cohort of patients admitted to any hospital in England prior to introduction of AQ (2007/08) • Compare 3 methods for estimating remaining life years using data from 2007/08 only • Compare to observed survival of this cohort now available up to 31st March 2011
  22. 22. Purpose of development cohort • Illustrate difference in magnitude of estimated remaining life years of a patient population when: – Additional information available on survival past 30 days is utilised – Risk of death is taken from the population under investigation rather than general population figures • Exercise also used to select the most appropriate functional form for the survival models later used to evaluate AQ
  23. 23. Method i Simple application of published life expectancy tariffs • Simplified version of DANQALE applied in original evaluation (does not incorporate discounting or QoL) • 30 day mortality assessed as a binary outcome • Gender-specific general population life expectancy estimates at each single year of age (18 – 100) attached to patients surviving beyond 30 days to estimate remaining life expectancy
  24. 24. Method i Remaining life expectancy: 𝐿𝑖 𝑔𝑎 = 𝑠𝑖 30 ∙ 𝐿 𝑔𝑎 𝐿 𝑔𝑎 is the life expectancy of an individual of gender g who is currently aged a 𝑠𝑖 30 equals 1 if individual i survives more than 30 days and 0 otherwise
  25. 25. Method i Implicitly assumes that individuals surviving beyond 30 days after admission survive, on average, the life expectancy of the general population Will produce an inaccurate estimate of the actual life expectancy: a) Period of survival within 30 days is not incorporated b) Assumes life expectancy of individuals surviving beyond 30 days after admission will be equal to that of the general population of their age and gender Ignores information on observed survival available in data
  26. 26. Method ii Short-term observed survival plus application of published life expectancy estimates • Extend method i to utilise all information on mortality available within year of data (2007/08) • Can follow patients for between 1 – 365 days depending on admission date • For those that died during the period, number of days survived between admission and death are counted • Life expectancy again applied to those surviving past the end of the financial year
  27. 27. Method ii Improves on method i by: • Eliminating problem a) period of survival within 30 days is not incorporated • Reducing, but not eliminating, inaccuracies due to problem b) assuming life expectancy of those surviving beyond 30 days after admission will be equal to that of the general population of their age and gender
  28. 28. Method iii Extrapolation using survival models • Improve on method used for extrapolation by estimating parametric survival models on the observed year of data • Predict lifetime survival based on mortality rates of the population of interest • Considered six standard parametric models (exponential, Weibull, Gompertz, log-logistic, log-normal, generalised gamma)
  29. 29. Method iii Model fit assessed using: • AIC • Tests of whether restrictions on the parameters in the generalised gamma model suggest it could be reduced to the simpler models it nests • Examination of residual plots External validity of extrapolations assessed by comparing proportion of the cohort predicted to be alive at annual intervals to the observed survival now available to 31st March 2011
  30. 30. Method iii • In our case, while standard parametric models were able to fit the observed data well, the tails of these distributions did not correctly represent the pattern of future mortality • Hazard rates experienced by our patient cohort changed over time – extremely high-risk period shortly after emergency hospital admission not representative of lifetime risk of those surviving past this period
  31. 31. Method iii Solution = estimate survival in 2 separate models: • Short-term survival during the first year estimated on the observed 1 year of data • Extrapolation of long-term survival (1 year + after admission) based on a model estimated on data excluding first 30 days following admission Long-term models represent hazards experienced by our patient cohort after the initial high-risk period following emergency admission – still much higher than general population but significantly lower than when first admitted
  32. 32. Allowed us to estimate the effect of covariates on survival in both the observed and extrapolated period
  33. 33. Method iii Improves on method i by: • Again eliminating problem a) period of survival within 30 days is not incorporated • Using information on mortality risk from the patient population under investigation rather than general population estimates We compare results at each stage as assumptions are dropped – illustrates materiality of these developments
  34. 34. Application 2. Application to the evaluation of AQ • Stage 1 demonstrates the materiality of the difference survival analysis makes to the estimated life years remaining of our patient cohort • Stage 2 illustrates how these models can be used in an applied programme evaluation
  35. 35. Dichotomous difference-in-differences (DiD) design: 𝐿𝑖𝑗𝑡 = 𝑓(𝑎 + 𝑋′ 𝑏 + 𝑢𝑗 + 𝑣 𝑡 + 𝛿𝐷𝑗 1 ∙ 𝐷𝑡 2 + 𝜀𝑖𝑗𝑡) • 𝐿𝑖𝑗𝑡 is the life expectancy of individual i treated in hospital j at time t • f(∙) is the link function • X is the vector of case-mix covariates • 𝑢𝑗 are provider fixed effects • 𝑣 𝑡 are time fixed effects • 𝐷𝑗 1 is a dummy = 1 for hospitals that become part of AQ • 𝐷𝑡 2 is a dummy = 1 in the periods after the introduction of AQ • 𝜀𝑖𝑗𝑡 is an individual-specific error terms • 𝛿 is the DiD term, which is our coefficient of interest
  36. 36. Application First, consider situation where data on admissions and deaths are available for 1 financial year pre and post AQ • Pre AQ (2007/08) • Post AQ (2009/10) BUT, survival analysis can utilise additional follow-up on the pre-intervention group collected during same period as initial follow-up of the post-intervention group • Examine how life expectancy estimates are affected when including additional follow-up available (2008/09 – 2009/10) on pre-AQ group – should improve accuracy of estimates
  37. 37. Application Use average partial effects to calculate the effect of AQ on life expectancy Estimate life expectancy for individuals admitted to AQ hospitals in the post-AQ period under 2 scenarios: – DiD term set = 0 (absence of AQ) – DiD term set = 1 (presence of AQ) Compare results to those obtained using methods i and ii (linear regression on gen pop life expectancy estimates)
  38. 38. RESULTS Part 1: Development cohort
  39. 39. Annual mortality rates for females Age General population Patients admitted for pneumonia 2007/08 years % % (n deaths) 20 0.02 6.12 (98) 30 0.04 4.71 (191) 40 0.10 11.33 (309) 50 0.24 17.53 (291) 60 0.56 27.23 (584) 70 1.46 42.78 (783) 80 4.52 59.63 (1,469) 90 14.60 77.50 (1,142) 100 39.19 89.90 (109) • Highlights importance of using information on the risk of death from the patient cohort under investigation rather than general population figures when estimating remaining life years • Using gen pop figures would underestimate the annual mortality rate by a factor of between 2 (age > 100 years) and over 300 (age 20)
  40. 40. Exponential Weibull Gompertz Log- normal Log- logistic Generalised gamma Internal validity: AIC 326,943 288,141 291,563 283,531 285,139 283,386 External validity: Time point Predicted survival, % Observed survival, % 31/03/08 56.76 60.02 60.02 60.10 59.78 60.21 61.05 31/03/09 36.02 46.51 52.87 49.08 47.63 48.96 49.73 31/03/10 25.93 39.26 52.37 43.92 41.77 43.67 43.86 31/03/11 20.04 34.30 52.24 40.49 37.92 40.14 39.31 Internal and external validity of different parametric survival functions • Lowest AIC • Wald test confirmed does not reduce to the log-normal • Best performance on external validity – predicted proportion of cohort alive to within 1% of observed survival at each of 4 annual time points available
  41. 41. Method Assessment period Extrapolation method Those alive at end of assessment period, n (%) Estimated life years remaining, mean i Admission to 30 days Gen pop life expectancy 82,208 (72.56%) 13.15 ii Admission to end of financial year Gen pop life expectancy 69,158 (61.05%) 11.98 iii Admission to end of financial year Parametric survival models 69,158 (61.05%) 9.19 Comparison of estimates of remaining life years for patients admitted for pneumonia 2007/08 (n = 113,289) • Taking account of additional information on survival past 30 days reduced the estimate of average remaining life years by 9% (method ii) • Using survival models to extrapolate future survival reduced original estimate by 30% (method iii)
  42. 42. RESULTS Part 2: Application to the evaluation of AQ
  43. 43. Patients admitted in 2007/08 Patients admitted in 2009/10 North West Rest of England North West Rest of England n 17,993 95,296 19,946 106,365 Age at admission 71.7 72.2 71.9 72.8 Female, % 49.8% 48.7% 50.3% 49.1% Comorbidities, n 1.79 1.65 1.99 1.92 Unadjusted mortality within 30 days 28.4% 27.3% 25.6% 26.0% Dead by end of the financial year 40.7% 38.6% 37.3% 37.3% Descriptive statistics for patients admitted for pneumonia, by region and time period • Pre-AQ the unadjusted mortality rate was higher in the North West within 30 days of admission and persisted in the longer-term to end of financial year • Unadjusted mortality rates decreased in both regions, with a greater reduction in the North West – positive effect of AQ on mortality previously detected
  44. 44. Estimated effect of AQ on the remaining life expectancy of patients admitted to hospitals in the North West in 2009/10
  45. 45. Method i Method ii Method iii Source of life expectancy estimates Gen pop life tables Gen pop life tables Survival analysis using 1 financial year of follow-up Short-term model: Entry time = admission Long-term model: Entry time = 31 days post admission Estimation method OLS OLS Generalised gamma Generalised gamma DiD coefficient (robust t stat) 0.154 (2.39) 0.221 (3.04) 0.103 (2.64) 0.089 (1.71) Observations 239,600 239,600 239,600 156,860 Deaths, n 63,845 91,272 91,272 26,785 Life expectancy for patients admitted in North West 2009/10 13.218 11.982 9.041 Counterfactual estimate, life expectancy for patients in North West in absence of AQ 13.064 11.761 8.730 Effect of AQ on life expectancy for those admitted in North West 0.154 0.221 0.311
  46. 46. Interpretation • Lower absolute estimates of life expectancy both in the presence and absence of AQ were expected from methods ii and iii – additional deaths were observed and risk taken from patient cohort under investigation • Despite lower absolute estimates of life expectancy, estimate of the effect of AQ increased – indicates that AQ impacted on survival beyond 30 day post-admission window • Generalised gamma parameterized in the AFT metric – coefficients < 1 indicate time passes more slowly – failure (death) expected to occur later as a result of AQ
  47. 47. Method iii Source of life expectancy estimates Survival analysis using 1 financial year of follow-up Survival analysis using all available follow-up ( to 31/03/10) Short-term model: Entry time = admission Long-term model: Entry time = 31 days post admission Short-term model: Entry time = admission Long-term model: Entry time = 31 days post admission DiD coefficient (robust t stat) 0.103 (2.64) 0.089 (1.71) 0.142 (3.62) 0.101 (2.26) Observations 239,600 156,860 239,600 164,438 Deaths, n 91,272 26,785 110,747 45,290 Life expectancy for patients admitted in North West 2009/10 9.041 8.439 Counterfactual estimate, life expectancy for patients in North West in absence of AQ 8.730 8.059 Effect of AQ on life expectancy for those admitted in North West 0.311 0.380
  48. 48. Interpretation Utilising additional follow-up data available on pre-AQ group: • Increased precision of estimates • Slightly decreased estimated remaining life expectancy for the cohort both in the presence and absence of AQ • Further increased estimated treatment effect of AQ
  49. 49. Discussion Demonstrated: – Feasibility of using parametric survival models to extrapolate future survival in policy evaluations – Materiality of the impact this has on estimates of remaining life years of a patient cohort and a policy treatment effect Detected impact of AQ beyond 30 day window usually assessed shows advantage of survival analysis – ability to capture effects of policies over the whole life course In pre- and post- evaluation design, survival models can be developed on the pre-intervention population and predictive performance evaluated against observed follow-up available during post-intervention period – external validity
  50. 50. Future work • For estimates of life years gained to be used in CEA, the stream of remaining life years need to be adjusted for QoL and discounted – quite simple extensions • More sophisticated survival models Reference Meacock, Sutton, Kristensen, Harrison. (2017). Using survival analysis to improve estimates of life year gains in policy evaluations. Medical Decision Making, 37, 415-426.
  51. 51. Overall conclusions • Development of methods and applications of economic evaluation of service interventions has the potential to improve allocative efficiency • Still a LONG way to go, but (hopefully) offered some useful approaches • Both strands of health economics have made impressive methodological progress in different aspects of evaluation – could learn a lot from each other
  52. 52. THANK YOU Contact details: rachel.meacock@manchester.ac.uk @RachelMeacock
  53. 53. Method i Remaining life expectancy: 𝐿𝑖 𝑔𝑎 = 𝑠𝑖 30 ∙ 𝐿 𝑔𝑎 𝐿 𝑔𝑎 is the life expectancy of an individual of gender g who is currently aged a 𝑠𝑖 30 equals 1 if individual i survives more than 30 days and 0 otherwise
  54. 54. Method ii 𝐿𝑖 𝑔𝑎 = 𝑠𝑖 𝑡∗ ∙ 𝐿 𝑔𝑎 + (1 - 𝑠𝑖 𝑡∗ ) ∙ (𝑡𝑖 ϯ - 𝑡𝑖 0 ) Where 𝑠𝑖 𝑡∗ is a binary indicator equal to 1 if individual i survives to the end of the observation period 𝑡∗ 𝑡𝑖 ϯ is the date of death for individuals who die before the end of the observation period 𝑡𝑖 0 is the date of admission
  55. 55. Method iii
  56. 56. Following estimation of survival models, created additional rows of data for each individual for each possible future year up to age 100 Estimated the probability of surviving to that year, allowing for the progression of time and increments in age – analogous to estimating transition probabilities in a Markov model: 𝑚𝑖 𝑡 (𝑎𝑖0, 𝑥𝑖) = 𝑠 𝑖𝑡 (𝑎 𝑖𝑡,𝑥 𝑖) 𝑠 𝑖,𝑡−1 (𝑎 𝑖𝑡,𝑥 𝑖) – 1 𝑚𝑖 𝑡 is the probability that individual i will die by time t, given that they have survived to time t-1 𝑠𝑖𝑡 is the probability that individual i will survive to time t, given the values of their covariates x and their age 𝑎𝑖 at the time of their admission
  57. 57. Then calculated the individual’s life expectancy using the sum of the probability of surviving to the end of the first year and the survival rates for each subsequent year, to a max age of 100: 𝐿𝑖 = 1 − 𝑚𝑖 1 ∙ 𝑡∗ − 𝑡𝑖 0 + 𝑗= 𝑎 𝑖0+1 𝐴 𝑠𝑖,𝑗−𝑎 𝑖0 𝑎𝑖𝑗, 𝑥𝑖 ∙ (1 − 1 𝑚𝑖 𝑗+1− 𝑎 𝑖0 ) 𝐿𝑖 is the life expectancy of individual i 𝑚𝑖 1 is the probability that individual i will die by the end of the first year 𝑡∗ - 𝑡𝑖 0 is the length of time between the individual’s admission date and the end of the first year

×