Download as PDF, PPTX
More Related Content
Similar to Illustrating uncertainty in extrapolating evidence for cost-effectiveness modelling
![PERI-PROSTHETIC FRACTURE NAIL-PLATE CONSTRUCT [NPC].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/drarunkumardrmohamedashrafperiprostheticfrasturenail-plateconstructnpc-260209164459-7e9d15a1-thumbnail.jpg?width=640&height=640&fit=bounds)
![CTEV [ clubfoot] DR ARUN LAL ,DR MOHAMED ASHRAF travancore medical college k...](https://cdn.slidesharecdn.com/ss_thumbnails/ctevclubfootdrarunlaldrmohamedashraftravancoremedicalcollegekollamkeralaindia-260208063247-18fc466c-thumbnail.jpg?width=640&height=640&fit=bounds)
Illustrating uncertainty in extrapolating evidence for cost-effectiveness modelling
- 1. The CLAHRC Yorkshire &Humber Illustrating uncertainty in extrapolating evidence for cost- effectiveness modelling Laura Bojke
- 2. The CLAHRC Yorkshire &Humber Extrapolation project team • Stephen Palmer, Andrea Manca, Ronan Mahon, Miqdad Asaira (University of York) • Alan Brennan (PI), John Stevens, Nick Latimer, Paul Tappenden, Suzy Paisley, Kate Ren (University of Sheffield) • Keith Abrams (University of Leicester) • Chris Jackson (University of Cambridge)
- 3. The CLAHRC Yorkshire &Humber • The need for extrapolation • Extrapolation methods – Not extrapolation from surrogate outcomes • Uncertainty in extrapolation • Approaches to dealing with uncertainty in extrapolation – Examples • Areas for further research Structure of presentation
- 4. The CLAHRC Yorkshire &Humber “estimating beyond the original observation range” • An appropriate time horizon for evaluation – All (incremental) positive/negative effects observed – Patients lifetime when there are mortality effects • Evidence base falls short of this - censoring – High costs of research – Loss to follow up – Early market entry • Modelling to extrapolate short term outcomes – Involves assumptions and may be data sparse Impetus for extrapolation
- 5. The CLAHRC Yorkshire &Humber Evidence gap resulting from time horizon mismatch Just extrapolation? Temporal Uncertainty in Cost-effectiveness Decision Models. Mahon R, Manca A, Bojke L, Palmer, S Jackson C, et al.
- 6. The CLAHRC Yorkshire &Humber When does the mismatch matter? • Not just the time difference between observed and unobserved • Relates to data maturity – Number of people that have experienced the event of interest • Earlier outcomes may be of more value – Proportion of costs & QALYs attributable to observed period – Discounting
- 7. The CLAHRC Yorkshire &Humber Mature data Mean overall survival gain with aflibercept plus FOLFIRI vs placebo plus FOLFIRI in patients with previously treated metastatic colorectal cancer. F Joulain, I Proskorovsky, C Allegra, J Tabernero, M Hoyle, S U Iqbal and E Van Cutsem.
- 8. The CLAHRC Yorkshire &Humber Immature data
- 9. The CLAHRC Yorkshire &Humber Extrapolating TTE parameters • Assumptions about how trends will continue – Progression free or overall survival in cancer trials • Estimation involves: (1) modelling the observed data (Kaplan-Meier); (2) modelling the unobserved data period. – Statistical models (often parametric) are fitted to observed data • Choice of model informed by goodness of fit parameters – Fitted model is used to extrapolate the un-observed period to determine the TTE – Assume hazard trends will continue
- 10. The CLAHRC Yorkshire &Humber Choice of distribution • Choice of parametric distribution to fit: – Exponential, Weibull, log-normal etc. • Some advocate use of exponential as the default – Proportional hazards, constant hazards • Different assumptions/ survival estimates • Data limitations – IPD or aggregate (constant hazard)
- 11. The CLAHRC Yorkshire &Humber Extrapolating non-TTE parameters • Repeated measurement of individuals over time – Genuinely discrete or genuinely continuous • Similar principles to TTE – Set of observations on one subject tends to be inter- correlated • Driven by data availability – Aggregate = discretised (Markov models) – IPD = patient experienced models, continuous outcomes – regression, e.g. risk equations
- 12. The CLAHRC Yorkshire &Humber Extrapolating resources use/costs & utilities • Typically models employ a simplistic approach to extrapolation of costs and utilities – Homogeneous w.r.t both time and patients' characteristics – Follow the dynamics of associated TTE parameters • Resource use/costs and utilities may be more nuanced – Adaptation to health state – Uncertainty may not resolve over time/with further evidence
- 13. The CLAHRC Yorkshire &Humber Uncertainty in extrapolation • Others have termed this ‘temporal’ uncertainty • Extent of uncertainty is difficult to determine • Uncertainty cannot be resolved (now), because we cannot observe the future • Efforts should be made to characterise any uncertainty − Accurate estimates of long-term cost-effectiveness − Better adoption decisions − Assess the need for further research − Input into the design of further research − Delay decisions?
- 14. The CLAHRC Yorkshire &Humber Does uncertainty in extrapolation matter? Just extrapolation? Temporal Uncertainty in Cost-effectiveness Decision Models. Mahon R, Manca A, Bojke L, Palmer, S Jackson C, et al.
- 15. The CLAHRC Yorkshire &Humber “primary source of extrapolation uncertainty in decision model results is the choice of survival projection function rather than sampling variation” Bagust & Beale, 2014 in response to Latimer, 2013
- 16. The CLAHRC Yorkshire &Humber How should uncertainty be assessed? • NICE recommends that any extrapolation should be assessed by ‘‘both clinical and biological plausibility of the inferred outcome as well as its coherence with external data sources’’ – Does not suggest specific methods to do this • Latimer, 2013 recommends an assessment of plausibility – Concerns may be context specific
- 17. The CLAHRC Yorkshire &Humber Model selection • Assessment of goodness-of-fit for observed – Log-cumulative hazard, residuals plots, AIC/BIC etc – Suggested the focus should be on fit for stable portion of data • Lots of things that prevent good model fit – More flexible model forms, generalised gamma • A model that fits the sample data well may not provide good long-term predictions • Clinical plausibility • Causally known associations
- 18. The CLAHRC Yorkshire &Humber Just scenarios? • Where there is no evidence to form an opinion about the plausibility of assumptions/do not want to rely on this – Deterministic sensitivity analysis – Illustrate how conclusions may change if the evidence were to change and possibly inform the need for further validation – Not easy for decision-makers to interpret • May implicitly average different scenarios
- 19. The CLAHRC Yorkshire &Humber Model averaging • Explore uncertainty in the true underlying distribution for survival extrapolation – Bayesian view of model averaging – Likelihood used to measure fit of a model to observed data (plausibility) – Unlikely scenarios: down-weighted/excluded – Recommended that models should ‘stake out the corners in the model space’ • Implications for extrapolation uncertainty – Model-averaged inferences will generally have greater uncertainty
- 20. The CLAHRC Yorkshire &Humber Discrepancy parameters • Where there are many sources of uncertainty in a model – Strong, et al method to identify the most important sources of structural uncertainty – Discrepancy parameters are added to intermediate outputs of the model, rather than model inputs. • e.g. the years of life spent in a health state could be replaced by LY + delta_LY, where the discrepancy parameter delta_LY is represented as a probability distribution based on experts priors – PSA to quantify which of the discrepancy parameters are associated with the greatest variations in net benefit
- 21. The CLAHRC Yorkshire &Humber Stability of survival extrapolation • Negrin, et al, 2016 explore uncertainty about the stability of extrapolated parameters over time – Explore past behaviour of distributions • Built intermediate data sets (4, 5, 6 and 7 year) – Numerous survival models fitted to 8 year data for 2 hip prosthesis types separately (Charley, Spectron) • Two additional distributions to reflect stability over past behaviour (optimistic and sceptical) • BMA(1) using BIC (1/6 weight to 6 distributions) • BMA(2) questions the value of BIC to select or average across models (1/3 weight assigned to optimistic, sceptical, others)
- 22. The CLAHRC Yorkshire &Humber Negrin, Nam, Briggs. Bayesian solutions for handling uncertainty in survival extrapolation. Medical Decision Making. 2016 forthcoming
- 23. The CLAHRC Yorkshire &Humber Using external sources of data • Sources: administrative data, disease registries, cohort studies – Longitudinal data over an appropriate time horizon with sufficient follow up points – Cross sectional data with sufficient coverage of patients • Assumed relationship between internal and external data (baseline and treatments) • Combine using Bayesian estimation
- 24. The CLAHRC Yorkshire &Humber Elicitation • Use formally elicited priors • Represents current level of knowledge regarding the uncertainty of interest – Expressed quantitatively • Uncertainty about how this can be implemented: – Alternative survivor function? – Weights for survivor functions, e.g. TIDI – Synthesis with internal data? – Could elicit at multiple time points – how?
- 25. The CLAHRC Yorkshire &Humber Elicitation (2) • Complexity of elicitation task – Who are the relevant experts – How can we weight experts according to accuracy – Complex parameters difficult to elicit – Time requirements – Synthesis • Eliciting uncertainty about priors – Needed to fully characterise uncertainty and inform long term follow up
- 26. The CLAHRC Yorkshire &Humber Case study: Extrapolation project • Cost-effectiveness of biologics for PsA – Probabilistic cohort model – Time horizon = 40 years (3-month cycles) – Initial response using PsARC @ 3-months – Associated HAQ gain – Assumptions that biologics halt progression – Constant rate of progression on biologics (mean = 0) • Different ways in which this data can be utilised in the model
- 27. The CLAHRC Yorkshire &Humber 1) Initial HAQ gain due to treatment 2)Long term HAQ trajectory while on treatment 3)Rebound of HAQ once withdrawn from treatment 4)Long term HAQ trajectory post withdrawal from treatment 0 0.5 1 1.5 2 2.5 3 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 HAQScore Time (Years) 2. HAQ on treatment 1. Initial HAQ Gain Natural History 3. HAQ Rebound 4. HAQ Post- Withdrawal Treatment Arm Issues regarding HAQ trajectory
- 28. The CLAHRC Yorkshire &Humber Impact of uncertainties in PsA model Cumulative NMB over time base case 3-month HAQ change of 0. Within trial period Breakeven point etanercept
- 29. The CLAHRC Yorkshire &Humber Exploring long term HAQ trajectory for treatment responders • Key driver for cost-effectiveness. • Assumed mean of 0 + expert elicitation exercise (SD) • But don’t know: – Constant HAQ gradient? – Independent HAQ gradient at each 3-month model cycle? • Limited by lack of external data.
- 30. The CLAHRC Yorkshire &Humber Impact of different HAQ trajectories
- 31. The CLAHRC Yorkshire &Humber Constant HAQ gradient over time • Randomly draw HAQ change from a normal distribution (mean = 0 and SD = 0.022) • Apply this as the constant 3-month HAQ score assuming perfect serial correlation. • Process is repeated for each iteration of the model
- 32. The CLAHRC Yorkshire &Humber Cumulative NMB over time – constant HAQ gradient over time. Breakeven point etanercept Within trial period
- 33. The CLAHRC Yorkshire &Humber Independent HAQ gradient at each cycle • Previously perfectly serially correlated HAQ trajectories • Independently sampled HAQ scores for each cycle • Repeatedly randomly draw HAQ changes from a normal distribution with mean 0 and standard deviation 0.022
- 34. The CLAHRC Yorkshire &Humber Cumulative NMB over time - independent HAQ gradient at each 3-month model cycle. Breakeven point etanercept
- 35. The CLAHRC Yorkshire &Humber Comparing the two scenarios Constant HAQ gradient over time Independent HAQ gradient at each cycle Palliative Care Probability cost-effective (£20,000 per QALY) 0.30 0.38 Etanercept 0.31 0.39 Infliximab 0.04 0.00 Adalimumab 0.18 0.17 Golimumab 0.18 0.07 EVPI (population) £768 mil £378 mil
- 36. The CLAHRC Yorkshire &Humber Conclusions • Models may be well characterised over the observed period, but there may be considerable uncertainty regarding behaviour over time. • None of the methods described can fully describe the extent of extrapolation uncertainty. – Statistical models for extrapolation cannot be fully validated. • Focus on which influence conclusions: adoption decision, decision uncertainty and VOI. – Degree of effort should be proportional to the influence of uncertainty.
- 37. The CLAHRC Yorkshire &Humber Areas for further research • Ability to validate for ‘crucial’ uncertainties: – Internal/external validity – Use of external evidence/experts priors • How to generate experts priors for extrapolation – Which parameters to elicit • Link to OIR/delayed decisions – Break even point important • Discounting out the problem
- 38. The CLAHRC Yorkshire &Humber (some) References • Methods of Extrapolating RCT Evidence for Use in Economic Evaluation Models, MRC report. • Extrapolating Survival from Randomized Trials Using External Data: A Review of Methods. Jackson C, et al. Medical Decision Making, 2016. • Survival Analysis and Extrapolation: Modeling of Time-to-Event Clinical Trial Data for Economic Evaluation: An Alternative Approach. Medical Decision Making. Bagust A & Beale S. 2014. • Davies C, Briggs A, Lorgelly P, et al. The “hazards” of extrapolating survival curves. Medical Decision Making. 2013; 33 (3). • Negrin, Nam, Briggs. Bayesian solutions for handling uncertainty in survival extrapolation. Medical Decision Making. 2016 forthcoming. • NICE DSU Technical support document 14: Survival analysis for economic evaluations alongside clinical trials: Extrapolation with patient-level data. Latimer N. 2011.