Projecting ‘Time-to-event’ Outcomes in
Technology Assessment: an Alternative Paradigm
Adrian Bagust
CHE Economic Evaluatio...
Context
 LRiG is one of nine independent multi-disciplinary
academic research groups providing evidence
assessment for NI...
Problem, Objective & Focus
 TTE / Survival outcome estimation is often the
major source of uncertainty in NICE appraisals...
Survival Analysis: Kaplan-Meier
 Basic data comes from Kaplan-Meier analysis of
observed events
 K-M is a non-parametric...
Censoring matters
Censoring
at last
observation
biases end
of the curve
and leads to
misfitting
parametric
functions

4
‘Standard’ method for projective modelling
 Fit a limited set of ‘simple’ functions to the whole
trial data set (i.e. nor...
Problems with the ‘Standard’ method (1)
 Essentially descriptive – not clear whether a good
description of known data wil...
Problems with the ‘Standard’ method (2)
 All standard functions are well-behaved smooth
continuous formulations to descri...
Problems with the ‘Standard’ method (3)
 “AIC can tell nothing about the quality of the
model in an absolute sense. If al...
Standard functions give very different results
1.00

K-M data
K-M confidence limits
Limit of mature data
Weibull model
Exp...
LRiG’s distinctive approach











Primacy of experimental data over projections
Understand the trial and th...
The nature of clinical trials









Trials are about altering risk
Trials look for differences
Trial patients a...
Examining the data
H(t) vs t plot
shows long-term

parallel trends
more clearly than
Ln(H(t)) vs Ln(t) plot

12
Typical oncology trial
 New treatment initiated on Day 1 and continues
either for a specific maximum duration, or until
p...
Typical oncology trial - PFS

14
Typical oncology trial - OS

15
Typical oncology trial – OS Hazard

16
Typical oncology trial - PPS

17
Post-progression
survival
Frequently, after
progression there is no
difference between
treatments – a common
PPS fixed ris...
Underlying question: how to fit multi-phase
convolution functions to empirical data
 Exponential PFS Exponential PPS is...
1.0
Y K-M data

Case study

0.9

X K-M data
Fitted joint exponential model

0.8
0.7

No difference for PPS

PPS

0.6
0.5
0...
Basic issue: multi-phase convolution functions
 Other combinations are more difficult
in usable formExponential
e.g. We...
Heterogeneity & mixed models
 T
 T
 T

22
Case Study: segmented model & PH assumption

23
Case Study: segmented model & PH assumption

24
Case Study: segmented model & PH assumption

HR = 1.34

HR = 1.71

HR = 1.00

25
Case Study: segmented model & PH assumption

26
Is projective modelling always necessary?

27
Conclusion
 We consider that the ‘standard method’ of projection
does not provide an adequate basis for secondary
analysi...
Discussion

29
An example: data hypothesis confirmation
 Long-term project begun in 1997 on costeffectiveness in type 2 diabetes
 A...
Finding data…
 We identified an historic clinical trial (Belfast Diet
Study), which looked only at the effect of
controll...
Is there a combined trend to explain data?

32
Time-shifting cohorts shows 2-phase pattern

33
Lab research indicates mechanism

Topp B, et al A model of b-cell mass, insulin and glucose
kinetics J Theor Biol 2000; 20...
Upcoming SlideShare
Loading in …5
×

Projecting ‘time to event’ outcomes in technology assessment: an alternative paradigm

525 views
392 views

Published on

CHE economic evaluation seminar presented by Professor Adrian Bagust 13th February 2014

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
525
On SlideShare
0
From Embeds
0
Number of Embeds
75
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Projecting ‘time to event’ outcomes in technology assessment: an alternative paradigm

  1. 1. Projecting ‘Time-to-event’ Outcomes in Technology Assessment: an Alternative Paradigm Adrian Bagust CHE Economic Evaluation seminar 13th February 2014
  2. 2. Context  LRiG is one of nine independent multi-disciplinary academic research groups providing evidence assessment for NICE technology appraisals  Reliant on drug manufacturer for clinical evidence – usually 1 or 2 key RCTs for STA topic  Most trials close early for commercial reasons  Access to trial data restricted to summaries and publications – full patient data withheld. 1
  3. 3. Problem, Objective & Focus  TTE / Survival outcome estimation is often the major source of uncertainty in NICE appraisals  Problem: How to estimate life time survival gains from incomplete/immature trial data?  Objective: To estimate the expected mean survival beyond the available trial data (Kaplan-Meier)  Focus: Appraisal of interventions for (mainly) advanced/metastatic cancers 2
  4. 4. Survival Analysis: Kaplan-Meier  Basic data comes from Kaplan-Meier analysis of observed events  K-M is a non-parametric technique, which accumulates risk per unit of time between events  K-M gives unbiased estimates of survival vs time provided any censoring is uninformative  Mean survival can be estimated as the area under the curve (AUC) of the K-M survival plot 3
  5. 5. Censoring matters Censoring at last observation biases end of the curve and leads to misfitting parametric functions 4
  6. 6. ‘Standard’ method for projective modelling  Fit a limited set of ‘simple’ functions to the whole trial data set (i.e. normal, exponential, Weibull, Gompertz, logistic, gamma, log-normal, log-logistic, extreme value)  Select ‘best fit’ function based on AIC / BIC scores  Apply selected function to model whole period  Often use a single model to represent both trial arms, with treatment as a covariate “For every complex problem there is an answer that is clear, simple and wrong.” H.L Mencken 5
  7. 7. Problems with the ‘Standard’ method (1)  Essentially descriptive – not clear whether a good description of known data will give reliable estimates of unknown future events (projective)  Mechanistic process – is the selected function suitable/appropriate? Is there causal logic?  No account taken of trial design (inclusion/exclusion criteria, drug kinetics/dynamics, drug response/resistance, monitoring protocols) 6
  8. 8. Problems with the ‘Standard’ method (2)  All standard functions are well-behaved smooth continuous formulations to describe risk varying over time according to a single mechanism  Clinical trials are designed to induce changes in risk trajectories over time: treatment is introduced, achieves full efficacy, loses efficacy, another treatment may be offered, palliative care 7
  9. 9. Problems with the ‘Standard’ method (3)  “AIC can tell nothing about the quality of the model in an absolute sense. If all candidate models fit poorly, AIC will not give any warning of that.” Wikipedia  Projection with standard functions can be highly sensitive to the choice of model despite minimal differences in ‘fit’ scores. 8
  10. 10. Standard functions give very different results 1.00 K-M data K-M confidence limits Limit of mature data Weibull model Exponential model 0.75 Log-logistic model Log-normal model Gompertz model PFS Gamma model 0.50 0.25 0.00 0 5 10 15 20 25 Years from randomization 9
  11. 11. LRiG’s distinctive approach           Primacy of experimental data over projections Understand the trial and the context Focus on the primary objective (beyond the trial) Search for meaning - all effects have a cause Hypothesis formulation and testing Avoid preconceptions (no ‘painting by numbers’) Realism - no effect, no cause (prove otherwise!) Parsimony – KISS / Occam’s razor Question everything….. All models are wrong - Nature makes fools of us all! 10
  12. 12. The nature of clinical trials         Trials are about altering risk Trials look for differences Trial patients are selected for ‘success’ Few treatments work immediately Few treatments work indefinitely Few treatments work for everyone Inclusion/exclusion criteria impact on survival Trial populations change during the trial 11
  13. 13. Examining the data H(t) vs t plot shows long-term parallel trends more clearly than Ln(H(t)) vs Ln(t) plot 12
  14. 14. Typical oncology trial  New treatment initiated on Day 1 and continues either for a specific maximum duration, or until patient condition worsens (progression)  Progression-free survival (PFS) = time to disease progression or death from any cause  Overall survival (OS) = time to death from any cause  Post-progression survival (PPS)* = OS – PFS  PPS may involve several subsequent different phases of treatment * Not usually reported 13
  15. 15. Typical oncology trial - PFS 14
  16. 16. Typical oncology trial - OS 15
  17. 17. Typical oncology trial – OS Hazard 16
  18. 18. Typical oncology trial - PPS 17
  19. 19. Post-progression survival Frequently, after progression there is no difference between treatments – a common PPS fixed risk applies. This corresponds to parallel risks at the end of the overall survival plot. 18
  20. 20. Underlying question: how to fit multi-phase convolution functions to empirical data  Exponential PFS Exponential PPS is straightforward PFS(t) = exp(– r1.t); PPS(t) = exp(– r2.t) OS(t) = p * PFS(t) + (1 - p) * PFS(t)  PPS(t) = p * exp(– r1.t) + (1- p) * {r1.exp(– r1.t) – r2.exp(– r2.t)} / (r2 - r1), where p = proportion of progression events which are fatal 19
  21. 21. 1.0 Y K-M data Case study 0.9 X K-M data Fitted joint exponential model 0.8 0.7 No difference for PPS PPS 0.6 0.5 0.4 0.3 Difference for PFS 0.2 0.1 0.0 0 Convolution of two exponential functions generates short-term inflection in OS curve 100 200 300 400 500 600 700 800 900 1000 Days 1.0 Y K-M survival 0.9 Y exponential convolution model X K-M survival 0.8 X exponential convolution model Overall Survival 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 100 200 300 400 500 Days 600 700 800 900 20 1000
  22. 22. Basic issue: multi-phase convolution functions  Other combinations are more difficult in usable formExponential e.g. Weibull (fitting to empirical data)  Representing Weibull as a mixture of exponentials is promising: “…any Weibull distribution with shape parameter less than 1 arises as a mixture of exponentials. Also the exponential distribution itself arises as a mixture of Weibull distributions with fixed shape parameter p, so long as p > 1.” Jewell NP Mixtures of exponential distributions Annals of Statistics 1982, 10(2): 479-484 21
  23. 23. Heterogeneity & mixed models  T  T  T 22
  24. 24. Case Study: segmented model & PH assumption 23
  25. 25. Case Study: segmented model & PH assumption 24
  26. 26. Case Study: segmented model & PH assumption HR = 1.34 HR = 1.71 HR = 1.00 25
  27. 27. Case Study: segmented model & PH assumption 26
  28. 28. Is projective modelling always necessary? 27
  29. 29. Conclusion  We consider that the ‘standard method’ of projection does not provide an adequate basis for secondary analysis of RCT data and the projection of time-toevent outcomes data to end of life, nor does it give sufficient regard to the primacy of experimental data.  The alternative approach outlined moves away from a limited mechanistic procedure, and avoids many of its unwarranted assumptions.  We believe that modelling should pursue a scientific approach based on observation, hypothesis formulation and testing to identify relevant and informative models.  We are seeking to develop further the analytical methods to support this approach. 28
  30. 30. Discussion 29
  31. 31. An example: data hypothesis confirmation  Long-term project begun in 1997 on costeffectiveness in type 2 diabetes  After working with different models & methods, we concluded that better understanding was required of the natural history of the disease  How does mildly elevated blood glucose develop until patients are dependent on insulin?  How do drugs affect this progression of disease? 30
  32. 32. Finding data…  We identified an historic clinical trial (Belfast Diet Study), which looked only at the effect of controlled diet – no drugs at all!  We made contact with the PI who agreed to give access to detailed data on results for re-analysis  We analysed the changes in beta-cell function (from HOMA model) to look for temporal trends, stratifying by time to diet failure 31
  33. 33. Is there a combined trend to explain data? 32
  34. 34. Time-shifting cohorts shows 2-phase pattern 33
  35. 35. Lab research indicates mechanism Topp B, et al A model of b-cell mass, insulin and glucose kinetics J Theor Biol 2000; 206:605-19 34

×