CHE economic evaluation seminar presented by Professor Adrian Bagust 13th February 2014

1. Projecting ‘Time-to-event’ Outcomes in
Technology Assessment: an Alternative Paradigm
Adrian Bagust
CHE Economic Evaluation seminar
13th February 2014

2. Context
 LRiG is one of nine independent multi-disciplinary
academic research groups providing evidence
assessment for NICE technology appraisals
 Reliant on drug manufacturer for clinical evidence
– usually 1 or 2 key RCTs for STA topic
 Most trials close early for commercial reasons
 Access to trial data restricted to summaries and
publications – full patient data withheld.
1

3. Problem, Objective & Focus
 TTE / Survival outcome estimation is often the
major source of uncertainty in NICE appraisals
 Problem: How to estimate life time survival gains
from incomplete/immature trial data?
 Objective: To estimate the expected mean survival
beyond the available trial data (Kaplan-Meier)
 Focus: Appraisal of interventions for (mainly)
advanced/metastatic cancers
2

4. Survival Analysis: Kaplan-Meier
 Basic data comes from Kaplan-Meier analysis of
observed events
 K-M is a non-parametric technique, which
accumulates risk per unit of time between events
 K-M gives unbiased estimates of survival vs time
provided any censoring is uninformative
 Mean survival can be estimated as the area under
the curve (AUC) of the K-M survival plot
3

5. Censoring matters
Censoring
at last
observation
biases end
of the curve
and leads to
misfitting
parametric
functions
4

6. ‘Standard’ method for projective modelling
 Fit a limited set of ‘simple’ functions to the whole
trial data set (i.e. normal, exponential, Weibull,
Gompertz, logistic, gamma, log-normal, log-logistic,
extreme value)
 Select ‘best fit’ function based on AIC / BIC scores
 Apply selected function to model whole period
 Often use a single model to represent both trial
arms, with treatment as a covariate
“For every complex problem there is an
answer that is clear, simple and wrong.”
H.L Mencken
5

7. Problems with the ‘Standard’ method (1)
 Essentially descriptive – not clear whether a good
description of known data will give reliable
estimates of unknown future events (projective)
 Mechanistic process – is the selected function
suitable/appropriate? Is there causal logic?
 No account taken of trial design (inclusion/exclusion
criteria, drug kinetics/dynamics, drug
response/resistance, monitoring protocols)
6

8. Problems with the ‘Standard’ method (2)
 All standard functions are well-behaved smooth
continuous formulations to describe risk varying
over time according to a single mechanism
 Clinical trials are designed to induce changes in
risk trajectories over time: treatment is introduced,
achieves full efficacy, loses efficacy, another
treatment may be offered, palliative care
7

9. Problems with the ‘Standard’ method (3)
 “AIC can tell nothing about the quality of the
model in an absolute sense. If all candidate
models fit poorly, AIC will not give any warning of
that.” Wikipedia
 Projection with standard functions can be highly
sensitive to the choice of model despite minimal
differences in ‘fit’ scores.
8

10. Standard functions give very different results
1.00
K-M data
K-M confidence limits
Limit of mature data
Weibull model
Exponential model
0.75
Log-logistic model
Log-normal model
Gompertz model
PFS
Gamma model
0.50
0.25
0.00
0
5
10
15
20
25
Years from randomization
9

11. LRiG’s distinctive approach










Primacy of experimental data over projections
Understand the trial and the context
Focus on the primary objective (beyond the trial)
Search for meaning - all effects have a cause
Hypothesis formulation and testing
Avoid preconceptions (no ‘painting by numbers’)
Realism - no effect, no cause (prove otherwise!)
Parsimony – KISS / Occam’s razor
Question everything…..
All models are wrong - Nature makes fools of us all!
10

12. The nature of clinical trials








Trials are about altering risk
Trials look for differences
Trial patients are selected for ‘success’
Few treatments work immediately
Few treatments work indefinitely
Few treatments work for everyone
Inclusion/exclusion criteria impact on survival
Trial populations change during the trial
11

13. Examining the data
H(t) vs t plot
shows long-term
parallel trends
more clearly than
Ln(H(t)) vs Ln(t) plot
12

14. Typical oncology trial
 New treatment initiated on Day 1 and continues
either for a specific maximum duration, or until
patient condition worsens (progression)
 Progression-free survival (PFS) = time to disease
progression or death from any cause
 Overall survival (OS) = time to death from any cause
 Post-progression survival (PPS)* = OS – PFS
 PPS may involve several subsequent different
phases of treatment
* Not usually reported
13

15. Typical oncology trial - PFS
14

16. Typical oncology trial - OS
15

17. Typical oncology trial – OS Hazard
16

18. Typical oncology trial - PPS
17

19. Post-progression
survival
Frequently, after
progression there is no
difference between
treatments – a common
PPS fixed risk applies.
This corresponds to
parallel risks at the end
of the overall survival
plot.
18

20. Underlying question: how to fit multi-phase
convolution functions to empirical data
 Exponential PFS Exponential PPS is
straightforward
PFS(t) = exp(– r1.t); PPS(t) = exp(– r2.t)
OS(t) = p * PFS(t) + (1 - p) * PFS(t)  PPS(t)
= p * exp(– r1.t)
+ (1- p) * {r1.exp(– r1.t) – r2.exp(– r2.t)} / (r2 - r1),
where p = proportion of progression events which are fatal
19

21. 1.0
Y K-M data
Case study
0.9
X K-M data
Fitted joint exponential model
0.8
0.7
No difference for PPS
PPS
0.6
0.5
0.4
0.3
Difference for PFS
0.2
0.1
0.0
0
Convolution of two
exponential functions
generates short-term
inflection in OS curve
100
200
300
400
500
600
700
800
900
1000
Days
1.0
Y K-M survival
0.9
Y exponential convolution model
X K-M survival
0.8
X exponential convolution model
Overall Survival
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0
100
200
300
400
500
Days
600
700
800
900
20
1000

22. Basic issue: multi-phase convolution functions
 Other combinations are more difficult
in usable formExponential
e.g. Weibull (fitting to empirical data)
 Representing Weibull as a mixture of exponentials is
promising:
“…any Weibull distribution with shape parameter less than 1
arises as a mixture of exponentials. Also the exponential
distribution itself arises as a mixture of Weibull distributions
with fixed shape parameter p, so long as p > 1.”
Jewell NP Mixtures of exponential distributions Annals of Statistics 1982, 10(2):
479-484
21

23. Heterogeneity & mixed models
 T
 T
 T
22

24. Case Study: segmented model & PH assumption
23

25. Case Study: segmented model & PH assumption
24

26. Case Study: segmented model & PH assumption
HR = 1.34
HR = 1.71
HR = 1.00
25

27. Case Study: segmented model & PH assumption
26

28. Is projective modelling always necessary?
27

29. Conclusion
 We consider that the ‘standard method’ of projection
does not provide an adequate basis for secondary
analysis of RCT data and the projection of time-toevent outcomes data to end of life, nor does it give
sufficient regard to the primacy of experimental data.
 The alternative approach outlined moves away from a
limited mechanistic procedure, and avoids many of
its unwarranted assumptions.
 We believe that modelling should pursue a scientific
approach based on observation, hypothesis
formulation and testing to identify relevant and
informative models.
 We are seeking to develop further the analytical
methods to support this approach.
28

30. Discussion
29

31. An example: data hypothesis confirmation
 Long-term project begun in 1997 on costeffectiveness in type 2 diabetes
 After working with different models & methods, we
concluded that better understanding was required
of the natural history of the disease
 How does mildly elevated blood glucose develop
until patients are dependent on insulin?
 How do drugs affect this progression of disease?
30

32. Finding data…
 We identified an historic clinical trial (Belfast Diet
Study), which looked only at the effect of
controlled diet – no drugs at all!
 We made contact with the PI who agreed to give
access to detailed data on results for re-analysis
 We analysed the changes in beta-cell function
(from HOMA model) to look for temporal trends,
stratifying by time to diet failure
31

33. Is there a combined trend to explain data?
32

34. Time-shifting cohorts shows 2-phase pattern
33

35. Lab research indicates mechanism
Topp B, et al A model of b-cell mass, insulin and glucose
kinetics J Theor Biol 2000; 206:605-19
34