CHE economic evaluation seminar presented by Matthew Taylor, York Health Economics Consortium, University of York, 15th February 2018

1. Providing Consultancy &
Research in Health Economics
The implications of parameter
independence in probabilistic
sensitivity analysis:
An empirical test
Matthew Taylor, YHEC

2.  Background
 The use of PSA in healthcare decision making
 Some preliminary tests
 Using an empirical model and various approaches to
correlation
 Some preliminary recommendations
 What can we tell researchers and decision makers?
Overview

3.  ICERs (or NMB, NHB etc.) tell us about benefits
and opportunity costs
 Decision makers are concerned with uncertainty
 One-way sensitivity analysis is of limited use
(likely to be many interactions that are not
immediately obvious)
 Hence, multi-way sensitivity analysis or PSA
 Used for value of information analysis to guide future
research
PSA in healthcare

4. Adalsteinsson E & Toumi M J Market Access & Health Policy 2013
Overview

5. Adalsteinsson E & Toumi M J Market Access & Health Policy 2013
Overview

6. Scatter plots and CEACs

7.  Base case ICER
 Price of drug, effectiveness of drug, other
 Size of scatter plot
 e.g. confidence intervals from evidence base
 Shape of scatter plot
 Interaction between parameters
Drivers of CEAC

8. Scatter Plot A

9. Scatter Plot B

10.  More often than not, parameters are varied
independently
 Whilst it is possible to apply covariance between
parameters, this is often ignored due to:
– Arbitrary decision
– Lack of understanding of ‘real’ interactions and
covariance
– Lack of available data to generate covariance
matrices
In HTA submissions

11.  “Only 1 of the 18 reviewed TAs considered the
incorporation of correlation and dependencies
between parameters”
Lanitis T, Muszbek N & Tichy E 2014
 “The assumption of no correlation can lead to
misleading probabilistic results and the
overestimation of uncertainty”
ibid
Arbitrary (?) decision

12.  “Only 1 of the 18 reviewed Tas considered the
incorporation of correlation and dependencies
between parameters”
Lanitis T, Muszbek N & Tichy E 2014
 “The assumption of no correlation can lead to
misleading probabilistic results and the
overestimation of uncertainty”
ibid
Arbitrary (?) decision

13.  Quite typical to see:
 Choleski decomposition for survival functions (e.g.
alpha and beta parameters)
 Simple correlations between progression-free survival
and overall survival
 Dirichlet for multiple outcomes
 More (very) rare to see correlations between:
 Utility and cost
 Effectiveness and safety
 Dose intensity and effectiveness (…and safety)
Lack of understanding

14.  Even if we had:
 One single data source
 …containing all relevant inputs to a model
 …over a long enough time period
 …and used Bayesian Markov chain Monte Carlo
(MCDC) methods or other techniques
 …the sample size would not be sufficient to
capture all permutations of correlation (models
may contain >100 input parameters)
Lack of data

15.  To build a simple model to explore the impact of
different types of parameter interactivity
 To develop some simple rules of thumb for
reviewers and decision-makers to consider
when interpreting PSA
 To understand the direction of biases
Aims

16. Correlation and offsetting

17.  2 parameters, A and B
 A: Mean = 5, SE = 3 (assume normal distribution)
 B: Mean = 7, SE = 4 (assume normal distribution)
 Outcome = A + B
Illustration

18. 0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
=
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
M: 12.00
SD: (5.04)
Assuming Independent

19. 0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
= M: 12.00
SD: (7.31)
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
+’ve correlation

20. 0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
=
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
M: 12.00
SD: (5.04)
No correlation

21. 0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
= M: 12.00
SD: (7.31)
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
+’ve correlation

22. 0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
= M: 12.00
SD: (0.98)
0
50
100
150
200
250
300
350
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
-’ve correlation

23. Providing Consultancy &
Research in Health Economics
How does this apply to
CE models?
 Base case ICER
 Price of drug, effectiveness of drug, other
 Size of scatter plot
 e.g. confidence intervals from evidence base
 Shape of scatter plot
 Interaction between parameters

24. Providing Consultancy &
Research in Health Economics
How does this apply to
CE models?
 Base case ICER
 Price of drug, effectiveness of drug, other
 Size of scatter plot
 e.g. confidence intervals from evidence base
 Shape of scatter plot
 Interaction between parameters

25. Providing Consultancy &
Research in Health Economics
Example model
• Built an eight-state Markov model
• Applied costs and utilities to each state
• Then:
(i) Varied all parameters independently (i.e. no
correlation)
(ii) Varied costs together, utilities together and
probabilities together (“single multipliers”)
(iii)Varied all parameters together (i.e. 100%
correlation)

26. Providing Consultancy &
Research in Health Economics
Example

27. Providing Consultancy &
Research in Health Economics
Example
Independent sampling
in 144 parameters
leads to a cancelling
out effect
More pronounced for
effectiveness

28. Providing Consultancy &
Research in Health Economics
Example

29. Providing Consultancy &
Research in Health Economics
Example (mean outcomes)
Mean
Tx
cost
Cx
cost
Tx
QALY
Cx
QALY
Inc
costs
Inc
QALYs
ICER NMB CE
Deterministic £28,135 £17,167 2.47 1.62 £10,968 0.85 £12,875 £6,069 n/a
100% indep. £28,277 £17,299 2.49 1.63 £10,978 0.86 £12,819 £6,150 99.40%
Partially corr. £28,045 £17,191 2.46 1.60 £10,854 0.86 £12,592 £6,386 82.70%
100% corr. £28,069 £17,476 2.47 1.60 £10,592 0.87 £12,121 £6,885 72.50%

30. Providing Consultancy &
Research in Health Economics
Example (SDs)
Mean
Tx
cost
Cx
cost
Tx
QALY
Cx
QALY
Inc
costs
Inc
QALYs
ICER NMB CE
Deterministic £28,135 £17,167 2.47 1.62 £10,968 0.85 £12,875 £6,069 n/a
100% indep. £28,277 £17,299 2.49 1.63 £10,978 0.86 £12,819 £6,150 99.40%
Partially corr. £28,045 £17,191 2.46 1.60 £10,854 0.86 £12,592 £6,386 82.70%
100% corr. £28,069 £17,476 2.47 1.60 £10,592 0.87 £12,121 £6,885 72.50%
Standard deviation
Tx
cost
Cx
cost
Tx
QALY
Cx
QALY
Inc
costs
Inc
QALYs
ICER NMB CE
Deterministic n/a n/a n/a n/a n/a n/a n/a n/a n/a
100% indep. £1,856 £1,963 0.11 0.07 £1,964 0.09 n/a £2,316 n/a
Partially corr. £1,892 £2,238 0.41 0.23 £1,454 0.41 n/a £7,275 n/a
100% corr. £1,204 £3,072 0.52 0.07 £1,914 0.59 n/a £9,939 n/a

31.  Confidence intervals
 Number of health states
 Granularity of inputs
 Base case ICER/NMB (proximity to λ)
 Disease pathways
Other exploration

32.  Baseline (SE ~ 10% of mean)
 Smaller standard error (~5%)
 Larger standard error (~15%)
Degree of uncertainty

33. Degree of uncertainty

34. Degree of uncertainty

35. Degree of uncertainty

36. Degree of uncertainty
Small SE

37. Degree of uncertainty
Medium SE

38. Degree of uncertainty
Large SE

39. Degree of uncertainty
(How much does it matter?)
Small SE

40. Degree of uncertainty
(How much does it matter?)
Medium SE

41. Degree of uncertainty
(How much does it matter?)
Large SE

42. Degree of uncertainty
Range Small SE Baseline Large SE
100% independent 84.70% 69.40% 63.10%
Partially correlated 59.80% 54.50% 49.00%
100% correlated 61.70% 53.80% 50.80%
0%
20%
40%
60%
80%
100%
Small SE Baseline Large SE
100% independent Partially correlated 100% correlated

43. Degree of uncertainty
Range Small SE Baseline Large SE
100% independent 84.70% 69.40% 63.10%
Partially correlated 59.80% 54.50% 49.00%
100% correlated 61.70% 53.80% 50.80%
Range 24.90% 15.60% 14.10%
0%
10%
20%
30%
40%
50%
Small SE Baseline Large SE
How much does
it matter?

44. Degree of uncertainty
Small SE
Different λ

45. Degree of uncertainty
Medium SE
Different λ

46. Degree of uncertainty
Large SE
Different λ

47. Degree of uncertainty
Small SE Baseline Large SE
100% independent 84.70% 69.40% 63.10%
Partially correlated 59.80% 54.50% 49.00%
100% correlated 61.70% 53.80% 50.80%
Range 24.90% 15.60% 14.10%
0%
10%
20%
30%
40%
50%
Small SE Baseline Large SE
Small SE Baseline Large SE
100% independent 100.00% 99.66% 95.10%
Partially correlated 95.80% 81.45% 69.30%
100% correlated 88.90% 73.74% 66.80%
Range 11.10% 25.92% 28.30%
0%
10%
20%
30%
40%
50%
Small SE Baseline Large SE
λ = £20,000
(ICER close to λ)
λ = £30,000
(ICER well below λ)

48. Degree of uncertainty
λ = £20,000
(ICER close to λ)
λ = £30,000
(ICER well below λ)
Small SE Baseline Large SE
100% independent 84.70% 69.40% 63.10%
Partially correlated 59.80% 54.50% 49.00%
100% correlated 61.70% 53.80% 50.80%
0%
20%
40%
60%
80%
100%
Small SE Baseline Large SE
100% independent Partially correlated 100% correlated
Small SE Baseline Large SE
100% independent 100.00% 99.66% 95.10%
Partially correlated 95.80% 81.45% 69.30%
100% correlated 88.90% 73.74% 66.80%
0%
20%
40%
60%
80%
100%
Small SE Baseline Large SE
100% independent Partially correlated 100% correlated

49. Number of health states
 Base case = 8 states
 Tested: 3 to 8

50. Number of health states
8 states

51. Number of health states
7 states

52. Number of health states
6 states

53. Number of health states
5 states

54. Number of health states
4 states

55. Number of health states
3 states

56. Number of health states
Health states in model
3 4 5 6 7 8
100% independent 57.80% 62.41% 63.56% 66.68% 67.13% 69.85%
Partially correlated 58.96% 54.45% 54.17% 52.52% 52.85% 53.31%
100% correlated 63.14% 57.99% 54.43% 54.59% 53.72% 53.85%
0%
20%
40%
60%
80%
3 4 5 6 7 8
100% independent Partially correlated 100% correlated
%CE
Cancelling
out effect

57. Number of health states
Health states in model
Range
3 4 5 6 7 8
100% independent 57.80% 62.41% 63.56% 66.68% 67.13% 69.85%
Partially correlated 58.96% 54.45% 54.17% 52.52% 52.85% 53.31%
100% correlated 63.14% 57.99% 54.43% 54.59% 53.72% 53.85%
Range 5.34% 7.96% 9.39% 14.16% 14.28% 16.54%
0%
5%
10%
15%
20%
3 4 5 6 7 8

58. Granularity of costs
 Baseline = 1 single cost for each health
state
 Scenarios: 3, 5 and 10 components
 100% independent
 Partially correlated
 100% correlation

59. Granularity of costs
Sub-costs
%CE 1 3 5 10
100% independent 69.85% 72.13% 76.21% 83.48%
Partially correlated 53.31% 55.20% 56.12% 56.97%
100% correlated 53.85% 53.80% 53.91% 53.33%
0%
20%
40%
60%
80%
100%
1 3 5 10
100% independent Partially correlated 100% correlated
Cancelling
out effect

60. Granularity of costs
Sub-costs
Range 1 3 5 10
100% independent 69.85% 72.13% 76.21% 83.48%
Partially correlated 53.31% 55.20% 56.12% 56.97%
100% correlated 53.85% 53.80% 53.91% 53.33%
Range 16.54% 18.33% 22.30% 30.15%
0%
10%
20%
30%
40%
1 3 5 10

61. Proximity to threshold

62. Proximity to threshold

63. Proximity to threshold

64. Proximity to threshold

65. Proximity to threshold

66. Proximity to threshold

67. Proximity to threshold

68. Proximity to threshold

69. Proximity to threshold

70. Proximity to threshold
Cost of drug
%CE
£0 £5k £10k £15k £20k £25k £30k £35k
100% independent 100.00% 100.00% 99.66% 69.35% 4.40% 0.10% 0.00% 0.00%
Partially correlated 99.73% 96.64% 81.45% 52.31% 26.50% 10.30% 3.60% 1.70%
100% correlated 98.87% 90.00% 73.74% 53.68% 34.20% 21.20% 10.80% 6.20%
0%
20%
40%
60%
80%
100%
£0 £5k £10k £15k £20k £25k £30k £35k
100% independent Partially correlated 100% correlated
Always
CE Scatter
matters
50-50%
Scatter
matters
Never
CE

71. Proximity to threshold
£0 £5k £10k £15k £20k £25k £30k £35k
100% independent 100.00% 100.00% 99.66% 69.35% 4.40% 0.10% 0.00% 0.00%
Partially correlated 99.73% 96.64% 81.45% 52.31% 26.50% 10.30% 3.60% 1.70%
100% correlated 98.87% 90.00% 73.74% 53.68% 34.20% 21.20% 10.80% 6.20%
Range 1.13% 10.00% 25.92% 17.04% 29.80% 21.10% 10.80% 6.20%
NMB £16,069 £11,069 £6,069 £1,069 -£3,931 -£8,931 -£13,931 -£18,931
0%
10%
20%
30%
40%
50%
£0 £5k £10k £15k £20k £25k £30k £35k
Cost of drug
Range

72. £0 £5k £10k £15k £20k £25k £30k £35k
100% independent 100.00% 100.00% 99.66% 69.35% 4.40% 0.10% 0.00% 0.00%
Partially correlated 99.73% 96.64% 81.45% 52.31% 26.50% 10.30% 3.60% 1.70%
100% correlated 98.87% 90.00% 73.74% 53.68% 34.20% 21.20% 10.80% 6.20%
Range 1.13% 10.00% 25.92% 17.04% 29.80% 21.10% 10.80% 6.20%
NMB £16,069 £11,069 £6,069 £1,069 -£3,931 -£8,931 -£13,931 -£18,931
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
-£25,000 -£20,000 -£15,000 -£10,000 -£5,000 £0 £5,000 £10,000 £15,000 £20,000
Proximity to threshold
NMB
Range

73.  We already knew that parameter correlation
is important
 There is a perception that ignoring correlation
increases the uncertainty
 This can be misunderstood by decision-
makers
 Very often, ignoring correlation underestimates
the uncertainty
Summary

74.  We can assign covariance or other interaction
terms (e.g. using the Cholesky decomposition
method and other techniques)
 However, very unlikely that we will have suitable
data to quantify this adequately
 As a minimum, we should aim to understand the
likely direction of the consequences
What can we do?

75. What can we do?
Characteristic Consequence of ignoring correlation
More health states Likely to overstate confidence
Greater granularity of inputs Likely to overstate confidence
ICER proximity to λ
Likely to be a problem when fairly close
(but not very close) to threshold
other… etc…
other... etc…
But…

76. What can we do?

77. What can we do?

78. What can we do?
Too convoluted?

79. http://tinyurl.com/yhec-facebook
http://twitter.com/YHEC1
http://www.minerva-network.com/
http://tinyurl.com/YHEC-LinkedIn
Providing Consultancy &
Research in Health Economics
Thank you
matthew.taylor@york.ac.uk
Telephone: +44 1904 323631
Website: www.yhec.co.uk