CADTH 2015 Symposium

1. 1
CADTH Value Methods Panel
Using Best Worst Scaling to elicit
Values
Carlo Marra

2. Problem
2
?

3. Traditional DCEs
• Discrete Choice Experiments increasingly used
in health services research
• Respondents choose a preferred specification
of the good or service
• Aim is to obtain quantitative estimates of utility
(benefit) associated with different attribute levels
describing the good or service

4. Example of a DCE
4

6. Best-Worst Scaling
• Devised by Finn & Louviere (JPPM 1992)
– introduced to health care by McIntosh & Louviere
(HESG 2002)
– statistical proof paper Marley & Louviere (J Math
Psych 2005)
– ‘user guide’ by Flynn et al (JHE 2006)
• Differs from traditional DCEs in the nature of the
choice task
• Individuals choose the best and the worst
attribute based on the levels displayed in a given
specification

7. Statistical issues
• MNL is (usually) a first step
– Is there heterogeneity?
– Likely covariates that characterise it?
• More complex methods?
– Mixed logit
– Latent class analysis
• Non/semi parametric

8. Problem
8
?

9. Analyzing results
• To get around the dreaded BLACK BOX
• Best-minus-worst-scores
• Easy to understand
• Found to be linearly related to the ML estimates of
the conditional logit model in virtually every
empirical study to date
• Scores can help guide analysis of choice data
– eg. LCAs which may give spurious associations
9

10. Stratifying and Targeting Pediatric
Medulloblastoma
10

11. Purpose
To determine preferences of the general
population, parents and health
professionals regarding trade-offs between
treatment intensity and survival including
test characteristics, functional outcomes,
psychological outcomes and economic
burden.
11

12. 12
Parents:
• N=76 participants
Health professionals:
• N=193 participants
General population:
• N= 3006 participants

13. 13
100% accuracy of test
95% accuracy of test
90% accuracy of test
85% accuracy of test
1. Accuracy of test:
The possible levels of test accuracy in this survey are:

14. 14
2. QoL/ Functional ability
(Side effects of the radiotherapy):
The possible health states in this survey are:
Child will have normal healthy life.
Child will experience mild disability.
Child will experience partial disability.
Child will experience severe disability.

15. 3. Ten year survival rates:
The possible levels of survival in this survey are:
15
Good
prognosis
Intermediat
e prognosis
Poor
prognosis
Baseline
Survival Rate 90% 70% 40%
Levels 100% 85% 55%
95% 70% 40%
90% 55% 25%
80% 40% 10%

16. Best and Worst Survey Design
Clinicians’ Survey
16

17. Best-Worst estimated parameters (paired model) for
general public
good prognosis intermediate prognosis poor prognosis
Attributes Estimate Prob Attributes Estimate Prob Attributes Estimate Prob
Accuracyofthe
test
100% 2.15 <.0001 100% 3.81 <.0001 100% 4.32 <.0001
95% 1.08 <.0001 95% 3.12 <.0001 95% 3.57 <.0001
90% 0.52 <.0001 90% 2.45 <.0001 90% 2.89 <.0001
85% -0.26 <.0001 85% 1.89 <.0001 85% 2.37 <.0001
Qualityoflife
Normal life 2.97 <.0001 Normal life 4.35 <.0001 Normal life 4.92 <.0001
Mild disability -0.98 <.0001 Mild disability 1.07 <.0001 Mild disability 1.78 <.0001
Partial disability -1.58 <.0001 Partial disability 0.59 <.0001 Partial disability 1.29 <.0001
Severe disability -3.21 <.0001 Severe disability -1.28 <.0001 Severe disability -0.53 <.0001
Survivalrate
100% 3.28 <.0001 85% 3.01 <.0001 55% 2.09 <.0001
95% 2.29 <.0001 70% 2.10 <.0001 40% 1.40 <.0001
90% 1.37 <.0001 55% 0.56 <.0001 25% 0.55 <.0001
80%
0.00
40%
0.00
10%
0.00

18. Good prognosis
Number of respondent 901
Attribute
Times
Shown
Times
Selected
Best
Best Count
Proportion
Times
Selected
Worst
Worst
Count
Proportion
Best -
Worst
score
Accuracyof
thetest
100% 3612 1307 36.2% 128 3.5% 1179
95% 3588 615 17.1% 214 6.0% 401
90% 3597 283 7.9% 330 9.2% -47
85% 3604 163 4.5% 702 19.5% -539
Qualityoflife
Normal life 3642 2094 57.5% 155 4.3% 1939
Mild disability 3597 214 5.9% 1903 52.9% -1689
Partial disability 3605 141 3.9% 2822 78.3% -2681
Severe disability 3604 88 2.4% 3162 87.7% -3074
Survivalrate
100% 3596 2591 72.1% 117 3.3% 2474
95% 3596 1954 54.3% 145 4.0% 1809
90% 3595 913 25.4% 359 10.0% 554
80% 3612 449 12.4% 775 21.5% -326
Baseline survival rate is 90%.

19. Best-Worst estimated parameters (paired model) for parents
and clinicians – intermediate prognosis
Attributes
Parents Clinicians
Estimate Prob Estimate Prob
Accuracyofthe
test
100% 4.79 <.0001 3.49 <.0001
95% 4.25 <.0001 2.94 <.0001
90% 3.35 <.0001 2.14 <.0001
85% 3.11 <.0001 1.54 <.0001
Qualityoflife
Normal life 5.49 <.0001 3.67 <.0001
Mild disability 3.08 <.0001 1.38 <.0001
Partial disability 1.53 <.0001 0.90 <.0001
Severe disability -0.46 0.035 -0.68 <.0001
Survivalrate
85% 4.44 <.0001 3.44 <.0001
70% 2.89 <.0001 2.19 <.0001
55% 0.76 0.003 0.61 <.0001
40%
0.00 0.00
• Normal life, 85% survival rate and
100% accuracy of the test are more
favorable attributes for parents and
clinicians.
• Severe disability is the only attribute
that is less favorable than 40%
survival rate.
• Comparing coefficients of mild
disability for intermediate prognosis
and good prognosis shows that
parents and clinicians prefer mild
disability over low of survival rate.
For parents mild disability is more
favorable than 70% survival rate.

20. Summary of results for clinicians in different prognosis
20
good prognosis intermediate prognosis poor prognosis
Attributes Times shown
Times
Selected
Best
Times
Selected
Worst
Attributes Times shown
Times
Selected
Best
Times
Selected
Worst
Attributes Times shown
Times
Selected
Best
Times
Selected
Worst
Accuracyof
thetest
100% 928 285 46 100% 913 475 41 100% 901 537 51
95% 930 145 60 95% 909 382 41 95% 901 466 53
90% 931 62 164 90% 908 157 105 90% 900 220 83
85% 926 39 276 85% 914 81 163 85% 898 125 127
Qualityoflife
Normal life 926 356 42 Normal life 914 508 42 Normal life 903 550 41
Mild disability 928 37 430 Mild disability 912 104 216 Mild disability 897 132 158
Partial disability 927 37 635 Partial disability 911 51 380 Partial disability 901 72 238
Severe disability 927 59 746 Severe disability 914 65 645 Severe disability 899 64 510
Survivalrate
100% 929 736 28 85% 912 521 51 55% 901 275 130
95% 930 631 35 70% 910 277 140 40% 902 134 246
90% 928 288 93 55% 913 73 420 25% 899 53 508
80%
926 109 229
40%
914 42 492
10%
898 72 555
Best-Worst count score is equal to difference of times selected best and worst
divided by times shown.

21. Best-Worst count score for clinicians’ preferences
• Quality of life has the most impact
on clinicians’ decision making for
good prognosis. Severe, partial and
mild disability are least favorable
attributes, respectively.
Good
prognosis
attributes
Best - Worst
score
Intermediate
prognosis
attributes
Best - Worst
score
Poor
prognosis
attributes
Best - Worst
score
Accuracyofthetest
100% 25.8% 100% 47.5% 100% 53.9%
95% 9.1% 95% 37.5% 95% 45.8%
90% -11.0% 90% 5.7% 90% 15.2%
85% -25.6% 85% -9.0% 85% -0.2%
Qualityoflife
Normal life 33.9% Normal life 51.0% Normal life 56.4%
Mild disability -42.3%
Mild
disability
-12.3%
Mild
disability
-2.9%
Partial
disability
-64.5%
Partial
disability
-36.1%
Partial
disability
-18.4%
Severe
disability
-74.1%
Severe
disability
-63.5%
Severe
disability
-49.6%
Survivalrate
100% 76.2% 85% 51.5% 55% 16.1%
95% 64.1% 70% 15.1% 40% -12.4%
90% 21.0% 55% -38.0% 25% -50.6%
80%
-13.0%
40%
-49.2%
10%
-53.8%

22. Summary
• Strengths
– Easy
– Simple to calculate - no
black box
– Can be done online
– Use of scores might give
average applied
researcher more
confidence in results
• Weaknesses
– Does not meet
economists definition of
a trade-off
– Cannot on its own
produce QALYs