This document summarizes a study comparing healthcare technology assessment (HTA) recommendations across five countries (England, Scotland, Sweden, Canada, Australia) and three therapy areas (cancer, orphan diseases, central nervous system diseases). The study found low levels of agreement between HTA agencies and differing associations between agencies, recommendations, and therapy areas. Differences in recommendations were attributed to national considerations and differences in HTA processes, such as evidence used, methods, and interpretation. Understanding why recommendations vary can help improve HTA methodologies and determine when differences reflect legitimate national priorities versus inconsistent application of HTA principles.
1. A COMPARATIVE ANALYSIS OF HTA OUTCOMES IN FIVE
COUNTRIES ACROSS THREE THERAPY AREAS
Cancer, orphan and central nervous system diseases
Authors: Elena Nicod, Dr Panos Kanavos
London School of Economics and Political Science
LSE Health and Social Care
HTAI Conference
Bilbao, 26 June 2012
4. Objectives: Examine and compare HTA recommendations in 5 countries, and
across 3 therapy areas (cancer, orphan, central nervous system diseases)
Conceptual framework: It is assumed that HTA outcomes are likely to be similar
across settings, because the same clinical and safety evidence is considered
alongside the same methodology
In practice: HTA recommendations vary greatly across settings1, 2, 3, 4
Implications
Coverage decisions may vary across countries in the same geographical region, with
the same socio-economic level of development, and the same health care systems
Need to understand why these differences across countries occur
Need to approximate HTA processes across settings
Sources: 1. Clement 2010; 2. Morgan 2006; 3. Lexchin 2008; 4. Barbieri 2005.
6. DATA
• Database: all drug-indication pairs issued between January 2007 and December 2009
• Country selection: England, Scotland, Sweden, Canada, Australia
• Stratification per disease area: WHO ICD10 codes
• HTA recommendation classification: List (“L”), List with Conditions (“LWC”), Do not list
(“DNL”)
• Selection of disease areas: cancer, orphan, central nervous system indications
ANALYSIS
• Comparison of HTA recommendations across the study countries
• Measurement of the level of agreement between the study countries: Cohen’s Kappa
score
• Associations between HTA recommendations, HTA bodies, and 3 therapy areas (cancer,
orphan, central nervous system diseases): two-way correspondence analysis
• Case studies: to understand possible reasons for variations
8. Database - summary
= 287 drug-indication pairs collected between 2007-2009 in the 5 study countries
Figure represents the HTA recommendations issued in each country
Scotland
SMC 54 78 61
n=193
Sweden
TLV 79 26 6
n=111
Australia
PBAC 47 111 52
n=210
England
NICE 21 70 19
n=110
Canada
CDR/CED 4 62 64
n=130
0% 20% 40% 60% 80% 100%
Source: the authors
List LWC DNL
10. Level of agreement between agencies in the HTA recommendations
measured by kappa scores
CDR NICE PBAC TLV SMC
CDR - 0.038 0.165 -0.001 0.062
NICE - 0.178 0.228 0.105
PBAC - -0.023 0.132
TLV - 0.066
SMC -
Source: the authors
Interpretation of kappa score indicators
Poor Fair Moderate Good Very good
0 0.4 0.6 0.8 1
0.2
Source: Altman DG. Practical Statistics for Medical Research. (1991) London England: Chapman and Hall.
11. Associations between HTA bodies and their recommendations
CORA for all drugs (N=287) CORA for 3 therapy areas (N=141)
Legend: Recommendations/outcomes: L: list; LWC: restrict; DNL: reject;
HTA bodies: TLV - Sweden; SMC - Scotland; CDR - Canada; PBAC - Australia; NICE - England & Wales
Therapy areas: c – cancer; n- central nervous system; o- orphan indications
Source: the authors
12. Examples of reasons for differences identified in a number of
case studies (e.g. metastatic colorectal cancer, schizophrenia)
Differences are a consequence of:
• National considerations (e.g. national preferences, costs)
• HTA processes
• Evidence (e.g. clinical expertise)
• Methods (e.g. economic model, choice of comparator)
• Interpretation (e.g. what constitutes acceptable levels of evidence, clinical
endpoint)
• Other considerations (e.g. disease severity, symptomatic/curative, existing
treatment alternatives, orphan indication)
14. Inter-country variability in HTA recommendations:
• Low level of agreement between agencies
• Differing associations between:
• HTA bodies and recommendations issued
• HTA bodies and recommendations issued per therapy area
• Differences are a consequence of:
• National considerations (e.g. national preferences)
• HTA processes (e.g. evidence, methods, evidence
interpretation, other considerations)
15. Does it really matter that differences exist across countries?
• Aim of HTA:
• Efficiency in healthcare resource allocation
• Value for money
Therapy X is deemed cost-effective in country A, but not in
country B because of, for example, different levels of evidence
presented (e.g. clinical expertise)
It is individual interpretation of value for money according to
special circumstances
What does all this mean for HTA methodologies?
16. Need to understand why such differences exist: Ways forward
• Identify the reasons for these differences and differentiate
when they are a consequence of:
• National considerations
• HTA processes
• These preliminary findings have demonstrated that it is also
important to differentiate HTA processes per therapy area