When does NICE recommend the use of health technologies within a programme of evidence development? The study aimed to identify where NICE had made "Only in research" (OIR) and "Approval with research" (AWR) recommendations in its guidance, and the key considerations leading to those recommendations. The review found that 31 appraisals included OIR/AWR recommendations in draft guidance, with 29 in final guidance. OIR recommendations were more common due to uncertainties in relative treatment effects, while AWR addressed long-term effects. Both OIR and AWR recommendations have declined in recent years, especially under NICE's Single Technology Appraisal process.

1. When does NICE recommend the use of health
technologies within a programme of evidence
development?
Presenter: Louise Longworth1
Co-authors: J Youn1, L Bojke2, S Palmer2, S Griffin2,
E Spackman2 and K Claxton2
1 Health Economics Research Group (HERG), Brunel University, UK
2 Centre for Health Economics, University of York, UK
Based on a project funded by the UK MRC-NIHR Methodology Research programme
HTAi, Bilbao, Spain. 26 June 2012

2. Context
 Interest in linking reimbursement decisions with requirements for
evidence development is increasing
 In the UK, the National Institute for Health and Clinical Excellence
(NICE) has previously issued guidance using these types of
recommendations
 A range of terms are used to describe these reimbursement
recommendations. Two categories are defined for this study:
1. ‘Only in research’ (OIR): recommends the technology is not used
routinely and which recommends that further research is
conducted as part of the formal guidance.
2. ‘Approval with research’ (AWR): recommends the technology is
used routinely and which recommends further research is
conducted as part of the formal guidance

3. Current NICE Guidance
It [The Committee] should consider whether the intervention is
reasonably likely to benefit patients and the public, how easily the
research can be set up or whether it is already planned or in progress,
how likely the research is to provide further evidence, and whether the
research is good value for money”.
NICE Social value judgements. 2008

4. Aims
To identify:
1. where OIR and AWR recommendations were made or
considered in the development of NICE guidance; and
2. the considerations that led to those recommendations
and to identify any common characteristics in those
appraisals.

5. Methods
 All draft and final guidance produced by the NICE Technology
Appraisals programme up to Jan 2010 were reviewed and those
including OIR/AWR recommendations were identified
 Data were extracted from the guidance documents. These included:
– Appraisal process history and characteristics
– Information on the technology and condition
– Type of recommendation and research requested
– Cost-effectiveness data
– Key considerations leading to the OIR/AWR recommendation
 Thematic content analysis of the guidance documents was
conducted to extract the stated key considerations.

6. Results: appraisal characteristics
 Of 184 appraisals:
– 31 included OIR/AWR in draft guidance: 26 OIR; 5 AWR
– 29 included OIR/AWR in final guidance: 25 OIR; 4 AWR
Single Technology Appraisal
50%
(STA) process introduced
40%
30%
AWR
20% OIR
10%
0%
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

7. Results: cost-effectiveness estimates*
OIR AWR Total
Considered cost-effective 2 3 5
Not considered cost-effective 23 1 24
Total 25 4 29
* Based on conclusions or preferred estimates for the specific OIR/AWR
indication as stated by the Appraisal Committee in the final guidance

8. Key uncertainties leading to OIR and AWR
recommendations
Clinical effectiveness
Relative effectiveness (whole…
Relative effectiveness (OIR/AWR…
Long-term effects
Adverse effects AWR
Cost effectiveness OIR
Cost-effectiveness
Quality of life
Costs
0 5 10 15 20

9. Conclusions
 NICE has issued recommendations linking reimbursement to
evidence development
– usually as ‘Only in research’ but some uses of ‘approval with
research’ were evident
 The use of both OIR and AWR have recently declined
– OIR/AWR are rarely used within the STA process; but this does
not account for all reduction in use
 Differences in the stated rationale for OIR and AWR were observed
– uncertainty in relative treatment effects for OIR
– uncertainty in long-term effects for AWR
 This review has helped to inform the development of a formal
framework for the optimal use of OIR/AWR

10. Further information
 Longworth L, Youn J, Bojke L et al. When does NICE recommend the
use of health technologies within a programme of evidence
development? A systematic review of NICE guidance.
Pharmacoeconomics (forthcoming)
 Claxton K, Palmer S, Longworth L et al. Informing a decision
framework for when NICE should recommend the use of health
technologies only in the context of an appropriately designed
programme of evidence development. Health Technology
Assessment Report (forthcoming)
louise.longworth@brunel.ac.uk

11. Acknowledgement
This research was funded by the MRC-NIHR Methodology
Research programme in the UK as part of a larger project to
establish an improved framework for formulating approval
and research recommendations under uncertainty at NICE
‘Only in Research’ project (G0802659)