Assessing the International Useof Health Technology AssessmentsDr David Wright, Prof Ruairidh Milne, Alison Price,Nicola Tose, Dr Nick HicksNIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC)www.netscc.ac.uk 25/06/2012
Why measure HTA use? • HTA commissioning organisations need to demonstrate uptake of findings to: • justify existing levels of spend • meet requirements for accountability • inform and improve funding and commissioning processes • Most studies explore use and impact within the country in which the research was conducted • We know little about the use of research internationally
Study Question and Design Study aim: •To explore the merits of different methods for assessing the international use of UK funded research by the NIHR HTA programme Study design •Phase I: Updated Literature Review •Phase II: Exploring Methods • Bibliometrics • Webtrends • HTA CRD database search
Phase I: Literature review • Hanney’s systematic review examined the question: ‘What models are available to assess the impact of health research programmes, and what are their strengths and weaknesses? • The current study used the same search terms, inclusion / exclusion criteria for the period 2005 – September 20101 Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology AssessmentProgramme. Health Technol Assess. 2007; 11(53): 1 – 200.
Phase I: Literature review • Most models move beyond assessing research outputs (i.e. publications) to assessing research outcomes (i.e. changes in behaviour or practice) • The Payback framework is the most widely used approach • The dominance of the Payback framework supports findings in the Hanney et al. review (2007)1 Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology AssessmentProgramme. Health Technol Assess. 2007; 11(53): 1 – 200.
The PayBack model:(Hanney et al. 2007) Broader Knowledge economic benefits (e.g. published (e.g. healthy output) Workforce) Political and Benefits to future administrative research & research benefits (e.g. use policy (e.g. research Health sector decisions) capacity) benefits (e.g. health service delivery)Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology AssessmentProgramme. Health Technol Assess. 2007; 11(53): 1 – 200.
Phase II: Exploring methods Knowledge (Academic Use) Methods: Bibliometric analysis of no. publications, impact factor, citations and international citations HTA Use Internet Use Methods: Methods: Analysis of citations Webtrend analysis of in international HTA no. UK and reports identified non-UK visits through the CRD database.
Phase II: Bibliometric results • Bibliometric analysis was undertaken on the top 10 most cited HTA reports • The average number of published journal papers for the top 10 HTA reports was 2 publications • The average journal impact factor for the publications was 5.22 • 41% of the 549 journals citing NIHR HTA reports had their editorial base in the United States. 36% were based in the UK.
Phase II: Webtrends resultsAuthor Title Research No. UK Non-UK Type downloads visits (%) visits (%)Avenell Systematic review of the long- Systematic 354,166 27.76 72.24et al. term effects and economic Review /(2004) consequences of treatments for Primary obesity ResearchMurphy Qualitative research methods in Methodology 254,750 47.95 52.05et al. health technology assessment: a(1998) review of the literatureWald First and second trimester Primary 223,921 43.83 56.17et al. antenatal screening for Downs Research(2003) syndromeChen A systematic review of the Systematic 182,299 17.40 82.60et al. effectiveness of adalimumab, Review(2006) etanercept and infliximab for the treatment of rheumatoid arthritis in adultsDinnes A systematic review of rapid Systematic 177,047 12.23 87.77et al. diagnostic tests for the detection Review(2007) of tuberculosis infection International visits of top 5 downloaded HTA reports, Jan 1st 2004 – 30th Jun 2010
Proportion of website visits to Dinnes et al (2007) -Review of rapid diagnostic tests for tuberculosis bycountry, Jan 2004 – Jun 2010
Proportion of website visits to Avenell et al (2004)-Review of effects and consequences of obesitytreatments by country, Jan 2004 – Jun 2010
Phase II: CRD HTA database• Five highly cited NIHR HTA reports were ‘tracked’ through to identify pick up by international HTA agencies• Four HTA reports were collectively cited 28 times, 18 of which were by non-UK HTA agencies.• Canada cited HTA Programme reports the most with 5 citations.• Avenell et al. (2004) was the most internationally cited report with 10 non-UK HTA agencies citations.
To conclude…• Methods used for research impact assessment are useful in generating data on international uptake of HTA findings.• Webtrends revealed a high proportion of international visits, particularly for systematic reviews, although less so for methodology reports• HTA report citations provided evidence of the transfer of HTA findings internationally.
Recommendations…• A multi-dimensional model of HTA uptake is recommended• Bibliometrics can identify international academic uptake, but more developed analysis of international citation is recommended• Webtrends and HTA report citation can be used to provide indicators of international uptake• Further research is recommended using a case-study approach to explore the nature of HTA use.
Any questions…?Dr David WrightSenior Research Fellowt: +44 (0) 23 8059 7484f: +44 (0) 23 8080 5639e: D.Wright@southampton.ac.ukNIHR Evaluation, Trials and Studies Coordinating Centre(NETSCC) part of the National Institute of Health Research (NIHR)Alpha House, University of Southampton Science Park,Southampton SO16 7NSw: www.netscc.ac.ukAcknowledgement:This research was funded by the National Institute for Health Research (NIHR) and carried out by theEvaluation, Trials and Studies Coordinating Centre at the University of Southampton. The views expressedare those of the research team and not necessarily those of the NHS, the NIHR or the Department of Health.