Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NIHR Complex Reviews Support Unit (CRSU) - An Introduction

290 views

Published on

An Introduction to the NIHR Complex Reviews Support Unit (CRSU)
Professor Keith Abrams

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

NIHR Complex Reviews Support Unit (CRSU) - An Introduction

  1. 1. NIHR Complex Reviews Support Unit (CRSU) – An Introduction Keith R Abrams, PhD CStat Department of Health Sciences, University of Leicester, UK on behalf of CRSU Department of Health Disclaimer: The views and opinions expressed herein are those of the authors and do not necessarily reflect those of NIHR, NHS or the Department of Health.
  2. 2. Outline • Background & Objectives of CRSU • First 18 months of CRSU – Working with Cochrane Review Groups – Workshops & Training events – Development Work & Issues • Meta-analysis of DTA data • Network meta-analysis (NMA)
  3. 3. Background – 1 • Why are reviews increasingly complex? – Increasingly complex clinical and important policy questions – More interest in complex interventions – Existing evidence is often limited and heterogeneous – Multiple treatment/intervention options with no (or limited) head-to-head evidence – Outcomes of interest have complex data structure
  4. 4. Background – 2 • However, the lack of methodological expertise and researcher capacity in this area is a recognised major barrier to completing such complex reviews. • In response to this, the National Institute for Health Research Complex Reviews Support Unit (NIHR CRSU) was set up in July 2015 … • to support and encourage successful delivery of complex reviews of importance to the UK National Health Service (NHS) and … • to contribute to building capacity and capability within the research community.
  5. 5. Objectives – 1 • The primary objective of the unit is to build a successful working relationship with NIHR in supporting the UK NHS in delivering clinically and cost-effective services that are evidence- based. • The CRSU focuses on providing timely and appropriate support for the delivery of complex reviews that are funded and/or supported by NIHR.
  6. 6. Objectives – 2 • These include Cochrane reviews, reviews funded by the Systematic Review Programme and other NIHR programmes, and other NHS and NHS supported sources. • The unit will also work closely with NIHR to support scoping and prioritising of future complex reviews.
  7. 7. CRSU members … • University of Glasgow – Olivia Wu (Director) – Neil Hawkins (Deputy Director) – Moira Aitken (Project Manager) – David Stott – Hilary Thomson – Mhairi Mackenzie – Peter Langhorne – Terence Quinn • London School of Hygiene & Tropical Medicine (LSHTM) – Richard Grieve • University of Leicester – Nicola Cooper (Deputy Director) – Alex Sutton – Keith Abrams – Suzanne Freeman – Rhiannon Owen • Advisory/Governance Committee – Ken Stein (Chair) – Steve Palmer – Nicky Welton – David Tovey – Sally Bailey
  8. 8. Expertise Within CRSU, the key areas of expertise include: • Diagnostic Test Accuracy (DTA) reviews • Network Meta-Analysis (NMA) • Individual Participant Data (IPD) meta-analysis • Economic evaluation • Realist synthesis • Narrative synthesis of quantitative & qualitative data • Use of routine data and/or non-randomised studies • Prevalence & Prognostic reviews • Time-to-event/survival outcomes
  9. 9. First 18 months … • Working with Cochrane Review Groups • Workshops & Training events • Development work
  10. 10. Cochrane Reviews – 1 The CRSU has successfully supported the following Cochrane Review Groups so far: • Dementia Group – building on an existing and recent Cochrane review on diagnosing dementia in stroke patients, the CRSU is supporting the development of a proposal to demonstrate the potential added-value of the more complex approaches to diagnostic test evaluation, beyond those currently considered by the Cochrane Collaboration (see later).
  11. 11. Cochrane Reviews – 2 • Gynaecological, neuro-oncological and orphan cancers Group – provided advice on network meta-analysis of treatments for metastatic brain tumours. • Programme of 30 reviews on upper digestive disorder at University College London – provided advice on network meta- analysis for multiple reviews. • Heart Group and Airways Group – provided comments and advice on protocols
  12. 12. Cochrane Reviews – 3 CRSU is currently providing support to Cochrane Review Groups applying for awards from the NIHR Incentive Scheme and Programme Grants, including; • Cochrane Injuries • Cochrane Tobacco Addiction • Cochrane ENT • Cochrane Eyes and Vision • Cochrane Gynaecological, Neuro-oncology & Orphan Cancers (GNOC) • Cochrane Oral Health • Cochrane Airways
  13. 13. Workshops & Training • Workshops at Cochrane UK & Ireland Symposium 2016: – Methodological challenges in complex reviews – NIHR Systematic Reviews Programme: opportunities for greater impact • Workshops at Cochrane UK & Ireland Symposium 2017: – Can Cochrane Reviews take a more active role in informing the design of future trials? – Examples of collaborations with Cochrane Review Groups: assessment of complex interventions, test accuracy and network meta-analyses. • NICE Centre for Guidelines: – 1-day workshop on Introduction to Systematic Reviews & Meta-Analysis of Diagnostic Test Accuracy (DTA) data
  14. 14. Development Work & Issues • DTA Reviews – Raise awareness of the challenges of conducting diagnostic test accuracy (DTA) reviews and offer potential (simple to more complex) solutions to some but not all of the challenges (e.g. multiple thresholds) – Provoke discussion regarding how to ensure reviews of diagnostic tests answer clinically-relevant questions
  15. 15. Sensitivity vs. Specificity pdf Diagnostic variable, D Group 0 (Healthy) Group 1 (Diseased) TP TN Group 1 Diseased Group 0 Healthy Test + TP FP Test - FN TN DT Test +Test - Threshold Sensitivity = number of true positives/total with disease Specificity = number of true negatives/total without disease
  16. 16. Receiver Operating Characteristic (ROC) Curve: Selecting the Threshold T 1 - specificity (False positive rate) Sensitivity(Truepositiverate) 45o line = random guess Perfect classification Lower threshold Higher threshold Group 1 Diseased Group 0 Healthy Test + TP FP Test - FN TN 0 1 0 1 Point T gives Max. accuracy threshold BUT ignores relative opportunity costs of FP and FN results
  17. 17. Challenges of meta-analysing DTA data • More complex than for effectiveness data due to: – Two dependent outcomes – sensitivity and specificity – Variable & multiple test threshold levels (either explicit or implicit) between & within studies – Different reference tests (imperfect gold standard) • Other issues include: – Different populations and/or study conduct (leading to between-study heterogeneity) – Data quality & risk of bias
  18. 18. Naïve analysis – pooling sensitivity & specificity separately
  19. 19. 0 .1.2.3.4.5.6.7.8.9 1 Sensitivity 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Specificity Two dependent outcomes - Sensitivity and Specificity • Requires a meta-analysis model that models sensitivity, specificity and their correlation simultaneously Hierarchical sROC: sROC curve, 95% credible & 95% prediction region 0 .1.2.3.4.5.6.7.8.9 1 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Specificity Bivariate: Point estimate, 95% credible & 95% prediction region for sensitivity and specificity • Statistical models are equivalent although presentation of results are different
  20. 20. Challenges of meta-analysing DTA data • More complex than for effectiveness data due to: – Two dependent outcomes – sensitivity and specificity  – Variable & multiple test threshold levels (either explicit or implicit) between & within studies  – BUT data on test threshold in primary studies (if known) often ignored – Different reference tests (imperfect gold standard)  • Other issues include: – Different populations and/or study conduct (leading to between-study heterogeneity)  – Data quality & risk of bias  – Limited by the data & #studies
  21. 21. Discussion points • Whilst solutions exist, meta-analysis of DTA data is more complex and requires an understanding of the principles underlying the different approaches/methods … – http://methods.cochrane.org/sdt/handbook-dta-reviews – 1-day workshop on Introduction to Systematic Reviews & Meta- Analysis of Diagnostic Test Accuracy (DTA) data • … But also raises other (even more complex) issues for example when there are multiple candidate tests and NHS needs to make a decision as to which tests should be used/funded. • From a NICE perspective, could argue that only makes sense to consider tests & interventions, e.g. many oncology drugs now come with a companion diagnostic (test)
  22. 22. Development Work & Issues • DTA Reviews – Raise awareness of the challenges of conducting diagnostic test accuracy (DTA) reviews and offer potential (simple to more complex) solutions to some but not all of the challenges (e.g. multiple thresholds) – Provoke discussion regarding how to ensure reviews of diagnostic tests answer clinically-relevant questions • Network meta-analysis (NMA)
  23. 23. Early thrombolysis for AMI (Caldwell & Higgins BMJ 2005): NMA considers the network of all relevant evidence to a decision problem NMA methods allow comparison/effect estimates for all interventions & associated uncertainty
  24. 24. Assumptions in NMA • Similarity – Trials are clinically and methodologically similar and comparable • Exchangeability – If patients in one trial were substituted in another, the observed treatment estimates would be expected to be the same (allowing for random variation) • Transitivity – dAB = dAC − dBC & dAC = dAB – dCB • Consistency – Indirect and direct estimates are consistent
  25. 25. Discussion points • Can we add value to existing reviews using network meta-analysis? • Can this be readily incorporated in your current reviews? • Is network meta-analysis within the remit of Cochrane reviews? • Cochrane Methods – Comparing Multiple Interventions – http://methods.cochrane.org/cmi/comparing-multiple- interventions-cochrane-reviews • But methods can be complex & have to be implemented with care, and therefore there is a capacity & training issue.
  26. 26. Development Work & Issues • DTA Reviews – Raise awareness of the challenges of conducting diagnostic test accuracy (DTA) reviews and offer potential (simple to more complex) solutions to some but not all of the challenges (e.g. multiple thresholds) – Provoke discussion regarding how to ensure reviews of diagnostic tests answer clinically-relevant questions • Network meta-analysis (NMA) • Methods for synthesising time-to-event data to inform estimates of comparative effectiveness and decision models • Complex Interventions
  27. 27. Getting in touch & resources … www.nihrcrsu.org Follow us at @NIHRCRSU Google: “NIHR CRSU”
  28. 28. Thank you Any Questions?

×