Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spotlight Webinar: GRADE

123 views

Published on

GRADE: appraising the quality of evidence and strength of recommendations

The GRADE framework (Grading of Recommendations Assessment, Development and Evaluation) applies a rating of quality (i.e. confidence, certainty) of evidence and a grading of strength of recommendations for systematic reviews and clinical practice guidelines. The GRADE system classifies the quality of evidence and gives an overall rating of very low quality of evidence, low quality of evidence, moderate quality of evidence or high quality of evidence. The quality of evidence rating depends on a summary of many different factors.

How can GRADE help you?

The GRADE approach is useful when answering questions about interventions and when evidence-informed decision making is needed and recommendations are being produced. Originally developed for clinical interventions, the GRADE approach is designed to assess the quality of evidence for both randomized controlled trials and observational studies. A standard appraisal tool can be used to determine the risk of bias present in individual studies gathered from a systematic review, however GRADE addresses the quality of a body of evidence rather than individual studies.

Click here to access the GRADE tool: http://www.nccmt.ca/knowledge-repositories/search/304

The National Collaborating Centre for Methods and Tools is funded by the Public Health Agency of Canada and affiliated with McMaster University. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada.

NCCMT is one of six National Collaborating Centres (NCCs) for Public Health. The Centres promote and improve the use of scientific research and other knowledge to strengthen public health practices and policies in Canada.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Spotlight Webinar: GRADE

  1. 1. Follow us @nccmt Suivez-nous @ccnmo GRADE (Grading of Recommendations Assessment, Development and Evaluation) Presenters: Gordon Guyatt, MD, MSc, FRCP, OC Sarah McDowell, PhD March 22, 2018 1:00 - 2:30 PM EST
  2. 2. Follow us @nccmt Suivez-nous @ccnmo 2 Housekeeping Use Chat to post comments and/or questions during the webinar • ‘Send’ questions to All (not privately to ‘Host’) Connection issues • Recommend using a wired Internet connection (vs. wireless), • WebEx 24/7 help line • 1-866-229-3239 Participant Side Panel in WebEx Chat
  3. 3. Follow us @nccmt Suivez-nous @ccnmo 3 After Today The PowerPoint presentation (in English and French) and English audio recording will be made available. These resources will be available at: http://www.nccmt.ca/previous-webinars
  4. 4. Follow us @nccmt Suivez-nous @ccnmo 4 How many people are watching today’s session with you? Poll Question #1 A. Just me B. 1-3 C. 4-5 D. 6-10 E. >10
  5. 5. Follow us @nccmt Suivez-nous @ccnmo 5 GRADE: Appraising the quality of evidence and strength of recommendations http://www.nccmt.ca/knowledge-repositories/search/304
  6. 6. NCC Infectious Diseases Winnipeg, MB NCC Methods and Tools Hamilton, ON NCC Healthy Public Policy Montreal, QC NCC Determinants of Health Antigonish, NS NCC Aboriginal Health Prince George, BC NCC Environmental Health Vancouver, BC 6
  7. 7. Registry of Methods and Tools Online Learning Opportunities WorkshopsMultimedia Public Health+ Networking and Outreach NCCMT Products and Services 7
  8. 8. Follow us @nccmt Suivez-nous @ccnmo 8 Poll Question #2 How familiar are you with the method or tool we are discussing today? A. I am not familiar with the method or tool B. I have heard of the method or tool C. I have used the method or tool
  9. 9. Follow us @nccmt Suivez-nous @ccnmo 9 Presenters Gordon Guyatt, MD, MSc, FRCP, OC Distinguished Professor, Department of Health Research Methods, Evidence, and Impact, McMaster University Sarah McDowell, PhD Manager, Clinical Reviews, Health Quality Ontario
  10. 10. Gordon Guyatt McMaster University
  11. 11.  GRADE background  Two steps  confidence in estimates  strength of recommendation  Evidence profiles
  12. 12.  Group started to meet in 2000  First paper describing method BMJ 2004  Guidance for clinicians to understand GRADE  six part series in BMJ 20008  For systematic review, guideline developers  17 part series, 2011 to 2017, J Clinical Epidemiology  2 chapters in Cochrane Handbook
  13. 13. >100 organizations have adopted GRADE
  14. 14. No confidence Totally confidentHigh Moderate Low Two components Strength of recommendation: Strong and weak Very low
  15. 15.  RCTs start high  Observational studies start low  5 limitations can lower quality  Risk of bias  Inconsistency  Indirectness  Imprecision  Publication bias
  16. 16.  Well established  concealment  intention to treat principle observed  blinding  completeness of follow-up  More recent  selective outcome reporting bias  early stopping for benefit
  17. 17.  Small sample size  small number of events  Wide confidence intervals  uncertainty about magnitude of effect  How do you decide what is too wide?  Primary criterion:  would decisions differ at ends of CI
  18. 18. 01.0%
  19. 19.  Variation in size of effect  Overlap in confidence intervals  statistical significance of heterogeneity  I2
  20. 20. No worries 0% Only a little concerned Getting concerned Very concerned 100% Why are we pooling?
  21. 21. Learning Programs to Accelerate the BioPharma Transition Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures Chapuy et al, (2002) 0.85 (0.64, 1.13) Pooled Random Effect Model 0.82 (0.69 to 0.98) p= 0.05 for heterogeneity, I2=53% Chapuy et al, (1994) 0.79 (0.69, 0.92) Lips et al, (1996) 1.10 (0.87, 1.39) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000) 0.48 (0.13, 1.78) Meyer et al, (2002) 0.92 (0.68, 1.24) Trivedi et al, (2003) 0.67 (0.46, 0.99) Favors Vitamin D Favors Control Relative Risk 95% CI ' ' ' ' ' ' ' ' 0.1 1
  22. 22. Explanations of heterogeneity: Where to look?  Patients  Interventions  Outcomes  Methodology
  23. 23. Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose >400)
  24. 24. Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose = 400)
  25. 25.  Populations  older, sicker or more co-morbidity  Interventions  warfarin in trials vs clinical practice  Outcomes  important versus surrogate outcomes  glucose control versus CV events
  26. 26. Alendronate Risedronate Placebo Directness interested in A versus B available data A vs C, B vs C
  27. 27.  High likelihood could lower quality  When to suspect ▪ number of small studies ▪ industry sponsored
  28. 28.  Large magnitude can rate up one level  very large two levels  Common criteria  everyone used to do badly  almost everyone does well  quick action  Hip replacement for hip osteoarthritis
  29. 29. Certainty assessment criteria Study Design Confidence in estimates Lower if Higher if Randomized trials High Risk of bias -1 Serious -2Very serious Inconsistency -1 Serious -2Very serious Indirectness -1 Serious -2Very serious Imprecision -1 Serious -2Very serious Publication bias -1 Likely -2Very likely Large Effect + 1 Large + 1Very large Dose response +1 Evidence of a gradient All plausible confounding +1 Would reduce a demonstrated effect or +1 would suggest a spurious effect when results show no effect Moderate Observational studies Low Very Low
  30. 30.  Most systems just use evidence about primary benefit outcome  But what about others (harms)?  What to do?  Options  ignore all but primary  weakest of any outcome  some blended approach  weakest of critical outcomes
  31. 31. Quality Assessment Summary of Findings Quality Relative Effect (95% CI) Absolute risk difference Outcome Number of participants (studies) Risk of Bias Consistency Directness Precision Publication Bias Myocardial infarction 10,125 (9) No serious limitations No serious imitations No serious limitations No serious limitations Not detected High 0.71 (0.57 to 0.86) 1.5% fewer (0.7% fewer to 2.1% fewer) Mortality 10,205 (7) No serious limitations No serious limiations No serious limitations Imprecise Not detected Moderate 1.23 (0.98 – 1.55) 0.5% more (0.1% fewer to 1.3% more) Stroke 10,889 (5) No serious limitaions No serious limitations No serious limitations Serious limitations Not detected Moderate 1.67 (1.00 – 2.80) 0.3% more (0 more to 1.5% more) Beta blockers in non-cardiac surgery
  32. 32.  Strong recommendation  benefits clearly outweigh risks/hassle/cost  risk/hassle/cost clearly outweighs benefit  What can downgrade strength?  Low confidence in estimates  Close balance between up and downsides
  33. 33.  Aspirin after myocardial infarction  25% reduction in relative risk  side effects minimal, cost minimal  benefit obviously much greater than risk/cost  Anticoagulants versus aspirin in low risk atrial fibrillation  anticoagulants reduces stroke vs ASA by 50%  but if risk only 1% per year, ARR 0.5%  increased bleeds by 1% per year
  34. 34. Aspirin after MI – do it Warfarin rather than ASA in Afib -- probably do it -- probably don’t do it
  35. 35.  Variability in patient preference  strong, almost all same choice (> 90%)  weak, choice varies appreciably  Interaction with patient  strong, just inform patient  weak, ensure choice reflects values  Use of decision aid  strong, don’t bother; weak, use the aid  Quality of care criterion  strong, consider; weak, don’t consider
  36. 36.  Clinicians, policy makers need summaries  quality of evidence  strength of recommendations  Explicit rules  transparent, informative  GRADE  simple, transparent, systematic  increasing wide adoption
  37. 37. GRADE: A User Perspective NCCMT Spotlight Webinar 22 March 2018 SARAH MCDOWELL HEALTH QUALITY ONTARIO
  38. 38. 39 Outline • Overview of HQO and the Health Technology Assessment (HTA) Program • Three examples of how we have used GRADE in our HTAs
  39. 39. 40 Health Technology Assessment at Health Quality Ontario • HQO has a legislated mandated to make recommendations to the Minister of Health on public funding for health care services and medical devices • The Ontario Health Technology Advisory Committee (OHTAC) reviews the HTA evidence and makes recommendations to HQO • Within the HTAs, we consider: – Clinical benefits and harms – Value for money and affordability – Patient preferences and values
  40. 40. 41 Use of GRADE at Health Quality Ontario • Used within the clinical evidence component of the HTA • Have used GRADE since 2004 • OHTAC makes a yes/no recommendation, and does not GRADE the strength of the recommendations
  41. 41. 42 GRADE Examples
  42. 42. 43 Mechanical Thrombectomy for Patients With Acute Ischemic Stroke Research question • What are the clinical benefits and harms of endovascular treatment via new generation mechanical thrombectomy devices (with or without IVT) compared to IVT alone (if eligible) in patients with acute ischemic stroke caused by a proximal intracranial occlusion in the anterior circulation? Results • Systematic review identified 5 RCTs (ranged in size from 50 to 500 patients)
  43. 43. 44 Mechanical Thrombectomy for Patients With Acute Ischemic Stroke Mortality Functional independence
  44. 44. 45 Mechanical Thrombectomy for Patients With Acute Ischemic Stroke Outcome No. of participants (studies) Anticipated absolute effects (95% CI) Relative effect (95% CI) Certainty of the evidence (GRADE)Risk with IVT Risk difference with MT Functional independence N = 1278 (5 RCTs) 264 per 1,000 197 more per 1,000 (139 more to 258 more) OR: 2.39 (1.88 to 3.04) ⊕⊕⊕⊕ HIGH Mortality N = 1282 (5 RCTs) 188 per 1,000 32 fewer per 1,000 (66 fewer to 11 more) OR: 0.8 (0.6 to 1.07) ⊕⊕⊕ MODERATE due to imprecision Symptomatic intracerebral hemorrhage N = 1286 (5 RCTs) 43 per 1,000 4 more per 1,000 (14 fewer to 34 more) OR: 1.11 (0.66 to 1.87) ⊕⊕⊕ MODERATE due to imprecision MT = Mechanical thrombectomy; IVT = Intravenous thrombolysis
  45. 45. 46 Mechanical Thrombectomy for Patients With Acute Ischemic Stroke Funding recommendation (February 2016) • “Health Quality Ontario, under the guidance of the Ontario Health Technology Advisory Committee, recommends publicly funding stent retrievers and thromboaspiration devices for mechanical thrombectomy in patients with acute ischemic stroke, in selected stroke centres identified by the Ontario Stroke Network.” http://www.hqontario.ca/portals/0/documents/evidence/reports/recommendation- mechanical-thrombectomy-1602-en.pdf
  46. 46. 47 Electrical Stimulation for Pressure Injuries Research question • In adults with pressure injuries, what are the clinical benefits and harms of electrical stimulation plus standard wound care compared with standard wound care alone? Results • Systematic review identified 9 RCTs and 2 non- randomized controlled trials
  47. 47. 48 Electrical Stimulation for Pressure Injuries Outcome No. of participants (studies) Results Certainty of the evidence (GRADE) Time to heal N = 97 (2 RCTs) No difference between healing times comparing Estim vs. standard wound care ⊕⊕ LOW due to risk of bias and imprecision Complications N = 236 (5 RCTs) 3/5 studies: minor events reported ⊕⊕⊕⊕ HIGH Relative rate of healing N = 211 (1 RCT, 2 NRCTs) 2/3 studies found significant difference comparing Estim vs. standard wound care ⊕⊕ LOW due to risk of bias and indirectness
  48. 48. 49 Electrical Stimulation for Pressure Injuries Funding recommendation (November 2017) • “Health Quality Ontario, under the guidance of the Ontario Health Technology Advisory Committee, recommends against publicly funding electrical stimulation for pressure injuries” http://www.hqontario.ca/Portals/0/documents/evidence/reports/ohtac-recommendations- electrical-stimulation-1711-en.pdf
  49. 49. 50 Retinal Prosthesis System for Advanced Retinitis Pigmentosa Research question • What are the clinical benefits and harms of the Argus II retinal prosthesis system when used to treat patients with bare to no light perception vision as a result of advanced retinitis pigmentosa? Results • Systematic review identified 1 prospective cohort study of 30 patients
  50. 50. 51 Retinal Prosthesis System for Advanced Retinitis Pigmentosa Visual function Outcome No. of participants (studies) Results* Certainty of the evidence (GRADE) Object localization N = 30 (1 non-RCT) 1-year: 93.8% 3-year: 89.3% 5-year: 80.9% ⊕⊕⊕ MODERATE upgraded due to underlying trajectory of retinitis pigmentosa (+1) Direction of motion N = 30 (1 non-RCT) 1-year: 62.5% 3-year: 55.6% 5-year: 50.0% ⊕⊕⊕ MODERATE upgraded due to underlying trajectory of retinitis pigmentosa (+1) Grating visual acuity N = 30 (1 non-RCT) 1-year: 48.2% 3-year: 33.3% 5-year: 38.1% ⊕⊕⊕ MODERATE upgraded due to underlying trajectory of retinitis pigmentosa (+1) *Results were % patients performed significantly better (P<0.05) with Argus II on vs. off
  51. 51. 52 Retinal Prosthesis System for Advanced Retinitis Pigmentosa Funding recommendation (November 2017) • “Health Quality Ontario, under the guidance of the Ontario Health Technology Advisory Committee recommends publicly funding the Argus II retinal prosthesis system for advanced retinitis pigmentosa” http://www.hqontario.ca/Portals/0/documents/evidence/reports/ohtac-recommendations- retinal-prosthesis-1711-en.pdf
  52. 52. 53 General Advice and Reflections • Discuss the ratings with colleagues • Engage with clinicians to gain an understanding of the health care condition • GRADE summary of findings table is useful for conveying information • Follow GRADE working group
  53. 53. 54 Summary • GRADE provides us with a structured, consistent approach to evaluate and present the quality of the body of the evidence • GRADE provides important information for decision- making about the certainty of the effectiveness of a treatment.
  54. 54. 55 References HQO Evidence to Improve Care http://www.hqontario.ca/Evidence-to-Improve-Care Mechanical Thrombectomy in Patients With Acute Ischemic Stroke HTA: http://www.hqontario.ca/Portals/0/Documents/evidence/reports/hta-mechanical-thrombectomy-1602- en.pdf HQO recommendation: http://www.hqontario.ca/portals/0/documents/evidence/reports/recommendation- mechanical-thrombectomy-1602-en.pdf Retinal Prosthesis System for Advanced Retinitis Pigmentosa HTA: http://www.hqontario.ca/Portals/0/documents/evidence/reports/hta-retinal-prosthesis-1711.pdf HQO recommendation: http://www.hqontario.ca/Portals/0/documents/evidence/reports/ohtac- recommendations-retinal-prosthesis-1711-en.pdf Electrical Stimulation for Pressure Injuries HTA: http://www.hqontario.ca/Portals/0/documents/evidence/reports/hta-electrical-stimulation-1711.pdf HQO recommendation: http://www.hqontario.ca/Portals/0/documents/evidence/reports/ohtac- recommendations-electrical-stimulation-1711-en.pdf HQO HTA Process and Methods Guide: http://www.hqontario.ca/Portals/0/documents/evidence/reports/hta-methods-and-process-guide-en.pdf
  55. 55. Follow us @nccmt Suivez-nous @ccnmo 57 Your Comments/Questions • Use Chat to post comments and/or questions • ‘Send’ questions to All (not privately to ‘Host’) Chat Participant Side Panel in WebEx
  56. 56. Follow us @nccmt Suivez-nous @ccnmo 58 Poll Question #3 Could this method or tool be useful in practice? A. Very useful B. Somewhat useful C. Not at all useful D. Don’t know
  57. 57. Follow us @nccmt Suivez-nous @ccnmo 59 Your Feedback is Important Please take a few minutes to share your thoughts on today’s webinar. Your comments and suggestions help to improve the resources we offer and plan future webinars. The short survey is available at: https://surveys.mcmaster.ca/limesurvey/index.php/ 163839?lang=en
  58. 58. Follow us @nccmt Suivez-nous @ccnmo 60 Poll Question #4 What are your next steps? (Check all that apply) A. Access the method/tool referenced in the presentation B. Read the NCCMT summary about the method/tool described today C. Consider using the method/tool in practice D. Tell a colleague about the method/tool
  59. 59. Follow us @nccmt Suivez-nous @ccnmo Share your story! • Are you using EIDM in your practice? We want to hear about it! • Email us: nccmt@mcmaster.ca • Need support for EIDM? Contact us for help! • Email us: nccmt@mcmaster.ca • We typically respond within 24 business hours 61
  60. 60. Follow us @nccmt Suivez-nous @ccnmo Webinar Series from the NCCMT www.nccmt.ca/webinar-series • Spotlight on KT Methods and Tools • Topic-Specific Methods and Tools • Online Journal Club • Peer-to-peer Webinars 62
  61. 61. Follow us @nccmt Suivez-nous @ccnmo Funded by the Public Health Agency of Canada | Affiliated with McMaster University Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The views expressed here do not necessarily reflect the views of the Public Health Agency of Canada.. For more information about the National Collaborating Centre for Methods and Tools: NCCMT website www.nccmt.ca Contact: nccmt@mcmaster.ca

×