GRADE ( G rades of r ecommendation, a ssessment, d evelopment and e valuation)
Australian NMRC, SIGN, USPSTF, WHO, NICE, Oxford CEBM, CDC, CC
~ 25 meetings over last ten years
(~10 – 50 attendants)
GRADE Uptake Agencia sanitaria regionale, Bologna, Italia Agency for Health Care Research and Quality (AHRQ) Allergic Rhinitis and Group - Independent Expert Panel American Association for the study of liver diseases American College of Cardiology Foundation American College of Chest Physicians American College of Emergency Physicians American College of Physicians American Endocrine Society American Society of Gastrointestinal Endoscopy American society of Interventional Pain Physicians American Thoracic Society (ATS) BMJ Clinical Evidence British Medical Journal Canadian Agency for Drugs and Technology in Health Canadian Cardiovascular Society Canadian Task Force on Preventive Health Care Centers for Disease Control Cochrane Collaboration EBM Guidelines Finland Emergency Medical Services for Children National Resource Center European Association for the Study of the Liver European Respiratory Society European Society of Thoracic Surgeons Evidence-based Nursing Sudtirol, Alta Adiga, Italy Finnish Office of Health Technology Assessment German Agency for Quality in Medicine Heelth Inspectorate for Scotland Infectious Disease Society of America Japanese Society of Oral and Maxillofacial Radiology Joslin Diabetes Center Journal of Infection in Developing Countries Kaiser Permanente Kidney Disease International Guidelines Organization National and Gulf Centre for Evidence-based Medicine National Institute for Clinical Excellence (NICE) National Kidney Foundation Norwegian Knowledge Centre for the Health Services Ontario MOH Medical Advisory Secretariat Panama and Costa Rica National Clinical Guidelines Program Polish Institute for EBM Scottish Intercollegiate Guideline Network (SIGN) Society of Critical Care Medicine Society of Pediatric Endocrinology Society of Vascular Surgery Spanish Society of Family Practice (SEMFYC) Stop TB Diagnostic Working Group Surviving sepsis campaign Swedish Council on Technology Assessment in Health Care Swedish National Board of Health and Welfare University of Pennsylvania Health System for EB Practice UpToDate WINFOCUS World Allergy Organization World Health Organization (WHO)
High: We are very confident that t he true effect lies close to that of the estimate of the effect.
Moderate: We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different.
Low: Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect.
Very low: We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect.
Health Care Question (PICO) Systematic reviews Studies Outcomes Important outcomes Rate the quality of evidence for each outcome , across studies RCTs start high, observational studies start low (-) Study limitations Imprecision Inconsistency of results Indirectness of evidence Publication bias likely Final rating of quality for each outcome: high, moderate, low, or very low (+) Large magnitude of effect Dose response Plausible confounders would ↓ effect when an effect is present or ↑ effect if effect is absent Decide on the direction (for/against) and grade strength (strong/weak*) of the recommendation considering: Quality of the evidence Balance of desirable/undesirable outcomes Values and preferences Decide if any revision of direction or strength is necessary considering: Resource use *also labeled “conditional” or “discretionary” Rate overall quality of evidence (lowest quality among critical outcomes) S1 S2 S3 S4 OC1 OC2 OC3 OC4 OC1 OC3 Critical outcomes OC4 Generate an estimate of effect for each outcome OC2 S5
Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures
Chapuy et al, (2002) 0.85 (0.64, 1.13)
Pooled Random Effect Model 0.82 (0.69 to 0.98) p= 0.05 for heterogeneity, I 2 =53% Chapuy et al, (1994) 0.79 (0.69, 0.92) Lips et al, (1996) 1.10 (0.87, 1.39) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000) 0.48 (0.13, 1.78) Meyer et al, (2002) 0.92 (0.68, 1.24) Trivedi et al, (2003) 0.67 (0.46, 0.99) Favours Vitamin D Favours Control Relative Risk 95% CI
Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose >400) Chapuy et al, (1994) 0.70 (0.69, 0.92) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000) .48 (0.13, 1.78) Chapuy et al, (2002) 0.85 (0.64, 1.13) Trivedi et al, (2003) 0.67 (0.46, 0.99) Pooled Random Effect Mode 0.75 (0.63 to 0.89) p= 0.26 for heterogeneity, I 2 =24% Favours Vitamin D Favours Control Relative Risk 95% CI
Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose = 400) Lips et al (1996) 1.10 (0.87, 1.39) Meyer et al (2002) 0.92 (0.68, 1.24) Pooled Random Effect Mode 1.03 (0.86 to 1.24) p = 0.35 heterogeneity, I 2 =0% Favours Vitamin D Favours Control Relative Risk 95% CI
Figure 6: Hierarchy of outcomes according to their patient-importance to assess the effect of phosphate lowering drugs in patients with renal failure and hyperphophatemia Flatulence Importance of endpoints 2 5 Pain due to soft tissue Calcification / function 6 Fractures 7 Myocardial infarction 8 Mortality 9 3 4 1 Coronary calcification Ca 2+ /P- Product Surrogates of declining importance Bone density Ca 2+ /P- Product Soft tissue calcification Ca 2+ /P- Product Lower by one level for indirectness Lower by two levels for indirectness Critical for decision making Important, but not critical for decision making Of low patient- importance
Alendronate Risedronate Placebo Directness interested in A versus B available data A vs C, B vs C
most systems just use evidence about primary benefit outcome
but what about others (risk)?
what to do?
ignore all but primary
weakest of any outcome
some blended approach
weakest of critical outcomes
Beta blockers in non-cardiac surgery Quality Assessment Summary of Findings Quality Relative Effect (95% CI) Absolute risk difference Outcome Number of participants (studies) Risk of Bias Consistency Directness Precision Reporting Bias Myocardial infarction 10,125 (9) No serious limitations No serious imitations No serious limitations No serious limitations Not detected High 0.71 (0.57 to 0.86) 1.5% fewer (0.7% fewer to 2.1% fewer) Mortality 10,205 (7) No serious limitations Possiblly inconsistent No serious limitations Imprecise Not detected Moderate or low 1.23 (0.98 – 1.55) 0.5% more (0.1% fewer to 1.3% more) Stroke 10,889 (5) No serious limitaions No serious limitations No serious limitations Possible imprecision Not detected Moderate or High 2.21 (1.37 – 3.55) 0.5% more (0.2% more to 1.3% more0
Pressure limited ventilation Quality Assessment Summary of Findings Quality Relative Risk (95% CI) p-value Illustrative risks Outcome No. of patients (studies) Risk of Bias Inconsistency Indirectness Imprecision Publication Bias Example control rate Associated risk with PVL Hospital mortality 1,664 (9) Inability to blind. 2 trials stopped early with few events and large effects; were also confounded by ‘open lung’ strategies. p = 0.07 I 2 = 45.6% Varied populations, interventions. Not robust in sensitivity analyses Direct Precise Undetected Moderate (due to inconsistency) 0.82 (0.68 – 0.99) p = 0.04 40% 32.8% (27.2 – 39.6) Barotrauma 1,497 (7) Inability to blind. p = 0.24 I 2 = 25.3% Varied populations, interventions Direct Imprecise Undetected Moderate (due to imprecision) 0.90 (0.66 – 1.24) p = 0.53 NS NS Paralysis 1,202 (5) Inability to blind. p = 0.004 I 2 = 59% Varied populations, interventions, measurements Direct Precise Not assessed Moderate (due to inconsistency) 1.37 (1.04 – 1.82) p = 0.03 30% 41.1% (31.2 – 54.6) Dialysis 173 (2) Inability to blind. p = 0.26 I 2 = 22.8% Varied populations, interventions Direct Imprecise Not assessed Moderate (due to imprecision) 1.76 (0.79 – 3.90) p = 0.16 NS NS
Zoster vaccine Quality Assessment Summary of Findings Quality Relative Risk (95% CI) p-value Illustrative risks Outcome No. of patients (studies) Risk of Bias Inconsistency Indirectness Imprecision Publication Bias control rate vaccinated rate Zoster episodes 38,546 (1) No serious risk only one study Direct Precise Undetected High not reported 11.12 per 1,000 patient-years 5.42 (difference 5.7 per 1,000 pt-years (p< 0.001) Post-herpetic neuralgia 38,546 (1) No serious risk only one study Direct Precise Undetected High not reported 1.38 per 1,000 patient-years 0.46 (difference 0.92 per 1,000 pt-years (p< 0.001) Serious adverse events 38,546 (1) No serious risk only one study Direct Precise Undetected High Not reported 13 per 1,000 19 (difference 6 per 1,000)
High versus low PEEP in ALI and ARDS Population No. of participants (trials) † Higher PEEP Lower PEEP Adjusted Relative Risk (95% CI; P -value) ‡ Adjusted Absolute Risk Difference (95% CI) Quality Patients with ARDS 1892 (3) 324/951 (34.1%) 368/941 (39.1%) 0.90 (0.81 to 1.00; 0.049) -3.9% (-7.4% to -0.04%) High Patients without ARDS 404 (3) 50/184 (27.2%) 41/220 (18.6%) 1.37 (0.98 to 1.92; 0.065) 6.9% (-0.4% to 17.1%) Moderate (imprecision)
Whipples procedure pancreatic cancer with or without duodenectomy
Patients or population: Anyone taking a long flight (lasting more than 6 hours) Settings: International air travel Intervention: Compression stockings 1 Comparison: Without stockings Outcomes Illustrative comparative risks * (RANGE OF UNCERTAINTY) Relative effect (95% CI) Number of participants (studies) Quality of the evidence (GRADE) Comments Assumed risk Corresponding risk Without stockings With stockings (95% CI) Symptomatic deep vein thrombosis (DVT) See comment See comment Not estimable 2637 (9 studies) See comment 0 participants developed symptomatic DVT in these studies. Symptomatic deep vein thrombosis –surrogate symptomless deep vein thrombosis Low risk population 2 RR 0.10 (0.04 to 0.25) 2637 (9 studies) Moderate 3 Estimates of control group asymptomatic thrombosis from the primary studies range from 15 per 1,000 in low risk patients to 25 per 1,000 in high risk patients 5 per 10,000 0.5 per 10,000 (0 to 1) High risk population 2 18 per 10,000 1.8 per 10,000 (0.5 to 4) Superficial vein thrombosis 13 per 1000 6 per 1000 (2 to 15) RR 0.45 (0.18 to 1.13) 1804 (8 studies) Moderate 4
Test accuracy is a surrogate for patient important outcomes
When clinicians think about diagnostic tests, they focus on their accuracy
Underlying assumption: obtaining a better idea of whether a target condition is present or absent will result in superior patient management and improved outcome.
Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.
For women age 40 – 49 we suggest NOT screening. (Weak recommendation based, moderate quality evidence. 2B) Values and Preferences: This recommendation places a relatively low value on a very small, uncertain mortality decrease and reflects concerns with false positive results, unnecessary biopsies, and unnecessary dx of breast cancer Women who place a higher value on a small reduction in breast cancer mortality and are less concerned about the undesirable consequences will choose screening
For women age 50 - 74 we suggest screening. (Weak recommendation, moderate quality evidence. 2B) Women who do not place a high value on a small reduction in breast cancer mortality and are concerned about false positive results, unnecessary biopsies, and unnecessary diagnosis of breast cancer will decline screening
Stroke guideline: patients with TIA clopidogrel over aspirin (Grade 2B).
Underlying values and preferences : This recommendation to use clopidogrel over aspirin places a relatively high value on a small absolute risk reduction in stroke rates, and a relatively low value on minimizing drug expenditures.
for presentation of resource use evidence need to specify perspective
Quality assessment Studies Design Limitations Inconsistency Indirectness Imprecision No of patients Relative effect (95% CI) Quality Eclampsia Duley 2003 RCT No one trial only No No 9,992 RR 0.41 (0.29-0,58) High Maternal death Duley 2003 RCT No one trial only No Imprecision 9,992 RR 0.54 (0.26-1,10) Moderate
Quality assessment Design Limita-tions Inconsis-tency Indirect- ness Impre-cision Resources Costs per patient Studies per patient (US $; year 2001) Placebo MgSO4 Placebo MgSO4 Magnesium sulphate High GNI 0 6 0 20 Simon 2005 Middle GNI RCT No one trial only No No 0 6 0 3 High Low GNI 0 6 0 5 Administration of the drug High GNI 0 1 0 66 Simon 2005 Middle GNI RCT No one trial only No No 0 1 0 14 High Low GNI 0 1 0 8 Other hospital resources a, b High GNI Large variationresources c NA NA 12,839 12,818 Simon 2005 Middle GNI RCT No one trial only No NA NA 1,412 1,416 Moderate Low GNI NA NA 155 157
Outcomes Typical control group risk Typical absolute effect (95% CI) Relative effect (95% CI) Nr. of participants (studies) Quality of the evidence Comments Clinical outcomes Eclampsia Severe RR 0.41 (0.29 - 0.58) 11,444 High 27 per 1,000 16 fewer per 1,000 (11 to 19) Not severe 15 per 1,000 9 fewer per 1,000 (6 to 11) Maternal death Severe RR 0.54 (0.26 - 1.10) 10,795 Moderate 2 6 per 1,000 3 fewer per 1,000 (0.6 more to 4 fewer) Not severe 3 per 1,000 1 fewer per 1,000 (0.3 more to 2 fewer) Side effects 46 per 1,000 3 196 more per 1,000 (165 to 231) RR 5.26 (4.59 - 6.03) 9.992 High Mostly flushing. Other side effects include nausea, vomiting, slurred speech, muscle weakness, dizziness, drowsiness, confusion and headache.
Resource use from the perspective of the health system control grop difference trt vs control Magnesium sulphate ampoules 0 6 10 ml. ampoules per woman 9.996 High Cost High GNI Middle GNI Low GNI $20 more per patient $ 3 more per patient $ 5 more per patient Administration of magnesium sulphate 0 1 per woman 9.996 High Cost High GNI Middle GNI Low GNI $66 per patient $14 per patient $ 8 per patient Resources for administering magnesium sulphate included midwife time (main cost), intravenous cannula/needle, syringe, IV fluids, drug. Other hospital resources Varied widely 9.996 Moderate 5 There was large variation in the use of other hospital resources in both intervention and control groups. Cost High GNI Middle GNI Low GNI $12,839 $ 1,416 $ 157 $20 less per woman (0 to 60) $ 4, less per woman (0 to 10) $ 2 less per woman (1 to 3) Other hospital costs have been adjusted based on the influence of eclampsia to control for the many other factors that influenced these costs.