1. Grading the quality of evidence and the strength of recommendations San Sebastián 12 - 14 May 2008 Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann

3. Introduction Andy

13. What is quality of evidence and how should it be graded?

21. Quality assessment criteria

24. Determinants of strength of recommendation Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.

28. General questions about GRADE, the GRADE Working Group or the agenda for this workshop?

29. Relative importance of outcomes Holger

31. Hierarchy of outcomes according to their importance to assess the effect of phosphate lowering drugs in patients with renal failure and hyperphosphatemia Flatulence 2 Importance of endpoints 5 Pain due to soft tissue 6 calcification / function Fractures 7 Myocardial infarction 8 Mortality 9 3 4 1 Critical for decision making Important, but not critical for decision making Of low importance

36. 2 Importance of endpoints 5 6 7 8 9 3 4 1 Critical for decision making Important, but not critical for decision making Of low importance

38. Questions about the relative importance of outcomes?

39. Study limitations Victor

41. Testosterone for BMD in men

42. Assessment of the risk of bias Cochrane Handbook for Systematic Reviews of Interventions, Draft chapter of version 5 - Assessing risk of bias in included studies http://www.cochrane.org/resources/handbook/handbook5drafts.htm

43. Should patients with cancer receive parenteral anticoagulation (heparin) to prolong survival. Outcomes: Survival, bleeding

47. Detailed study design and execution

50. Detailed study design and execution

52. Bleeding in patients with cancer receiving heparing for prolonging survival

54. Summarizing study limitations for randomized trials

55. Questions about study limitations?

56. Inconsistency of results Holger

58. Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “ Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer ”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650. Heparin or vitamin K antagonists for survival in patients with cancer

59. Non-steroidal drug use and risk of pancreatic cancer Capurso G, Sch ünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories. Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.

60. Example: Thromboprophylaxis and bleeding in the hospital Dentali et al. Ann Int Med, 2007

61. Questions about inconsistency of results?

62. Indirectness of evidence Victor

67. Questions about indirectness of evidence?

68. Imprecise results Victor

72. Stroke – periop beta blockers

74. 0 1.0%

75. 0 1.0%

76. 0 1.0%

77. 0 0.5% 1.0%

78. 0 0.5% 1.0%

79. 0 1.0%

80. 0 1.0%

82. Rate down for imprecision? Rate down for imprecision for guidelines?

83. Bleeding in patients with cancer receiving heparing for prolonging survival Rate down for imprecision for guidelines?

85. Questions about imprecise results?

86. Publication bias Victor

88. Furukawa TA et al. JAMA 2007; 297: 468-9 1 10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 <20% 20-39% 40-59% 60-80% >80% Continuous outcomes Dichotomous outcomes % eligible trials contributing data to meta-analysis Effect size Odds ratio

94. Questions about publication bias?

95. Factors that can raise the quality of evidence Andy

98. What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a large magnitude of effect? Why would you or would you not upgrade the quality of evidence?

102. What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a dose response relationship?

105. What are examples of interventions you can think of for which you might upgrade the quality of evidence when an effect (association) was found - because all plausible confounders would result in an underestimate of effect or when an effect (association) was not found - because all plausible biases would result in an overestimate of effect?

106. Questions about factors that can raise the quality of evidence?

107. Judgements about the overall quality of evidence Andy

108. Quality assessment criteria

117. Threshold for downgrading highest lowest OK rate down quality

118. Overall quality across criteria highest lowest high moderate low very low

120. Questions about judgements of the overall quality of evidence?

122. Should routine HPV vaccination be recommended for 12 year old girls in Spain? Introduction to small group work

123. Questions about the small group work?

124. Grading the quality of evidence and the strength of recommendations San Sebastián 13 May 2008 Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann

126. Any questions from yesterday about quality of evidence or about the agenda for today?

127. Summarising and presenting findings Andy

128. Questions about summarising and presenting findings?

129. Strength of recommendations Holger

132. Determinants of strength of recommendation Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.

138. Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? Summary of findings Transmission : No human to human transmission Patient or population: Hospitalised, clinical and serologically confirmed cases of avian influenza

140. Judgments about the strength of a recommendation - oseltamivir for treatment of patients hospitalised with avian influenza (H5N1) Factors Comments Balance between desirable and undesirable effects The benefits are uncertain. Quality of the evidence The quality of the evidence is very low. Values and preferences All patients and care providers would accept treatment for H5N1 disease. Costs (resource use) The cost is not high for treatment of sporadic cases.

142. Questions about the strength of recommendations?

143. Wording of recommendations ??

144. Questions about the wording of recommendations?

145. Recommendations to use an intervention only in the context of research Andy

150. What are examples of interventions you can think of for which you might recommend that it should only be used in the context of research?

151. Questions about recommendations to use an intervention only in the context of research?

152. Judgements about priorities (importance) Andy

153. Priority setting: From a health system or public health perspective A strong recommendation may not be important

156. Questions about judgements about priorities (importance)?