1. Grading the quality of evidence and the strength of recommendations San Sebastián 12 - 14 May 2008 Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann

3. Introduction Andy

13. What is quality of evidence and how should it be graded?

21. Quality assessment criteria

24. Determinants of strength of recommendation Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.

28. General questions about GRADE, the GRADE Working Group or the agenda for this workshop?

29. Relative importance of outcomes Holger

31. Hierarchy of outcomes according to their importance to assess the effect of phosphate lowering drugs in patients with renal failure and hyperphosphatemia Flatulence 2 Importance of endpoints 5 Pain due to soft tissue 6 calcification / function Fractures 7 Myocardial infarction 8 Mortality 9 3 4 1 Critical for decision making Important, but not critical for decision making Of low importance

36. 2 Importance of endpoints 5 6 7 8 9 3 4 1 Critical for decision making Important, but not critical for decision making Of low importance

38. Questions about the relative importance of outcomes?

39. Study limitations Victor

41. Testosterone for BMD in men

42. Assessment of the risk of bias Cochrane Handbook for Systematic Reviews of Interventions, Draft chapter of version 5 - Assessing risk of bias in included studies http://www.cochrane.org/resources/handbook/handbook5drafts.htm

43. Summarizing study limitations for randomized trials

44. Questions about study limitations?

45. Inconsistency of results Holger

47. Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “ Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer ”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650. Heparin or vitamin K antagonists for survival in patients with cancer

48. Non-steroidal drug use and risk of pancreatic cancer Capurso G, Sch ünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories. Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.

49. Example: Thromboprophylaxis and bleeding in the hospital Dentali et al. Ann Int Med, 2007

50. Questions about inconsistency of results?

51. Indirectness of evidence Victor

56. Questions about indirectness of evidence?

57. Imprecise results Victor

61. Stroke – periop beta blockers

63. 0 1.0%

64. 0 1.0%

65. 0 1.0%

66. 0 0.5% 1.0%

67. 0 0.5% 1.0%

68. 0 1.0%

69. 0 1.0%

71. Rate down for imprecision? Rate down for imprecision for guidelines?

72. Bleeding in patients with cancer receiving heparing for prolonging survival Rate down for imprecision for guidelines?

74. Questions about imprecise results?

75. Publication bias Victor

77. Furukawa TA et al. JAMA 2007; 297: 468-9 1 10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 <20% 20-39% 40-59% 60-80% >80% Continuous outcomes Dichotomous outcomes % eligible trials contributing data to meta-analysis Effect size Odds ratio

83. Questions about publication bias?

84. Factors that can raise the quality of evidence Andy

87. What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a large magnitude of effect? Why would you or would you not upgrade the quality of evidence?

91. What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a dose response relationship?

94. What are examples of interventions you can think of for which you might upgrade the quality of evidence when an effect (association) was found - because all plausible confounders would result in an underestimate of effect or when an effect (association) was not found - because all plausible biases would result in an overestimate of effect?

95. Questions about factors that can raise the quality of evidence?

96. Judgements about the overall quality of evidence Andy

97. Quality assessment criteria

106. Threshold for downgrading highest lowest OK rate down quality

107. Overall quality across criteria highest lowest high moderate low very low

109. Questions about judgements of the overall quality of evidence?

111. Should routine HPV vaccination be recommended for 12 year old girls in Spain (with continued screening)? Introduction to small group work

112. Decide which outcomes are important

113. Assess the quality of evidence

114. Questions about the small group work?

115. Grading the quality of evidence and the strength of recommendations San Sebastián 13 May 2008 Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann

117. Any questions from yesterday about quality of evidence or about the agenda for today?

118. Summarising and presenting findings Andy

120. Quality assessment

121. Summary of Findings

122. GRADE Evidence Profile

124. Headings specify the comparison and applicability of the SoF table

125. Up to 7 patient-important outcomes including important outcomes for which no data are available

126. The relative effect (e.g. risk ratio)

127. Illustrative comparative risks present the absolute effect for specified (“assumed”) baseline or control group risks

128. How much evidence there is

129. The quality of the evidence

130. Comments

131. Footnotes About the intervention High and low risk groups Judgements about quality Explanation of outcomes

132. Explanations for SoF tables

133. Questions about summarising and presenting findings?

134. Strength of recommendations and values & preferences Holger

138. You have to decide where you want to spend the next five years of your live. Which of the following cities would you prefer to live in?

139. ← Option 1 (Buffalo, purple) Option 2 -> (Rome, yellow)

141. You are hiking. Which of the following animals would you prefer to encounter?

142. ← Option 1 (bunny, purple) Option 2 -> (tiger, yellow)

143. You are buying an ice cream. Which flavor do you prefer?

144. ← Option 1 (strawberry, purple) Option 2 -> (chocolate, yellow) Chocolate Strawberry

145. You are buying a new car. Which one would you buy?

146. ← Option 1 (Ferrari, purple) Option 2 -> (Volkswagen, yellow) Yellow fox Red Ferrari

147. What determines your choices?

156. Determinants of the strength of recommendation

157. Determinants of the strength of recommendation

158. Determinants of the strength of recommendation

164. Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? Summary of findings Transmission : No human to human transmission Patient or population: Hospitalised, clinical and serologically confirmed cases of avian influenza

167. Judgments about the strength of a recommendation - oseltamivir for treatment of patients hospitalised with avian influenza (H5N1) Factors Comments Balance between desirable and undesirable effects The benefits are uncertain. Quality of the evidence The quality of the evidence is very low. Values and preferences All patients and care providers would accept treatment for H5N1 disease. Costs (resource use) The cost is not high for treatment of sporadic cases.

168. Questions about the strength of recommendations?

169. Costs (resource use) Andy

170. BMJ 2008;336:1049-1051 (10 May)

171. Determinants of strength of recommendation Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.

178. Cost-effectiveness analyses of HPV vaccination Lancet Infect Dis 2007; 7:289–96

179. Resource implications of HPV vaccination Lancet Infect Dis 2007; 7:289–96

180. Questions about the impact of costs (resource use) on the strength of a recommendation?

181. Wording of recommendations Holger

187. ATS/ERS COPD guidelines

188. NICE COPD guidelines

189. GOLD COPD guidelines

191. Questions about the wording of recommendations?

192. Recommendations to use an intervention only in the context of research Andy

197. What are examples of interventions you can think of for which you might recommend that it should only be used in the context of research?

198. Questions about recommendations to use an intervention only in the context of research?

199. Judgements about priorities (importance) Andy

200. Priority setting: From a health system or public health perspective A strong recommendation may not be important

203. Questions about judgements about priorities (importance)?