0
Grading the quality of evidence and the strength of recommendations San Sebastián 12 - 14 May 2008 Pablo Alonso, Victor Mo...
Workshop agenda Monday, 12 May 2008 <ul><li>08.15 – 13.30 h - Plenary </li></ul><ul><li>Introduction to GRADE </li></ul><u...
Introduction Andy
Why bother about grading? <ul><li>People draw conclusions about the </li></ul><ul><ul><li>quality of evidence </li></ul></...
Many grading systems <ul><li>Evidence Recommendation </li></ul><ul><li>B Class I </li></ul><ul><li>C+  1 </li></ul><ul><li...
A common international grading system? <ul><li>International group </li></ul><ul><ul><li>ACCP, AHRQ, Australian NMRC, BMJ ...
<ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Ag...
<ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Ag...
<ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Ag...
<ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Ag...
<ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Ag...
<ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Ag...
What is quality of evidence and how should it be graded?
 
Judging the quality of evidence requires considering the context <ul><li>In the context of making recommendations   </li><...
Although the degree of confidence is a continuum, we suggest using four categories <ul><li>High </li></ul><ul><li>Moderate...
Categories of quality <ul><li>High : Further research is very unlikely to change our confidence in the estimate of effect....
Study design is important <ul><li>Early systems of grading the quality of evidence focused almost exclusively on study des...
What can lower the quality of evidence? <ul><li>Study limitations </li></ul><ul><li>Inconsistency of results  </li></ul><u...
What can raise the quality of evidence? <ul><li>Large magnitude of effect  </li></ul><ul><li>A dose response relationship ...
Quality assessment criteria
Strength of recommendation <ul><li>The degree of confidence that the desirable effects of adherence to a recommendation ou...
Categories of recommendations <ul><li>Although the degree of confidence is a continuum, we suggest using two categories: s...
Determinants of strength of recommendation   Factors Impact on the strength of a recommendation Balance between desirable ...
Workshop agenda Monday, 12 May 2008 Quality of evidence <ul><li>08.15 – 13.30 h - Plenary </li></ul><ul><li>Introduction t...
Workshop agenda Tuesday, 13 May 2008 Strength of recommendation <ul><li>08.30 – 13.30 h - Plenary </li></ul><ul><li>Going ...
Workshop agenda Wednesday, 14 May 2008 Special challenges <ul><li>08.30 – 11.30 h – Plenary + small group work </li></ul><...
General questions about GRADE, the GRADE Working Group or the agenda for this workshop?
Relative importance of outcomes Holger
The quality of evidence needs to be considered for each important outcome <ul><li>The quality of evidence may be different...
Hierarchy of outcomes according to their importance to assess the effect of phosphate lowering drugs in patients with rena...
A different clinical question <ul><li>Population:  In patients with chronic atrial  fibrillation and no prior history of  ...
<ul><li>Population:  In patients with chronic atrial  fibrillation and no prior history of  stroke </li></ul><ul><li>Inter...
<ul><li>All disabling or fatal stroke (isch. and hemorrh.) </li></ul><ul><li>Major hemorrhage (non IC) </li></ul><ul><li>A...
Outcomes/endpoints <ul><li>How important is the endpoint for decision making? </li></ul><ul><li>Judgment about the relativ...
2  Importance  of endpoints 5 6   7 8 9 3 4 1 Critical  for decision making Important,  but   not critical for  decision m...
<ul><li>All disabling or fatal stroke (isch. and hemorrh.) </li></ul><ul><li>Major hemorrhage (non IC) </li></ul><ul><li>A...
Questions about the relative importance of outcomes?
Study limitations Victor
Study limitations (Risk of bias) for Randomized trials <ul><li>Lack of allocation concealment </li></ul><ul><li>Lack of bl...
Testosterone for BMD in men
Assessment of the risk of bias Cochrane Handbook for Systematic Reviews of Interventions, Draft chapter of version 5 - Ass...
Should patients with cancer receive parenteral anticoagulation (heparin) to prolong survival.  Outcomes: Survival, bleeding
Survival <ul><li>HR 0.77 (0.65 to 0.91) </li></ul>
<ul><li>How confident are you that these results are true? </li></ul>
Factors that lower quality <ul><li>detailed design and execution </li></ul><ul><li>inconsistency </li></ul><ul><li>indirec...
Detailed study design and execution
Quality now? <ul><li>High </li></ul>
Major bleeding – start high <ul><li>RR 1.50 (0.26 – 8.80) </li></ul>
Detailed study design and execution
Quality now? <ul><li>Moderate </li></ul>
Bleeding in patients with cancer receiving heparing for prolonging survival
Quality now? <ul><li>Low </li></ul><ul><li>Observational studies could have provided higher quality evidence </li></ul>
Summarizing study limitations for randomized trials
Questions about study limitations?
Inconsistency of results Holger
Inconsistency of results <ul><li>Look for explanation </li></ul><ul><ul><li>patients, intervention, comparator, outcome, m...
Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ.  “ Anticoagulation for the long term treatment of venous...
Non-steroidal drug use and risk of pancreatic cancer Capurso G, Sch ünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Ca...
Example: Thromboprophylaxis and bleeding in the hospital Dentali et al. Ann Int Med, 2007
Questions about inconsistency of results?
Indirectness of evidence Victor
Indirectness of evidence <ul><li>Question of interest </li></ul><ul><li>Relative effectiveness of alendronate and risedron...
Indirectness of evidence <ul><li>Question of interest </li></ul><ul><li>Rosiglitazone for prevention of diabetic complicat...
Indirectness of evidence <ul><li>Question of interest </li></ul><ul><li>Oseltamivir for prophylaxis of avian flu caused by...
Indirectness of evidence <ul><li>Guidelines for the treatment of osteogenesis imperfecta in adults </li></ul><ul><li>In pa...
Questions about indirectness of evidence?
Imprecise results Victor
How to decide if the evidence about a particular outcomes is imprecise <ul><li>1. If the evidence were to come from a sing...
 
How to decide if the evidence about a particular outcomes is imprecise <ul><ul><li>Optimal information size </li></ul></ul...
Stroke – periop beta blockers
Offer all effective treatments? <ul><li>atrial fib at risk of stroke </li></ul><ul><li>warfarin increases serious gi bleed...
0 1.0%
0 1.0%
0 1.0%
0 0.5% 1.0%
0 0.5% 1.0%
0 1.0%
0 1.0%
How to decide if the evidence about a particular outcomes is imprecise <ul><ul><li>Optimal information size </li></ul></ul...
Rate down for imprecision? Rate down for imprecision for guidelines?
Bleeding in patients with cancer receiving heparing for prolonging survival Rate down for imprecision for guidelines?
How to decide if the evidence is imprecise for a recommendation <ul><ul><li>300 events </li></ul></ul><ul><ul><li>Evaluate...
Questions about imprecise results?
Publication bias Victor
<ul><li>In 102 Danish protocols|122 reports in 2003 </li></ul><ul><li>50% of outcomes were incompletely reported </li></ul...
Furukawa TA et al.  JAMA 2007; 297: 468-9  1 10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 <20% 20-39% 40-59% 60-80% >80% Continuous ...
<ul><li>Perioperative beta-blocker </li></ul><ul><li>Positive </li></ul><ul><li>Poldermans (NEJM, n=112), RRR 80%  </li></...
 
 
 
Publication bias <ul><li>Reporting of outcomes </li></ul><ul><ul><li>Selective outcome reporting </li></ul></ul><ul><li>Re...
Questions about publication bias?
Factors that can raise the quality of evidence Andy
What can raise the quality of evidence? <ul><li>Large magnitude of effect  </li></ul><ul><li>A dose response relationship ...
Large magnitude of effect <ul><li>Rules of thumb based on simulation studies </li></ul><ul><ul><li>RR < 0.5 </li></ul></ul...
What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a large...
 
When do strong associations provide compelling evidence of effect (causation)? <ul><li>Some interventions have such dramat...
Dose response relationship <ul><li>Risk of bleeding in patients who have supratherapeutic anticoagulation levels  </li></u...
What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a dose ...
All plausible confounding  would result in an underestimate of the treatment effect <ul><li>Higher death rates in private ...
All plausible biases  would result in an overestimate of effect <ul><li>Hypoglycaemic drug phenformin causes lactic acidos...
What are examples of interventions you can think of for which you might upgrade the quality of evidence  when an effect (a...
Questions about factors that can raise the quality of evidence?
Judgements about the overall quality of evidence Andy
Quality assessment criteria
Judgements about the overall quality of evidence <ul><li>Most systems not explicit </li></ul><ul><li>Options: </li></ul><u...
Flavanoids for Hemorrhoids <ul><li>venotonic agents </li></ul><ul><ul><li>mechanism unclear, increase venous return </li><...
Systematic Review <ul><li>14 trials, 1432 patients </li></ul><ul><li>Key outcome </li></ul><ul><ul><li>risk not improving/...
What can lower the quality of evidence? <ul><li>Study limitations </li></ul><ul><ul><li>lack of detail re concealment </li...
 
Publication bias? <ul><li>Size of studies </li></ul><ul><ul><li>40 to 234 patients, most around 100 </li></ul></ul><ul><li...
 
What is the overall quality of evidence across criteria? <ul><li>High : Further research is very unlikely to change our co...
Threshold for downgrading highest lowest OK rate down quality
Overall quality across criteria highest lowest high moderate low very  low
Is France right? <ul><li>Recommend </li></ul><ul><ul><li>for or against? </li></ul></ul><ul><li>Strength of recommendation...
Questions about judgements of the overall quality of evidence?
Workshop agenda Monday, 12 May 2008 <ul><li>08.15 – 13.30 h - Plenary </li></ul><ul><li>Introduction to GRADE </li></ul><u...
Should routine HPV vaccination  be recommended for  12 year old girls in Spain? Introduction to small group work
Questions about the small group work?
Grading the quality of evidence and the strength of recommendations San Sebastián 13 May 2008 Pablo Alonso, Victor Montori...
Workshop agenda Tuesday, 13 May 2008 Strength of recommendation <ul><li>08.30 – 13.30 h - Plenary </li></ul><ul><li>Going ...
Any questions from yesterday about quality of evidence  or about the agenda for today?
Summarising and presenting findings Andy
Questions about summarising and presenting findings?
Strength of recommendations Holger
Strength of recommendation <ul><li>The degree of confidence that the desirable effects of adherence to a recommendation ou...
Categories of recommendations <ul><li>Although the degree of confidence is a continuum, we suggest using two categories: s...
Determinants of strength of recommendation   Factors Impact on the strength of a recommendation Balance between desirable ...
Judgements about the strength of a recommendation <ul><li>No precise threshold for going from a strong to a weak recommend...
Implications of a strong recommendation <ul><li>Patients : Most people in your situation would want the recommended course...
Implications of a weak recommendation <ul><li>Patients : The majority of people in your situation would want the recommend...
Is France right? <ul><li>Recommend </li></ul><ul><ul><li>for or against? </li></ul></ul><ul><li>Strength of recommendation...
Another example   Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? <ul><li>S...
Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? Summary of findings Transmi...
What would you recommend? <ul><li>Strong recommendation : the panel is confident that the desirable effects of adherence t...
Judgments about the strength of a recommendation -  oseltamivir for treatment of patients hospitalised with avian influenz...
Another example   Should aspirin be given to children with fever? <ul><li>Very low quality evidence of increased risk of R...
Questions about  the strength of recommendations?
Wording of recommendations ??
Questions about  the wording of recommendations?
Recommendations to use an intervention only in the context of research Andy
Possible recommendations when there is very low quality evidence include: <ul><li> Recommend using an intervention </li>...
Recommendations to use an intervention only in the context of research may be appropriate <ul><li>Two conditions must be m...
Examples <ul><li>National Institute of Health and Clinical Excellence (NICE)  </li></ul><ul><ul><li>Of its first 95 techno...
Implications for research <ul><li>Core elements </li></ul><ul><li>E Evidence (What is the current state of the evidence?) ...
What are examples of interventions you can think of for which you might recommend that it should only be used in the conte...
Questions about  recommendations to use an intervention only in the context of research?
Judgements about priorities (importance) Andy
Priority setting: From a health system or public health perspective   A  strong  recommendation may not be  important
Judgements about importance may depend on <ul><li>Strength   of the recommendations  </li></ul><ul><li>Severity   of the p...
From the perspective of individual patients and clinicians <ul><li>Most patients would want and should receive  strongly r...
Questions about  judgements about priorities (importance)?
Workshop agenda Tuesday, 13 May 2008 Strength of recommendation <ul><li>08.30 – 13.30 h - Plenary </li></ul><ul><li>Going ...
Upcoming SlideShare
Loading in...5
×

Grade San Sebastian 2008 05 12hjs

393

Published on

Published in: Health & Medicine, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
393
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Grade San Sebastian 2008 05 12hjs"

  1. 1. Grading the quality of evidence and the strength of recommendations San Sebastián 12 - 14 May 2008 Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann
  2. 2. Workshop agenda Monday, 12 May 2008 <ul><li>08.15 – 13.30 h - Plenary </li></ul><ul><li>Introduction to GRADE </li></ul><ul><li>Grading the quality of evidence for each important outcome </li></ul><ul><ul><li>Deciding which outcomes are important </li></ul></ul><ul><ul><li>Factors that can lower the quality of evidence </li></ul></ul><ul><ul><li>Factors that can raise the quality of evidence </li></ul></ul><ul><li>Judgements about the overall quality of evidence </li></ul><ul><li>15.00 – 17.00 h - Small groups </li></ul><ul><li>The quality of evidence for HPV vaccination </li></ul><ul><li>17.00 – 18.00 h - Plenary </li></ul><ul><li>Discussion </li></ul>
  3. 3. Introduction Andy
  4. 4. Why bother about grading? <ul><li>People draw conclusions about the </li></ul><ul><ul><li>quality of evidence </li></ul></ul><ul><ul><li>strength of recommendations </li></ul></ul><ul><li>Systematic and explicit approaches can help </li></ul><ul><ul><li>protect against errors </li></ul></ul><ul><ul><li>resolve disagreements </li></ul></ul><ul><ul><li>facilitate critical appraisal </li></ul></ul><ul><ul><li>communicate information </li></ul></ul><ul><li>However, there is wide variation in currently used approaches </li></ul>
  5. 5. Many grading systems <ul><li>Evidence Recommendation </li></ul><ul><li>B Class I </li></ul><ul><li>C+ 1 </li></ul><ul><li>IV C </li></ul><ul><li>Organization </li></ul><ul><li>AHA </li></ul><ul><li>ACCP </li></ul><ul><li>SIGN </li></ul>Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease <ul><li>Australian NMRC </li></ul><ul><li>Oxford Center for Evidence-based Medicine </li></ul><ul><li>Scottish Intercollegiate Guidelines (SIGN) </li></ul><ul><li>US Preventative Services Task Force </li></ul><ul><li>Professional organizations </li></ul><ul><ul><li>AHA/ACC, ACCP, AAP, Endocrine society, etc.... </li></ul></ul><ul><li>Lots of confusion and dismay </li></ul>
  6. 6. A common international grading system? <ul><li>International group </li></ul><ul><ul><li>ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC, CDC, CC, NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, WHO </li></ul></ul><ul><li>> 60 contributors </li></ul><ul><ul><li>methodologists, guideline developers, systematic reviewers, researchers, clinicians, editors </li></ul></ul><ul><li>~ 20 meetings over last seven years </li></ul><ul><ul><li>~10 – 40 participants </li></ul></ul>www.gradeworkinggroup.org
  7. 7. <ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Agency for Health Care Research and Quality (AHRQ) </li></ul><ul><li>Allergic Rhinitis in Asthma Guidelines </li></ul><ul><li>American College of Chest Physicians </li></ul><ul><li>American Endocrine Society </li></ul><ul><li>American Thoracic Society </li></ul><ul><li>BMJ Clinical Evidence </li></ul><ul><li>British Medical Journal </li></ul><ul><li>Canadian Agency for Drugs and Technology in Health </li></ul><ul><li>Cochrane Collaboration </li></ul><ul><li>EBM Guidelines Finland </li></ul><ul><li>European Respiratory Society </li></ul><ul><li>European Society of Thoracic Surgeons </li></ul><ul><li>Evidence-based Nursing Sudtirol, Alta Adiga, Italy </li></ul><ul><li>Journal of Infection in Developing Countries - International </li></ul><ul><li>Kidney disease: Improving global outcome </li></ul><ul><li>National Institute for Clinical Excellence (NICE) </li></ul><ul><li>Norwegian Knowledge Centre for the Health Services </li></ul><ul><li>Ontario MOH Medical Advisory Secretariat </li></ul><ul><li>Polish Institute for EBM </li></ul><ul><li>Society of Critical Care Medicine </li></ul><ul><li>Society of Pediatric Endocrinology </li></ul><ul><li>Society of Vascular Surgery </li></ul><ul><li>Surviving sepsis campaign </li></ul><ul><li>The German Agency for Quality in Medicine </li></ul><ul><li>University of Pennsylvania Health System Center for EB Practice </li></ul><ul><li>UpToDate </li></ul><ul><li>World Health Organization </li></ul>GRADE Uptake
  8. 8. <ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Agency for Health Care Research and Quality (AHRQ) </li></ul><ul><li>Allergic Rhinitis in Asthma Guidelines </li></ul><ul><li>American College of Chest Physicians </li></ul><ul><li>American Endocrine Society </li></ul><ul><li>American Thoracic Society </li></ul><ul><li>BMJ Clinical Evidence </li></ul><ul><li>British Medical Journal </li></ul><ul><li>Canadian Agency for Drugs and Technology in Health </li></ul><ul><li>Cochrane Collaboration </li></ul><ul><li>EBM Guidelines Finland </li></ul><ul><li>European Respiratory Society </li></ul><ul><li>European Society of Thoracic Surgeons </li></ul><ul><li>Evidence-based Nursing Sudtirol, Alta Adiga, Italy </li></ul><ul><li>Journal of Infection in Developing Countries - International </li></ul><ul><li>Kidney disease: Improving global outcome </li></ul><ul><li>National Institute for Clinical Excellence (NICE) </li></ul><ul><li>Norwegian Knowledge Centre for the Health Services </li></ul><ul><li>Ontario MOH Medical Advisory Secretariat </li></ul><ul><li>Polish Institute for EBM </li></ul><ul><li>Society of Critical Care Medicine </li></ul><ul><li>Society of Pediatric Endocrinology </li></ul><ul><li>Society of Vascular Surgery </li></ul><ul><li>Surviving sepsis campaign </li></ul><ul><li>The German Agency for Quality in Medicine </li></ul><ul><li>University of Pennsylvania Health System Center for EB Practice </li></ul><ul><li>UpToDate </li></ul><ul><li>World Health Organization </li></ul>GRADE Uptake
  9. 9. <ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Agency for Health Care Research and Quality (AHRQ) </li></ul><ul><li>Allergic Rhinitis in Asthma Guidelines </li></ul><ul><li>American College of Chest Physicians </li></ul><ul><li>American Endocrine Society </li></ul><ul><li>American Thoracic Society </li></ul><ul><li>BMJ Clinical Evidence </li></ul><ul><li>British Medical Journal </li></ul><ul><li>Canadian Agency for Drugs and Technology in Health </li></ul><ul><li>Cochrane Collaboration </li></ul><ul><li>EBM Guidelines Finland </li></ul><ul><li>European Respiratory Society </li></ul><ul><li>European Society of Thoracic Surgeons </li></ul><ul><li>Evidence-based Nursing Sudtirol, Alta Adiga, Italy </li></ul><ul><li>Journal of Infection in Developing Countries - International </li></ul><ul><li>Kidney disease: Improving global outcome </li></ul><ul><li>National Institute for Clinical Excellence (NICE) </li></ul><ul><li>Norwegian Knowledge Centre for the Health Services </li></ul><ul><li>Ontario MOH Medical Advisory Secretariat </li></ul><ul><li>Polish Institute for EBM </li></ul><ul><li>Society of Critical Care Medicine </li></ul><ul><li>Society of Pediatric Endocrinology </li></ul><ul><li>Society of Vascular Surgery </li></ul><ul><li>Surviving sepsis campaign </li></ul><ul><li>The German Agency for Quality in Medicine </li></ul><ul><li>University of Pennsylvania Health System Center for EB Practice </li></ul><ul><li>UpToDate </li></ul><ul><li>World Health Organization </li></ul>GRADE Uptake
  10. 10. <ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Agency for Health Care Research and Quality (AHRQ) </li></ul><ul><li>Allergic Rhinitis in Asthma Guidelines </li></ul><ul><li>American College of Chest Physicians </li></ul><ul><li>American Endocrine Society </li></ul><ul><li>American Thoracic Society </li></ul><ul><li>BMJ Clinical Evidence </li></ul><ul><li>British Medical Journal </li></ul><ul><li>Canadian Agency for Drugs and Technology in Health </li></ul><ul><li>Cochrane Collaboration </li></ul><ul><li>EBM Guidelines Finland </li></ul><ul><li>European Respiratory Society </li></ul><ul><li>European Society of Thoracic Surgeons </li></ul><ul><li>Evidence-based Nursing Sudtirol, Alta Adiga, Italy </li></ul><ul><li>Journal of Infection in Developing Countries - International </li></ul><ul><li>Kidney disease: Improving global outcome </li></ul><ul><li>National Institute for Clinical Excellence (NICE) </li></ul><ul><li>Norwegian Knowledge Centre for the Health Services </li></ul><ul><li>Ontario MOH Medical Advisory Secretariat </li></ul><ul><li>Polish Institute for EBM </li></ul><ul><li>Society of Critical Care Medicine </li></ul><ul><li>Society of Pediatric Endocrinology </li></ul><ul><li>Society of Vascular Surgery </li></ul><ul><li>Surviving sepsis campaign </li></ul><ul><li>The German Agency for Quality in Medicine </li></ul><ul><li>University of Pennsylvania Health System Center for EB Practice </li></ul><ul><li>UpToDate </li></ul><ul><li>World Health Organization </li></ul>GRADE Uptake
  11. 11. <ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Agency for Health Care Research and Quality (AHRQ) </li></ul><ul><li>Allergic Rhinitis in Asthma Guidelines </li></ul><ul><li>American College of Chest Physicians </li></ul><ul><li>American Endocrine Society </li></ul><ul><li>American Thoracic Society </li></ul><ul><li>BMJ Clinical Evidence </li></ul><ul><li>British Medical Journal </li></ul><ul><li>Canadian Agency for Drugs and Technology in Health </li></ul><ul><li>Cochrane Collaboration </li></ul><ul><li>EBM Guidelines Finland </li></ul><ul><li>European Respiratory Society </li></ul><ul><li>European Society of Thoracic Surgeons </li></ul><ul><li>Evidence-based Nursing Sudtirol, Alta Adiga, Italy </li></ul><ul><li>Journal of Infection in Developing Countries - International </li></ul><ul><li>Kidney disease: Improving global outcome </li></ul><ul><li>National Institute for Clinical Excellence (NICE) </li></ul><ul><li>Norwegian Knowledge Centre for the Health Services </li></ul><ul><li>Ontario MOH Medical Advisory Secretariat </li></ul><ul><li>Polish Institute for EBM </li></ul><ul><li>Society of Critical Care Medicine </li></ul><ul><li>Society of Pediatric Endocrinology </li></ul><ul><li>Society of Vascular Surgery </li></ul><ul><li>Surviving sepsis campaign </li></ul><ul><li>The German Agency for Quality in Medicine </li></ul><ul><li>University of Pennsylvania Health System Center for EB Practice </li></ul><ul><li>UpToDate </li></ul><ul><li>World Health Organization </li></ul>GRADE Uptake
  12. 12. <ul><li>American College of Physicians </li></ul><ul><li>Agencia sanitaria regionale, Bologna, Italia </li></ul><ul><li>Agency for Health Care Research and Quality (AHRQ) </li></ul><ul><li>Allergic Rhinitis in Asthma Guidelines </li></ul><ul><li>American College of Chest Physicians </li></ul><ul><li>American Endocrine Society </li></ul><ul><li>American Thoracic Society </li></ul><ul><li>BMJ Clinical Evidence </li></ul><ul><li>British Medical Journal </li></ul><ul><li>Canadian Agency for Drugs and Technology in Health </li></ul><ul><li>Cochrane Collaboration </li></ul><ul><li>EBM Guidelines Finland </li></ul><ul><li>European Respiratory Society </li></ul><ul><li>European Society of Thoracic Surgeons </li></ul><ul><li>Evidence-based Nursing Sudtirol, Alta Adiga, Italy </li></ul><ul><li>Journal of Infection in Developing Countries - International </li></ul><ul><li>Kidney disease: Improving global outcome </li></ul><ul><li>National Institute for Clinical Excellence (NICE) </li></ul><ul><li>Norwegian Knowledge Centre for the Health Services </li></ul><ul><li>Ontario MOH Medical Advisory Secretariat </li></ul><ul><li>Polish Institute for EBM </li></ul><ul><li>Society of Critical Care Medicine </li></ul><ul><li>Society of Pediatric Endocrinology </li></ul><ul><li>Society of Vascular Surgery </li></ul><ul><li>Surviving sepsis campaign </li></ul><ul><li>The German Agency for Quality in Medicine </li></ul><ul><li>University of Pennsylvania Health System Center for EB Practice </li></ul><ul><li>UpToDate </li></ul><ul><li>World Health Organization </li></ul>GRADE Uptake
  13. 13. What is quality of evidence and how should it be graded?
  14. 15. Judging the quality of evidence requires considering the context <ul><li>In the context of making recommendations </li></ul><ul><li>The quality of evidence reflects the extent to which our confidence in an estimate of the effect is adequate to support a particular recommendation. </li></ul><ul><li>In the context of a systematic review </li></ul><ul><li>The quality of evidence reflects the extent to which we are confident that an estimate of effect is correct. </li></ul>
  15. 16. Although the degree of confidence is a continuum, we suggest using four categories <ul><li>High </li></ul><ul><li>Moderate </li></ul><ul><li>Low </li></ul><ul><li>Very low </li></ul>
  16. 17. Categories of quality <ul><li>High : Further research is very unlikely to change our confidence in the estimate of effect. </li></ul><ul><li>Moderate : Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. </li></ul><ul><li>Low : Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. </li></ul><ul><li>Very low : Any estimate of effect is very uncertain. </li></ul>
  17. 18. Study design is important <ul><li>Early systems of grading the quality of evidence focused almost exclusively on study design </li></ul><ul><li>Randomised trials provide, in general, far stronger evidence than observational studies. </li></ul><ul><ul><li>Randomised trials start out at High </li></ul></ul><ul><ul><li>Observational studies start out at Low </li></ul></ul><ul><li>However, other factors may decrease or increase the quality of evidence </li></ul>
  18. 19. What can lower the quality of evidence? <ul><li>Study limitations </li></ul><ul><li>Inconsistency of results </li></ul><ul><li>Indirectness of evidence </li></ul><ul><li>Imprecise results </li></ul><ul><li>Publication bias </li></ul>
  19. 20. What can raise the quality of evidence? <ul><li>Large magnitude of effect </li></ul><ul><li>A dose response relationship </li></ul><ul><li>All plausible confounding </li></ul>
  20. 21. Quality assessment criteria
  21. 22. Strength of recommendation <ul><li>The degree of confidence that the desirable effects of adherence to a recommendation outweigh the undesirable effects. </li></ul><ul><li>Desirable effects </li></ul><ul><li>health benefits </li></ul><ul><li>less burden </li></ul><ul><li>savings </li></ul><ul><li>Undesirable effects </li></ul><ul><li>harms </li></ul><ul><li>more burden </li></ul><ul><li>costs </li></ul>
  22. 23. Categories of recommendations <ul><li>Although the degree of confidence is a continuum, we suggest using two categories: strong and weak. </li></ul><ul><li>Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects. </li></ul><ul><li>Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident. </li></ul>Recommend   Suggest  
  23. 24. Determinants of strength of recommendation Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.
  24. 25. Workshop agenda Monday, 12 May 2008 Quality of evidence <ul><li>08.15 – 13.30 h - Plenary </li></ul><ul><li>Introduction to GRADE </li></ul><ul><li>Grading the quality of evidence for each important outcome </li></ul><ul><ul><li>Deciding which outcomes are important </li></ul></ul><ul><ul><li>Factors that can lower the quality of evidence </li></ul></ul><ul><ul><li>Factors that can raise the quality of evidence </li></ul></ul><ul><li>Judgements about the overall quality of evidence </li></ul><ul><li>15.00 – 17.00 h - Small groups </li></ul><ul><li>The quality of evidence for HPV vaccination </li></ul><ul><li>17.00 – 18.00 h - Plenary </li></ul><ul><li>Discussion </li></ul>
  25. 26. Workshop agenda Tuesday, 13 May 2008 Strength of recommendation <ul><li>08.30 – 13.30 h - Plenary </li></ul><ul><li>Going from evidence to recommendations </li></ul><ul><ul><li>Summarising and presenting findings </li></ul></ul><ul><ul><li>Factors that determine the strength of a recommendation </li></ul></ul><ul><ul><ul><li>Balance of benefits and harms </li></ul></ul></ul><ul><ul><ul><li>(Quality of evidence) </li></ul></ul></ul><ul><ul><ul><li>Values and preferences </li></ul></ul></ul><ul><ul><ul><li>Costs (resource use) </li></ul></ul></ul><ul><ul><li>Judgements about the strength of recommendations </li></ul></ul><ul><ul><li>Wording of recommendations </li></ul></ul><ul><li>15.00 – 17.00 h - Small groups </li></ul><ul><li>Summary of findings and the strength of recommendation for HPV vaccination </li></ul><ul><li>17.00 – 18.00 h - Plenary </li></ul><ul><li>Discussion </li></ul>
  26. 27. Workshop agenda Wednesday, 14 May 2008 Special challenges <ul><li>08.30 – 11.30 h – Plenary + small group work </li></ul><ul><li>Economic evidence (resource use) </li></ul><ul><li>13.00 – 17.00 h - Plenary + small group work </li></ul><ul><li>Diagnostic tests </li></ul>
  27. 28. General questions about GRADE, the GRADE Working Group or the agenda for this workshop?
  28. 29. Relative importance of outcomes Holger
  29. 30. The quality of evidence needs to be considered for each important outcome <ul><li>The quality of evidence may be different for different outcomes </li></ul><ul><li>Decision makers (and review authors) need to consider the relative importance of outcomes </li></ul>
  30. 31. Hierarchy of outcomes according to their importance to assess the effect of phosphate lowering drugs in patients with renal failure and hyperphosphatemia Flatulence 2 Importance of endpoints 5 Pain due to soft tissue 6 calcification / function Fractures 7 Myocardial infarction 8 Mortality 9 3 4 1 Critical for decision making Important, but not critical for decision making Of low importance
  31. 32. A different clinical question <ul><li>Population: In patients with chronic atrial fibrillation and no prior history of stroke </li></ul><ul><li>Intervention: does oral anticoagulation </li></ul><ul><li>(comparison) compared with no therapy </li></ul><ul><li>Outcome: reduce the risk for </li></ul><ul><li>embolic stroke, hemorrhage and death? </li></ul>
  32. 33. <ul><li>Population: In patients with chronic atrial fibrillation and no prior history of stroke </li></ul><ul><li>Intervention: does oral anticoagulation </li></ul><ul><li>(comparison) compared with no therapy </li></ul><ul><li>Outcome: reduce the risk for </li></ul><ul><li>embolic stroke, hemorrhage and death? </li></ul>Other outcomes: Inconvenience, quality of life
  33. 34. <ul><li>All disabling or fatal stroke (isch. and hemorrh.) </li></ul><ul><li>Major hemorrhage (non IC) </li></ul><ul><li>All cause mortality </li></ul><ul><li>Health related quality of life </li></ul><ul><li>*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005, Issue 3. </li></ul>Outcomes/endpoints
  34. 35. Outcomes/endpoints <ul><li>How important is the endpoint for decision making? </li></ul><ul><li>Judgment about the relative importance for each endpoint on a scale from 9 (most important) to 1 (least important): </li></ul><ul><li>7 – 9: the endpoint is critical for decision making. </li></ul><ul><li>4 – 6: the endpoint is important but not critical for decision making. </li></ul><ul><li>1 – 3: the endpoint is not important . </li></ul>
  35. 36. 2 Importance of endpoints 5 6 7 8 9 3 4 1 Critical for decision making Important, but not critical for decision making Of low importance
  36. 37. <ul><li>All disabling or fatal stroke (isch. and hemorrh.) </li></ul><ul><li>Major hemorrhage (non IC) </li></ul><ul><li>All cause mortality </li></ul><ul><li>Health related quality of life </li></ul><ul><li>*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005, Issue 3. </li></ul>Outcomes/endpoints 9 8 9 7
  37. 38. Questions about the relative importance of outcomes?
  38. 39. Study limitations Victor
  39. 40. Study limitations (Risk of bias) for Randomized trials <ul><li>Lack of allocation concealment </li></ul><ul><li>Lack of blinding </li></ul><ul><li>Incomplete accounting of patients and outcome events </li></ul><ul><li>Selective outcome reporting </li></ul><ul><li>Other limitations </li></ul><ul><ul><li>e.g. stopping early for benefit; use of unvalidated patient-reported outcomes; carry-over effects in cross-over trials; recruitment bias in cluster-randomized trials </li></ul></ul>
  40. 41. Testosterone for BMD in men
  41. 42. Assessment of the risk of bias Cochrane Handbook for Systematic Reviews of Interventions, Draft chapter of version 5 - Assessing risk of bias in included studies http://www.cochrane.org/resources/handbook/handbook5drafts.htm
  42. 43. Should patients with cancer receive parenteral anticoagulation (heparin) to prolong survival. Outcomes: Survival, bleeding
  43. 44. Survival <ul><li>HR 0.77 (0.65 to 0.91) </li></ul>
  44. 45. <ul><li>How confident are you that these results are true? </li></ul>
  45. 46. Factors that lower quality <ul><li>detailed design and execution </li></ul><ul><li>inconsistency </li></ul><ul><li>indirectness </li></ul><ul><li>reporting bias </li></ul><ul><li>imprecision </li></ul>
  46. 47. Detailed study design and execution
  47. 48. Quality now? <ul><li>High </li></ul>
  48. 49. Major bleeding – start high <ul><li>RR 1.50 (0.26 – 8.80) </li></ul>
  49. 50. Detailed study design and execution
  50. 51. Quality now? <ul><li>Moderate </li></ul>
  51. 52. Bleeding in patients with cancer receiving heparing for prolonging survival
  52. 53. Quality now? <ul><li>Low </li></ul><ul><li>Observational studies could have provided higher quality evidence </li></ul>
  53. 54. Summarizing study limitations for randomized trials
  54. 55. Questions about study limitations?
  55. 56. Inconsistency of results Holger
  56. 57. Inconsistency of results <ul><li>Look for explanation </li></ul><ul><ul><li>patients, intervention, comparator, outcome, methods </li></ul></ul><ul><li>Judgment </li></ul><ul><ul><li>variation in size of effect </li></ul></ul><ul><ul><li>overlap in confidence intervals </li></ul></ul><ul><ul><li>statistical significance of heterogeneity </li></ul></ul><ul><ul><li>I 2 </li></ul></ul>
  57. 58. Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “ Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer ”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650. Heparin or vitamin K antagonists for survival in patients with cancer
  58. 59. Non-steroidal drug use and risk of pancreatic cancer Capurso G, Sch ünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories. Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.
  59. 60. Example: Thromboprophylaxis and bleeding in the hospital Dentali et al. Ann Int Med, 2007
  60. 61. Questions about inconsistency of results?
  61. 62. Indirectness of evidence Victor
  62. 63. Indirectness of evidence <ul><li>Question of interest </li></ul><ul><li>Relative effectiveness of alendronate and risedronate in osteoporosis </li></ul><ul><li>Source of indirectness </li></ul><ul><li>Indirect comparison : Randomized trials have compared alendronate to placebo and risedronate to placebo, but trials comparing alendronate to risedronate are unavailable. </li></ul>
  63. 64. Indirectness of evidence <ul><li>Question of interest </li></ul><ul><li>Rosiglitazone for prevention of diabetic complications in patients at high risk of developing diabetes </li></ul><ul><li>Source of indirectness </li></ul><ul><li>Differences in Outcome : A randomized trial shows delay in the development of biochemical diabetes with rosiglitazone, but was underpowered to address diabetic complications </li></ul>
  64. 65. Indirectness of evidence <ul><li>Question of interest </li></ul><ul><li>Oseltamivir for prophylaxis of avian flu caused by influenza A(H5N1) virus </li></ul><ul><li>Sigmoidoscopic screening for prevention of colon cancer mortality </li></ul><ul><li>Choice of medication for schizophrenia </li></ul><ul><li>Source of indirectness </li></ul><ul><li>Differences in Population : randomized trials of oseltamivir are available for seasonal influenza, but not for avian flu </li></ul><ul><li>Differences in Intervention : randomized trials of fecal occult blood screening provide indirect evidence bearing on the potential effectiveness of sigmoidoscopy </li></ul><ul><li>Differences in Comparison : A series of trials comparing newer generation neuroleptic agents to fixed doses of 20 mg. of haloperidol provide indirect evidence of how the newer agents would compare to the lower, flexible doses of haloperidol clinicians typically use </li></ul>
  65. 66. Indirectness of evidence <ul><li>Guidelines for the treatment of osteogenesis imperfecta in adults </li></ul><ul><li>In patients with osteoporosis, bisphosphonates, either oral or IV, increase BMD and reduce the risk of fragility fracture by 50% </li></ul><ul><li>RCT oral bisphosphonates showed improved BMD in patients with OI </li></ul><ul><li>RCT IV bisphosphonates showed improved cortical thickness in patients with OI </li></ul>
  66. 67. Questions about indirectness of evidence?
  67. 68. Imprecise results Victor
  68. 69. How to decide if the evidence about a particular outcomes is imprecise <ul><li>1. If the evidence were to come from a single RCT, how big should this RCT need to be to estimate the treatment effect? </li></ul><ul><ul><li>Optimal information size </li></ul></ul>
  69. 71. How to decide if the evidence about a particular outcomes is imprecise <ul><ul><li>Optimal information size </li></ul></ul><ul><ul><li>300 events </li></ul></ul>
  70. 72. Stroke – periop beta blockers
  71. 73. Offer all effective treatments? <ul><li>atrial fib at risk of stroke </li></ul><ul><li>warfarin increases serious gi bleeding </li></ul><ul><ul><li>3% per year </li></ul></ul><ul><li>1,000 patients 1 less stroke </li></ul><ul><ul><li>30 more bleeds for each stroke prevented </li></ul></ul><ul><li>1,000 patients 100 less strokes </li></ul><ul><ul><li>3 strokes prevented for each bleed </li></ul></ul><ul><li>where is your threshold? </li></ul><ul><ul><li>how many strokes in 100 with 3% bleeding? </li></ul></ul>
  72. 74. 0 1.0%
  73. 75. 0 1.0%
  74. 76. 0 1.0%
  75. 77. 0 0.5% 1.0%
  76. 78. 0 0.5% 1.0%
  77. 79. 0 1.0%
  78. 80. 0 1.0%
  79. 81. How to decide if the evidence about a particular outcomes is imprecise <ul><ul><li>Optimal information size </li></ul></ul><ul><ul><li>300 events </li></ul></ul><ul><ul><li>Evaluate the boundaries of the CI </li></ul></ul><ul><ul><ul><li>(if low event rates – CI for risk diff) </li></ul></ul></ul>
  80. 82. Rate down for imprecision? Rate down for imprecision for guidelines?
  81. 83. Bleeding in patients with cancer receiving heparing for prolonging survival Rate down for imprecision for guidelines?
  82. 84. How to decide if the evidence is imprecise for a recommendation <ul><ul><li>300 events </li></ul></ul><ul><ul><li>Evaluate the boundaries of the CI in light of other important outcomes </li></ul></ul>
  83. 85. Questions about imprecise results?
  84. 86. Publication bias Victor
  85. 87. <ul><li>In 102 Danish protocols|122 reports in 2003 </li></ul><ul><li>50% of outcomes were incompletely reported </li></ul><ul><li>Odds of full reporting: </li></ul><ul><li>2:1 </li></ul><ul><li>if the outcome was significant </li></ul>JAMA. 2004;291:2457-2465.
  86. 88. Furukawa TA et al. JAMA 2007; 297: 468-9 1 10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 <20% 20-39% 40-59% 60-80% >80% Continuous outcomes Dichotomous outcomes % eligible trials contributing data to meta-analysis Effect size Odds ratio
  87. 89. <ul><li>Perioperative beta-blocker </li></ul><ul><li>Positive </li></ul><ul><li>Poldermans (NEJM, n=112), RRR 80% </li></ul><ul><li>Negative </li></ul><ul><li>DIPOM (BMJ, n=921), RRI: 3% </li></ul><ul><li>POBBLE (J Vasc Surgery, n=103), RRI = 6% </li></ul><ul><li>MAVS (Am Heart J, n=496), RRR = 15% </li></ul>Statins for DM2 Positive CARDS (Lancet, 2004), RRR = 37% Negative ASPEN (Diabetes Care, 2006), RRR = 10%
  88. 93. Publication bias <ul><li>Reporting of outcomes </li></ul><ul><ul><li>Selective outcome reporting </li></ul></ul><ul><li>Reporting of studies </li></ul><ul><ul><li>Publication bias </li></ul></ul><ul><ul><ul><li>number of small studies </li></ul></ul></ul><ul><ul><ul><li>industry sponsored </li></ul></ul></ul><ul><li>High likelihood could lower quality </li></ul>
  89. 94. Questions about publication bias?
  90. 95. Factors that can raise the quality of evidence Andy
  91. 96. What can raise the quality of evidence? <ul><li>Large magnitude of effect </li></ul><ul><li>A dose response relationship </li></ul><ul><li>All plausible confounding </li></ul>
  92. 97. Large magnitude of effect <ul><li>Rules of thumb based on simulation studies </li></ul><ul><ul><li>RR < 0.5 </li></ul></ul><ul><ul><li>RR < 0.2 </li></ul></ul><ul><li>More compelling if: </li></ul><ul><ul><li>consistency of effect </li></ul></ul><ul><ul><ul><li>almost everyone used to do badly, now does well </li></ul></ul></ul><ul><ul><li>quick action </li></ul></ul><ul><ul><ul><li>reversal of trajectory </li></ul></ul></ul><ul><ul><li>strong indirect evidence </li></ul></ul><ul><li>Examples </li></ul><ul><ul><li>insulin for diabetic ketoacidosis </li></ul></ul><ul><ul><li>hip replacement for severe osteoarthritis </li></ul></ul>
  93. 98. What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a large magnitude of effect? Why would you or would you not upgrade the quality of evidence?
  94. 100. When do strong associations provide compelling evidence of effect (causation)? <ul><li>Some interventions have such dramatic effects that biases can be ruled out without randomised trials </li></ul><ul><li>Dramatic effects can be defined by the size of the treatment effect (signal) relative to the expected prognosis (noise) </li></ul><ul><li>Real treatment effects are likely if the signal to noise ratio is large (above 10) </li></ul><ul><li>Large ratios may be due to </li></ul><ul><ul><li>the high proportion of patients improved (consistency of effect) or </li></ul></ul><ul><ul><li>the rapidity of improvement (quick action) </li></ul></ul><ul><li>And are more compelling if they are “plausible” (strong indirect evidence) </li></ul>
  95. 101. Dose response relationship <ul><li>Risk of bleeding in patients who have supratherapeutic anticoagulation levels </li></ul><ul><ul><li>Dose response gradient between higher levels of the international normalized ratio (INR) and the increased risk of bleeding </li></ul></ul><ul><li>Risk of death in patients presenting with sepsis and hypotension </li></ul>
  96. 102. What are examples of interventions you can think of for which you might upgrade the quality of evidence because of a dose response relationship?
  97. 103. All plausible confounding would result in an underestimate of the treatment effect <ul><li>Higher death rates in private for-profit versus private not-for-profit hospitals </li></ul><ul><ul><li>patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals </li></ul></ul><ul><ul><li>for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for-profit hospitals (and thus have more resources with a spill over effect) </li></ul></ul>
  98. 104. All plausible biases would result in an overestimate of effect <ul><li>Hypoglycaemic drug phenformin causes lactic acidosis </li></ul><ul><li>The related agent metformin is under suspicion for the same toxicity. </li></ul><ul><li>Large observational studies have failed to demonstrate an association </li></ul><ul><ul><li>Clinicians would be more alert to lactic acidosis in the presence of the agent </li></ul></ul>
  99. 105. What are examples of interventions you can think of for which you might upgrade the quality of evidence when an effect (association) was found - because all plausible confounders would result in an underestimate of effect or when an effect (association) was not found - because all plausible biases would result in an overestimate of effect?
  100. 106. Questions about factors that can raise the quality of evidence?
  101. 107. Judgements about the overall quality of evidence Andy
  102. 108. Quality assessment criteria
  103. 109. Judgements about the overall quality of evidence <ul><li>Most systems not explicit </li></ul><ul><li>Options: </li></ul><ul><ul><li>Benefits </li></ul></ul><ul><ul><li>Primary outcome </li></ul></ul><ul><ul><li>Highest </li></ul></ul><ul><ul><li>Lowest </li></ul></ul><ul><li>Based on lowest of all the critical outcomes </li></ul><ul><li>Beyond the scope of a systematic review </li></ul>
  104. 110. Flavanoids for Hemorrhoids <ul><li>venotonic agents </li></ul><ul><ul><li>mechanism unclear, increase venous return </li></ul></ul><ul><li>popularity </li></ul><ul><ul><li>90 venotonics commercialized in France </li></ul></ul><ul><ul><li>none in Sweden and Norway </li></ul></ul><ul><ul><li>France 70% of world market </li></ul></ul><ul><li>possibilities </li></ul><ul><ul><li>French misguided </li></ul></ul><ul><ul><li>rest of world missing out </li></ul></ul>
  105. 111. Systematic Review <ul><li>14 trials, 1432 patients </li></ul><ul><li>Key outcome </li></ul><ul><ul><li>risk not improving/persistent symptoms </li></ul></ul><ul><ul><li>11 studies, 1002 patients, 375 events </li></ul></ul><ul><ul><li>RR 0.4, 95% CI 0.29 to 0.57 </li></ul></ul><ul><li>Minimal side effects </li></ul><ul><li>Is France right? </li></ul><ul><li>What is the quality of evidence? </li></ul>
  106. 112. What can lower the quality of evidence? <ul><li>Study limitations </li></ul><ul><ul><li>lack of detail re concealment </li></ul></ul><ul><ul><li>questionnaires not validated </li></ul></ul><ul><li>Rate down quality for study limitations? </li></ul><ul><li>Indirectness – no problem </li></ul><ul><li>Inconsistency, need to look at the results </li></ul>
  107. 114. Publication bias? <ul><li>Size of studies </li></ul><ul><ul><li>40 to 234 patients, most around 100 </li></ul></ul><ul><li>All industry sponsored </li></ul>
  108. 116. What is the overall quality of evidence across criteria? <ul><li>High : Further research is very unlikely to change our confidence in the estimate of effect. </li></ul><ul><li>Moderate : Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. </li></ul><ul><li>Low : Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. </li></ul><ul><li>Very low : Any estimate of effect is very uncertain. </li></ul>
  109. 117. Threshold for downgrading highest lowest OK rate down quality
  110. 118. Overall quality across criteria highest lowest high moderate low very low
  111. 119. Is France right? <ul><li>Recommend </li></ul><ul><ul><li>for or against? </li></ul></ul><ul><li>Strength of recommendation </li></ul><ul><ul><li>strong or weak </li></ul></ul>
  112. 120. Questions about judgements of the overall quality of evidence?
  113. 121. Workshop agenda Monday, 12 May 2008 <ul><li>08.15 – 13.30 h - Plenary </li></ul><ul><li>Introduction to GRADE </li></ul><ul><li>Grading the quality of evidence for each important outcome </li></ul><ul><ul><li>Deciding which outcomes are important </li></ul></ul><ul><ul><li>Factors that can lower the quality of evidence </li></ul></ul><ul><ul><li>Factors that can raise the quality of evidence </li></ul></ul><ul><li>Judgements about the overall quality of evidence </li></ul><ul><li>15.00 – 17.00 h - Small groups </li></ul><ul><li>The quality of evidence for HPV vaccination </li></ul><ul><li>17.00 – 18.00 h - Plenary </li></ul><ul><li>Discussion </li></ul>
  114. 122. Should routine HPV vaccination be recommended for 12 year old girls in Spain? Introduction to small group work
  115. 123. Questions about the small group work?
  116. 124. Grading the quality of evidence and the strength of recommendations San Sebastián 13 May 2008 Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann
  117. 125. Workshop agenda Tuesday, 13 May 2008 Strength of recommendation <ul><li>08.30 – 13.30 h - Plenary </li></ul><ul><li>Going from evidence to recommendations </li></ul><ul><ul><li>Summarising and presenting findings </li></ul></ul><ul><ul><li>Factors that determine the strength of a recommendation </li></ul></ul><ul><ul><ul><li>Balance of benefits and harms </li></ul></ul></ul><ul><ul><ul><li>(Quality of evidence) </li></ul></ul></ul><ul><ul><ul><li>Values and preferences </li></ul></ul></ul><ul><ul><ul><li>Costs (resource use) </li></ul></ul></ul><ul><ul><li>Judgements about the strength of recommendations </li></ul></ul><ul><ul><li>Wording of recommendations </li></ul></ul><ul><li>15.00 – 17.00 h - Small groups </li></ul><ul><li>Summary of findings and the strength of recommendation for HPV vaccination </li></ul><ul><li>17.00 – 18.00 h - Plenary </li></ul><ul><li>Discussion </li></ul>
  118. 126. Any questions from yesterday about quality of evidence or about the agenda for today?
  119. 127. Summarising and presenting findings Andy
  120. 128. Questions about summarising and presenting findings?
  121. 129. Strength of recommendations Holger
  122. 130. Strength of recommendation <ul><li>The degree of confidence that the desirable effects of adherence to a recommendation outweigh the undesirable effects. </li></ul><ul><li>Desirable effects </li></ul><ul><li>health benefits </li></ul><ul><li>less burden </li></ul><ul><li>savings </li></ul><ul><li>Undesirable effects </li></ul><ul><li>harms </li></ul><ul><li>more burden </li></ul><ul><li>costs </li></ul>
  123. 131. Categories of recommendations <ul><li>Although the degree of confidence is a continuum, we suggest using two categories: strong and weak. </li></ul><ul><li>Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects. </li></ul><ul><li>Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident. </li></ul>Recommend   Suggest  
  124. 132. Determinants of strength of recommendation Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.
  125. 133. Judgements about the strength of a recommendation <ul><li>No precise threshold for going from a strong to a weak recommendation </li></ul><ul><li>The presence of important concerns about one or more of these factors make a weak recommendation more likely. </li></ul><ul><li>Panels should consider all of these factors and make the reasons for their judgements explicit. </li></ul><ul><li>Recommendations should specify the perspective that is taken (e.g. individual patient, health system) and which outcomes were considered (including which, if any costs). </li></ul>
  126. 134. Implications of a strong recommendation <ul><li>Patients : Most people in your situation would want the recommended course of action and only a small proportion would not </li></ul><ul><li>Clinicians : Most patients should receive the recommended course of action </li></ul><ul><li>Policy makers : The recommendation can be adapted as a policy in most situations </li></ul>
  127. 135. Implications of a weak recommendation <ul><li>Patients : The majority of people in your situation would want the recommended course of action, but many would not </li></ul><ul><li>Clinicians : Be prepared to help patients to make a decision that is consistent with their own values </li></ul><ul><li>Policy makers : There is a need for substantial debate and involvement of stakeholders </li></ul>
  128. 136. Is France right? <ul><li>Recommend </li></ul><ul><ul><li>for or against? </li></ul></ul><ul><li>Strength of recommendation </li></ul><ul><ul><li>strong or weak </li></ul></ul>Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.
  129. 137. Another example Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? <ul><li>Schünemann HJ et al. WHO Rapid Advice Guidelines for the pharmacological management of human infection with avian influenza A (H5N1) virus. Lancet Infectious Diseases 2007; 7:21-31. </li></ul><ul><li>Schünemann HJ et al. WHO Rapid Advice Guidelines: Quick and Transparent. PLoS Medicine 2007; 4:e119. </li></ul>
  130. 138. Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? Summary of findings Transmission : No human to human transmission Patient or population: Hospitalised, clinical and serologically confirmed cases of avian influenza
  131. 139. What would you recommend? <ul><li>Strong recommendation : the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects. </li></ul><ul><li>Weak recommendation : the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident. </li></ul>Factors Impact on the strength of a recommendation Balance between desirable and undesirable effects Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted. Quality of the evidence Higher the quality of evidence, more likely a strong recommendation warranted. Values and preferences More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted. Costs (resource use) Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.
  132. 140. Judgments about the strength of a recommendation - oseltamivir for treatment of patients hospitalised with avian influenza (H5N1) Factors Comments Balance between desirable and undesirable effects The benefits are uncertain. Quality of the evidence The quality of the evidence is very low. Values and preferences All patients and care providers would accept treatment for H5N1 disease. Costs (resource use) The cost is not high for treatment of sporadic cases.
  133. 141. Another example Should aspirin be given to children with fever? <ul><li>Very low quality evidence of increased risk of Reye’s syndrome </li></ul><ul><li>Acetaminophen (paracetamol) is a cheap and effective alternative </li></ul>
  134. 142. Questions about the strength of recommendations?
  135. 143. Wording of recommendations ??
  136. 144. Questions about the wording of recommendations?
  137. 145. Recommendations to use an intervention only in the context of research Andy
  138. 146. Possible recommendations when there is very low quality evidence include: <ul><li> Recommend using an intervention </li></ul><ul><li> Suggest using an intervention </li></ul><ul><li>Recommend using an intervention only in the context of research </li></ul><ul><li> Suggest not using an intervention </li></ul><ul><li> Recommend not using an intervention </li></ul>
  139. 147. Recommendations to use an intervention only in the context of research may be appropriate <ul><li>Two conditions must be met </li></ul><ul><li>There must be in sufficient evidence for a panel to suggest using or not using an intervention. </li></ul><ul><li>Further research must have a large potential for reducing uncertainty about the effects of the intervention, and for doing so at a reasonable cost. </li></ul>
  140. 148. Examples <ul><li>National Institute of Health and Clinical Excellence (NICE) </li></ul><ul><ul><li>Of its first 95 technology appraisals, eight led to recommendations for use in the context of research. </li></ul></ul><ul><li>Centers for Medicare and Medicaid Services (CMMS) </li></ul><ul><ul><li>June 7, 2000 executive memorandum directing the Secretary of Health and Human Services to &quot;explicitly authorize [Medicare] payment for routine patient care costs...and costs due to medical complications associated with participation in clinical trials.“ </li></ul></ul><ul><ul><li>July 9, 2007 the following was added to the Clinical Trial Policy: </li></ul></ul><ul><ul><li>CMMS . . . through an individualized assessment of benefits, risks, and research potential, may determine that certain items and services for which there is some evidence of significant medical benefit, but for which there is insufficient evidence to support a “reasonable and necessary” determination, are only reasonable and necessary when provided in a clinical trial . . . </li></ul></ul>
  141. 149. Implications for research <ul><li>Core elements </li></ul><ul><li>E Evidence (What is the current state of the evidence?) </li></ul><ul><li>P Population (What is the population of interest?) </li></ul><ul><li>I Intervention (What are the interventions of interest?) </li></ul><ul><li>C Comparison (What are the comparisons of interest?) </li></ul><ul><li>O Outcome (What are the outcomes of interest?) </li></ul><ul><li>T Time stamp (Date of recommendation) </li></ul><ul><li>Optional elements </li></ul><ul><li>d Disease burden or relevance </li></ul><ul><li>t Time aspect of core elements of EPICOT </li></ul><ul><li>s Appropriate study type according to local need </li></ul>
  142. 150. What are examples of interventions you can think of for which you might recommend that it should only be used in the context of research?
  143. 151. Questions about recommendations to use an intervention only in the context of research?
  144. 152. Judgements about priorities (importance) Andy
  145. 153. Priority setting: From a health system or public health perspective A strong recommendation may not be important
  146. 154. Judgements about importance may depend on <ul><li>Strength of the recommendations </li></ul><ul><li>Severity of the problem </li></ul><ul><li>Prevalence of the problem </li></ul><ul><li>Magnitude of the net benefits </li></ul><ul><li>Efficiency (better use of resources) </li></ul><ul><li>Total cost </li></ul><ul><li>Impact on inequities </li></ul><ul><li>Potential for improved quality of care </li></ul>
  147. 155. From the perspective of individual patients and clinicians <ul><li>Most patients would want and should receive strongly recommended courses of action </li></ul><ul><li>The majority of people would want weakly recommended courses of action, but many would not and clinicians should be prepared to help patients make a decision that is consistent with their own values </li></ul><ul><li>The issue of priority setting (importance) is usually not important. </li></ul>
  148. 156. Questions about judgements about priorities (importance)?
  149. 157. Workshop agenda Tuesday, 13 May 2008 Strength of recommendation <ul><li>08.30 – 13.30 h - Plenary </li></ul><ul><li>Going from evidence to recommendations </li></ul><ul><ul><li>Summarising and presenting findings </li></ul></ul><ul><ul><li>Factors that determine the strength of a recommendation </li></ul></ul><ul><ul><ul><li>Balance of benefits and harms </li></ul></ul></ul><ul><ul><ul><li>(Quality of evidence) </li></ul></ul></ul><ul><ul><ul><li>Values and preferences </li></ul></ul></ul><ul><ul><ul><li>Costs (resource use) </li></ul></ul></ul><ul><ul><li>Judgements about the strength of recommendations </li></ul></ul><ul><ul><li>Wording of recommendations </li></ul></ul><ul><li>15.00 – 17.00 h - Small groups </li></ul><ul><li>Summary of findings and the strength of recommendation for HPV vaccination </li></ul><ul><li>17.00 – 18.00 h - Plenary </li></ul><ul><li>Discussion </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×