Evidence to Decision tables – 11th CE&B's research day presentation

1,235 views

Published on

A 10 minute presentation of a case study

Published in: Health & Medicine

Evidence to Decision tables – 11th CE&B's research day presentation

  1. 1. Case study From evidence to decisions Frameworks for Clinical Guidelines’ Development Carlos A. Cuello, MD, PhD(c) Health Research Methodology Program! Department of Clinical Epidemiology and Biostatistics March 27, 2014
  2. 2. Disclosure Authors belong to the CE&B department and to the GRADE / DECIDE working group. No other conflicts to declare.
  3. 3. Usefulness of the Evidence to DECISION table for clinical practice guidelines: a case study with a panel of international experts Carlos Cuello-Garcia, Jan Brozek, Juan José Yepes-Nuñez, Yuan Zhang, Shreyas Gandhi, Arnav Agarwal, Holger Schünemann Clinical Epidemiology and Biostatistics McMaster University MacGRADE Center
  4. 4. What are these frameworks? How can they help? Why should we use them?
  5. 5. decision (recommendation) research evidence benefits vs harms resource use patient values equity feasible? acceptable? Clinical Guideline
  6. 6. FRAMEWORKS evidence to decision (EtD)
  7. 7. just 4 columns & conclusions
  8. 8. CRITERIA Problem Quality of evidence Benefits & harms Values Resource use Equity Acceptability Feasibility JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION
  9. 9. JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION CRITERIA Problem Quality of evidence Benefits & harms Values Resource use Equity Acceptability Feasibility
  10. 10. JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION CRITERIA Problem Quality of evidence Benefits & harms Values Resource use Equity Acceptability Feasibility
  11. 11. JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION CRITERIA Problem Quality of evidence Benefits & harms Values Resource use Equity Acceptability Feasibility
  12. 12. JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION CRITERIA Problem Quality of evidence Benefits & harms Values Resource use Equity Acceptability Feasibility
  13. 13. CONCLUSIONS Balance of consequences Decision /recommendation Justification Implementation considerations Monitoring Evaluation Research priorities …
  14. 14. PURPOSE OF THE FRAMEWORK
  15. 15. PROS CONS Inform decision makers’ judgements about the pros and cons of each option (intervention) that is considered.
  16. 16. The important factors that determine a decision (criteria) are considered.
  17. 17. Structure discussion and manage disagreements Make the basis for decisions transparent to guideline users
  18. 18. Provide a concise summary of the best available research evidence to inform judgements about each criterion
  19. 19. A case study . . .
  20. 20. Guideline for Allergic Diseases Prevention
  21. 21. INTERNATIONAL PANEL
  22. 22. guidelines on allergy prevention • Probiotics • Prebiotics • Vitamin D
  23. 23. guidelines on allergy prevention • Probiotics • Prebiotics • Vitamin D
  24. 24. Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013 Generic EtD framework 1 Evidence to decision framework Question 1: Should probiotics vs. no probiotics be used in pregnant women? Population:!pregnant!women! Option:!probiotics! Comparison:!no!probiotics! Setting:3outpatient! Perspective:3individual!patient Background: The$intestinal$microbiome$could$play$an$important$role$in$the$immune$system$maturation,$ and$it$has$been$suggested$that$early6life$probiotic$administration,$whether$directly$to$the$infant$or$in$ their$mothers$breast$milk,$may$reduce$the$risk$of$allergies$in$childhood.$The$objective$of$this$question$is$ to$evaluate$the$impact$of$probiotics$administered$to$the$expecting$mothers$on$their$infant. Subgroup considerations: subpopulation of women at high risk for allergy in a child CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION PROBLEM Is the problem a priority? No Probably no Uncertain Probably yes Yes X Allergic diseases represent a spectrum of health conditions and a worldwide burden in different populations. (1) Are a large number of people affected? No Probably no Uncertain Probably yes Yes X As many as 40% of the worldwide population is affected by any type of allergy. In infants prevalence depends highly on the allergic status of their parents, being approximately of 10% in those without an allergic parent or sibling, versus 20% to 30% in those with the atopic background in their relatives. (2) Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
  25. 25. Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013 Generic EtD framework 2 CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS VALUES Is there important uncertainty or variability about how much people value the main outcomes? Important uncertainty or variability Possibly important uncertainty or variability Probably no important uncertainty or variability No important uncertainty or variability No known undesirable outcomes X Detailed judgements The relative importance or values of the main outcomes of interest: Outcome Relative importance Certainty of the evidence Eczema critical low Asthma/wheezing critical low Food allergy critical low Adverse effects critical low We judged that the outcomes eczema, asthma and food allergy are critical for people. The adverse outcomes are probably of high importance and the burden of taking daily pills is limited. Some immunocompromised women might not accept the risk. BENEFITS&HARMSOFTHEOPTIONS What is the overall certainty of the evidence of effectiveness? No included studies Very low Low Moderate High X Summary of findings: Outcome With [intervention] Without [intervention] Difference (per 100) (95%CI) Relative effect (RR) (95%CI) Certainty of the evidence (GRADE) Eczema (follow-up 1 to 5 years) 365/1520 (24%) 484/1515 (31.9%) 9 fewer per 100 (from 4 fewer to 13 fewer) RR 0.72 (0.6 to 0.86) ⊕⊝⊝⊝ VERY LOW Asthma/wheezing (follow-up 2 to 7 years) 143/992 (14.4%) 139/982 (14.2%) 0 fewer per 100 (from 3 fewer to 3 more) RR 0.97 (0.77 to 1.22) ⊕⊕⊝⊝ LOW Food allergy (follow-up 1 to 2 years) 36/279 (12.9%) 41/284 (14.4%) 1 more per 100 (from 3 fewer to 8 more) RR 1.08 (0.73 to 1.59) ⊕⊝⊝⊝ VERY LOW Adverse effects 101/394 (25.6%) 88/397 (22.2%) 3 more per 100 (from 4 fewer to 12 more) RR 1.13 (0.82 to 1.52) ⊕⊝⊝⊝ VERY LOW Link to detailed evidence profile Subgroup considerations: Link(s) to summary of findings and judgments for subgroups The data are indirect for all outcomes because they are primarily derived from studies that looked at mixed exposure in women during pregnancy and breastfeeding and of infants after birth. Only 1 RCT assessed the effect on eczema in pregnant women only: RR 0.88 (0.63 to 1.22); RD 5 fewer per 100 (from 14 fewer to 9 more) 5 RCTs included pregnant women + later breastfeeding mothers: RR 0.5 (0.4 to 0.63); RD 21 fewer per 100 (from 15 fewer to 25 fewer) 5 RCTs included pregnant women + infants after birth (follow-up 1 to 5 years): RR 0.87 (0.72 to 1.04), RD 4 fewer per 100 (from 8 fewer to 1 more) 3 RCTs included pregnant women + subsequently breastfeeding + infants (follow-up 3 to 4 years): RR 0.78 (0.49 to 1.24); RD 7 fewer per 100 (from 17 fewer to 8 more) No effects were observed on asthma/wheezing and food allergy. How substantial are the desirable anticipated effects? Don’t know Not important Somewhat important Moderately important Very important Varies X X Detailed judgements There was some disagreement among panel members whether the effect is somewhat or moderately important. How substantial are the undesirable anticipated effects? Don’t know Very important Moderately important Somewhat important Not important Varies X Detailed judgements No serious adverse effects, and no difference in mild adverse effects between the groups. Do the desirable effects outweigh the undesirable effects? No Probably No Don’t know Probably Yes Yes Varies X Detailed judgements Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
  26. 26. Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013 Generic EtD framework 2 CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS VALUES Is there important uncertainty or variability about how much people value the main outcomes? Important uncertainty or variability Possibly important uncertainty or variability Probably no important uncertainty or variability No important uncertainty or variability No known undesirable outcomes X Detailed judgements The relative importance or values of the main outcomes of interest: Outcome Relative importance Certainty of the evidence Eczema critical low Asthma/wheezing critical low Food allergy critical low Adverse effects critical low We judged that the outcomes eczema, asthma and food allergy are critical for people. The adverse outcomes are probably of high importance and the burden of taking daily pills is limited. Some immunocompromised women might not accept the risk. BENEFITS&HARMSOFTHEOPTIONS What is the overall certainty of the evidence of effectiveness? No included studies Very low Low Moderate High X Summary of findings: Outcome With [intervention] Without [intervention] Difference (per 100) (95%CI) Relative effect (RR) (95%CI) Certainty of the evidence (GRADE) Eczema (follow-up 1 to 5 years) 365/1520 (24%) 484/1515 (31.9%) 9 fewer per 100 (from 4 fewer to 13 fewer) RR 0.72 (0.6 to 0.86) ⊕⊝⊝⊝ VERY LOW Asthma/wheezing (follow-up 2 to 7 years) 143/992 (14.4%) 139/982 (14.2%) 0 fewer per 100 (from 3 fewer to 3 more) RR 0.97 (0.77 to 1.22) ⊕⊕⊝⊝ LOW Food allergy (follow-up 1 to 2 years) 36/279 (12.9%) 41/284 (14.4%) 1 more per 100 (from 3 fewer to 8 more) RR 1.08 (0.73 to 1.59) ⊕⊝⊝⊝ VERY LOW Adverse effects 101/394 (25.6%) 88/397 (22.2%) 3 more per 100 (from 4 fewer to 12 more) RR 1.13 (0.82 to 1.52) ⊕⊝⊝⊝ VERY LOW Link to detailed evidence profile Subgroup considerations: Link(s) to summary of findings and judgments for subgroups The data are indirect for all outcomes because they are primarily derived from studies that looked at mixed exposure in women during pregnancy and breastfeeding and of infants after birth. Only 1 RCT assessed the effect on eczema in pregnant women only: RR 0.88 (0.63 to 1.22); RD 5 fewer per 100 (from 14 fewer to 9 more) 5 RCTs included pregnant women + later breastfeeding mothers: RR 0.5 (0.4 to 0.63); RD 21 fewer per 100 (from 15 fewer to 25 fewer) 5 RCTs included pregnant women + infants after birth (follow-up 1 to 5 years): RR 0.87 (0.72 to 1.04), RD 4 fewer per 100 (from 8 fewer to 1 more) 3 RCTs included pregnant women + subsequently breastfeeding + infants (follow-up 3 to 4 years): RR 0.78 (0.49 to 1.24); RD 7 fewer per 100 (from 17 fewer to 8 more) No effects were observed on asthma/wheezing and food allergy. How substantial are the desirable anticipated effects? Don’t know Not important Somewhat important Moderately important Very important Varies X X Detailed judgements There was some disagreement among panel members whether the effect is somewhat or moderately important. How substantial are the undesirable anticipated effects? Don’t know Very important Moderately important Somewhat important Not important Varies X Detailed judgements No serious adverse effects, and no difference in mild adverse effects between the groups. Do the desirable effects outweigh the undesirable effects? No Probably No Don’t know Probably Yes Yes Varies X Detailed judgements Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
  27. 27. Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013 Generic EtD framework 3 CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS RESOURCEUSE How large are the resource requirements? Large costs Moderate costs Small Moderate savings Large savings Varies X Detailed judgements Prices are likely to vary substantially depending on the setting. This may be a particularly important consideration in low and middle-income countries. A level and type of insurance may play a substantial role as well. From a health systems point of view it might also be cost effective given that probiotic would be used for 9 months and cost of treatment of eczema may be distributed across many years. Extremely limited research evidence (internet searches of drug prices) Bifidobacterium bifidum (cost per person per year US$) Dose: 1 pill each day Lactobacillus gg (cost per person per year US$) 1 pill each day North- America Average $181.16 $341.6 South- America Average $174.3 $286 Europe Average $167.86 $251.56 Fewer office visits would occur as a result of eczema if the effects on eczema were true. How large is the incremental cost relative to the net benefit? Very large ICER Large ICER Moderate ICER Small ICER Savings Varies Detailed judgements No research evidence If eczema was reduced the intervention might be cost- effective given fewer office visits (between $17,400 and $34,100 to treat 100 people for 1 year or ¾ of that for 9 months) preventing 9 cases of eczema. In most studies probiotics were used in the last trimester of pregnancy, which, if used this same way, might reduce the cost per pregnant woman. EQUITY What would be the impact on health inequities? Increased Probably increased Uncertain Probably reduced Reduced Varies X Detailed judgements No research evidence In some settings it may be important to consider equity as the access may depend on socioeconomic status of the country or setting where coverage will depend on policymakers. ACCEPTABILITY Is the option acceptable to key stakeholders? No Probably No Uncertain Probably Yes Yes Varies X Detailed judgements No research evidence Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
  28. 28. Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013 Generic EtD framework 4 CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS FEASIBILITY Is the option feasible to implement? No Probably No Uncertain Probably Yes Yes Varies X Detailed judgements No research evidence Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
  29. 29. Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013 Generic EtD framework 5 Recommendation Should probiotics vs. no probiotics be used in pregnant women (exposing their children in utero)? Overall balance of consequences Undesirable consequences clearly outweigh desirable consequences Undesirable consequences probably outweigh desirable consequences The balance of desirable and undesirable consequences indicates they are very similar* Desirable consequences probably outweigh undesirable consequences Desirable consequences clearly outweigh undesirable consequences ! ! ! X ! We recommend against the option or for the alternative We suggest not to use the option or to use the alternative We suggest using the option We recommend the option ! ! X ! Panel decisions 3 panel members with potential COI recused themselves from participating in formulating the recommendation. Consensus was obtained from the rest of the team. Recommendation (text) The guideline panel suggests using probiotics in pregnant women at high risk for allergy in their children (conditional recommendation, very low quality evidence). Remarks and justification Most studies commenced probiotics in the last trimester of pregnancy. The very low quality evidence for adverse effects indicates that our confidence in the absence of increased adverse effects is low. Future research is needed (see definitions of very low quality) e.g., generalizing to immune-compromised children Subgroup considerations Women with high risk of allergy in their children Women with average risk of allergy in their children Implementation considerations This recommendation is based on trials investigating the probiotics or mixtures of probiotic listed below. We have not found a difference between these different probiotics, but that does not mean there is no difference Monitoring and evaluation considerations Research priorities Develop instruments for evaluating the risk of allergy in children as the family history predicts only about 30% of the population risk. There is some evidence that first child is at higher risk for allergy than subsequent children. Long-term follow-up of long-term effects. No direct evidence for the use of probiotics in formula – this should be evaluated in future research and is an unmet need. Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013
  30. 30. SURVEY
  31. 31. 24 panel members 13 RESPONDED
  32. 32. 11 ACCEPTED TO COMPLETE THE SURVEY
  33. 33. 1. The EtD table helped in the ORGANIZATION of the development process (i.e., for getting a recommendation from a body of evidence) Strongly disagree Disagree Somewhat disagree neutral Somewhat agree Agree Strongly agree
  34. 34. 2. The EtD is a structured process that is BETTER than an informal/unstructured discussion Strongly disagree Disagree Somewhat disagree neutral Somewhat agree Agree Strongly agree
  35. 35. 3. The EtD helped you UNDERSTAND the process of getting a recommendation from a body of evidence Strongly disagree Disagree Somewhat disagree neutral Somewhat agree Agree Strongly agree
  36. 36. 4. The EtD made you think about concepts like patient values & preferences, costs & risks of the intervention, equity, acceptance, and feasibility of the intervention Strongly disagree Disagree Somewhat disagree neutral Somewhat agree Agree Strongly agree
  37. 37. – Anonymous panel member “It made the process smooth, easy to understand and significantly reduced the time it would have otherwise taken to complete”
  38. 38. What are these frameworks? How can they help? Why should we use them?
  39. 39. Gracias @CharlieNeck cuelloca@mcmaster.ca

×