Analysis and Interpretation


Published on

Cochrane Review author training workshop, January 22-23, 2009 at the University of Calgary Health Sciences Centre

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Analysis and Interpretation

    1. 1. Analysis and Interpretation: Overview <ul><li>Analyses </li></ul><ul><ul><li>Narrative: summary and discussion </li></ul></ul><ul><ul><li>Quantitative: involving statistical analysis (including meta-analysis) </li></ul></ul><ul><li>Meta-analysis should only be used when appropriate </li></ul><ul><li>Inappropriate to define a systematic review as high quality based on whether it contains a meta-analysis </li></ul>
    2. 2. Framework for synthesis <ul><li>Whether narrative or quantitative, a general framework for synthesis: </li></ul><ul><li>What is the direction of effect? </li></ul><ul><li>What is the size of the effect? </li></ul><ul><li>Is the effect consistent across studies? </li></ul><ul><li>What is the strength of evidence for the effect? </li></ul>
    3. 3. Why Perform a Meta-analysis? <ul><li>Increases statistical power </li></ul><ul><li>To improve precision </li></ul><ul><li>Answer questions not posed by individual studies </li></ul><ul><li>Settle controversies from conflicting studies or generate new hypotheses </li></ul><ul><li>Meta-analyses: derive meaningful conclusions from data and help prevent errors in interpretation </li></ul>
    4. 4. More on Meta-analysis <ul><li>What it is not: adding up all the patients among trials; trials need to be weighted </li></ul><ul><li>May be possible to conduct for some comparisons/outcomes in a review and not for others </li></ul><ul><li>Need to determine whether the studies are similar enough to be meta-analyzed </li></ul><ul><li>Need to make a decision as to </li></ul><ul><li>whether it is appropriate! </li></ul>
    5. 5. When not Appropriate to do M/a <ul><li>If studies are clinically diverse </li></ul><ul><ul><li>Results may be meaningless </li></ul></ul><ul><ul><li>Genuine differences may be obscured </li></ul></ul><ul><li>If a mix of comparisons -> determine which need to be assessed separately </li></ul><ul><li>If outcomes too diverse </li></ul><ul><li>If includes studies at risk of bias, these results may be misleading </li></ul><ul><li>Presence of serious publication or reporting biases </li></ul>
    6. 6. Dichotomous Measures <ul><li>Whether individual study or meta-analysis: </li></ul><ul><li>Relative measures: Risk ratio (RR) or Odds ratio (OR) </li></ul><ul><li>Absolute measure: Risk difference (RD) </li></ul><ul><li>Number needed to treat (NNT) </li></ul>
    7. 7. Risk ratio (RR) aka relative risk RR = a / (a+b) c / (c+d) Risk/ probability/ chance of the occurrence of an event in treatment relative to control Intervention Control a+b=n I c+d=n C Event No event d c b a
    8. 8. Sample RR Calculation Death No death RR = 14/133 = 0.11 = 0.13 128/148 0.86 Drug 133 148 Placebo 20 128 119 14
    9. 9. Odds ratio (OR) Intervention Control No event Event OR = a / b c / d Odds of an event occurring to it not occurring for treatment relative to control a+b=n I c+d=n C d c b a
    10. 10. Sample OR Calculation Death No death Drug Placebo 133 148 OR = 14/119 = 0.12 = 0.019 128/20 6.4 20 128 119 14
    11. 11. Interpreting (for intervention) Increased odds (harmful) Increased odds (beneficial) OR>1 (6.4/0.12) Reduced odds (beneficial) Reduced odds (not beneficial) OR<1 (0.12/6.4) No difference No difference OR=1, RR=1 Increased risk (harmful) Increased risk (beneficial) RR>1 (0.86/0.11) Reduced risk (beneficial) Reduced risk (not beneficial) RR<1 (0.11/0.86) Bad outcome (e.g. infection) Good outcome (e.g. remission)
    12. 12. RR vs. OR <ul><li>Different measures – people make the mistake of interpreting them to be the same </li></ul><ul><li>Similar values when events are rare, but differences noted when events are common: </li></ul><ul><ul><li>When Rx increases chances of events, OR>RR </li></ul></ul><ul><ul><li>When Rx decreases chances of events, OR<RR </li></ul></ul><ul><ul><li>In both cases, if OR interpreted as RR, leads to overestimation of the intervention effect! </li></ul></ul><ul><li>RR for an event vs non-event not the same! </li></ul>
    13. 13. Closer Look at Odds RR = 0.11 / 0.86 = 0.13 ↑ A rate (11%) OR = 0.12 / 6.4 = 0.019 ↑ ~1:9 ↑ ~7:1
    14. 14. Absolute Effect Measures <ul><li>Relative measures don’t tell you the actual number of participants who benefited </li></ul><ul><ul><li>RR 2.0….same for 80% vs 40% as for 10% vs 5%...but these are very different event rates! </li></ul></ul>
    15. 15. Risk Difference (RD) Death No death Actual difference in risk of events Placebo Drug 133 148 RD = 14/133 – 128/148 = 0.11 – 0.86 = - 0.75 20 128 119 14
    16. 16. Risk Difference (RD) (continued) <ul><li>RD = 0, no difference between groups </li></ul><ul><li>RD<0 reduces risk ( ☺ for bad outcome, not for good outcome) </li></ul><ul><li>RD>0 increases risk (☺ for good outcome, harmful for bad) </li></ul>
    17. 17. NNT <ul><li>Expected number of people who need to receive the experimental rather than the comparator intervention for one additional person to incur or avoid an event in a give time frame </li></ul><ul><li>If a single study, can calculate from RD </li></ul><ul><li>Cannot be combined in a meta-analysis; need to calculate from another meta-analysis summary statistic </li></ul><ul><li>From a meta-analysis, should be calculated from either OR or RR </li></ul><ul><li>Chapter 12 </li></ul>
    18. 18. Uncertainty <ul><li>Confidence interval, usually 95% </li></ul><ul><ul><li>Range of values above and below the calculated treatment effect within which we can be reasonably certain (e.g., 95% certain) that the real effect lies. </li></ul></ul><ul><ul><li>For RR and OR, results are statistically significant if CI does not include 1 </li></ul></ul><ul><ul><li>For RD, results are statistically significant if CI does not include 0 </li></ul></ul>
    19. 19. Which effect measure for meta-analysis? <ul><li>Relative effect measures are, on average, suggested to be more consistent than absolute measures (empirical evidence) </li></ul><ul><li>Avoid RD unless clear reason to suspect consistency </li></ul><ul><li>Generally recommend: RR or OR, but remember risk of misinterpretion of OR </li></ul>
    20. 20. Meta-analysis in RevMan
    21. 21. Meta-analysis in RevMan (continued) <ul><li>Formulae for calculating effect measures and confidence intervals available on </li></ul><ul><li>Not available in RevMan: meta-regression </li></ul>
    22. 22. Fixed vs Random Effects <ul><li>Fixed effects : true effect of intervention (magnitude and direction) is the same value in every study </li></ul><ul><ul><li>‘ typical intervention effect’ </li></ul></ul><ul><ul><li>No study-to-study variability </li></ul></ul><ul><ul><li>Only within study variability </li></ul></ul><ul><li>Random effects : effects being estimated among studies are not identical but follow some distribution </li></ul><ul><ul><li>studies estimating different, yet related, intervention effects </li></ul></ul><ul><ul><li>estimate and CI: centre of the distribution of effects </li></ul></ul>
    23. 23. Fixed Effects Analysis in Picture View
    24. 24. Random Effects Analysis in Picture View
    25. 25. Random effects in RevMan 5 ← DerSimonian and Laird random effects model
    26. 26. Random effects in RevMan 5 (continued) ← DerSimonian and Laird random effects model
    27. 27. Sample Forest plot (RR) <ul><li># pts with events & total pts in each group </li></ul>
    28. 28. Meta-analysis for Continuous Data <ul><li>Two effect measures for data with normal distribution: MD and SMD </li></ul><ul><li>Data: Sample size, mean, standard deviation (SD) </li></ul><ul><li>Don’t confuse SD with standard error (SE) </li></ul><ul><li>SD = SE x √n </li></ul><ul><li>Fixed or random effects analysis </li></ul><ul><li>For change-from-baseline data: Chapters 7 and 9 </li></ul><ul><li>Skewed data: Chapter 9 </li></ul>
    29. 29. Mean Difference (MD) <ul><li>Formerly called weighted mean difference </li></ul><ul><li>When studies use same scale for outcomes </li></ul>
    30. 30. Standardized Mean Difference (SMD) <ul><li>Use when trials assess the same outcome but measure in a variety of ways, including using different scales </li></ul>
    31. 31. Heterogeneity <ul><li>Any kind of variability among studies </li></ul><ul><li>Clinical: participants, interventions, outcomes </li></ul><ul><ul><li>True intervention effect will be different in different studies </li></ul></ul><ul><li>Methodologic: trial design, quality </li></ul><ul><ul><li>Studies not estimating same quantity, suffer different degrees of bias </li></ul></ul><ul><li>Statistical: from clinical or methodologic…or both! </li></ul><ul><ul><li>Observed effects of intervention are more different than that expected by chance </li></ul></ul><ul><ul><li>In practice, can be difficult to separate the influence of clinical vs methodologic on observed statistical heterogeneity…likely due to both </li></ul></ul>
    32. 32. Clinical and Methodologic Heterogeneity <ul><li>Are differences across studies so great that they should not be combined? </li></ul><ul><li>At protocol stage, specify factors that you plan to investigate as potential causes of heterogeneity </li></ul><ul><li>Be transparent with a priori vs post hoc investigations of heterogeneity in a review </li></ul>
    33. 33. Statistical Heterogeneity <ul><li>To what extent are the results consistent? </li></ul><ul><li>Q test and I 2 statistic </li></ul>
    34. 34. Q test <ul><li>Q test: ‘chi-squared’ statistic </li></ul><ul><ul><li>Care must be taken in interpretation </li></ul></ul><ul><ul><li>Low power with few studies or small sample size </li></ul></ul><ul><ul><ul><li>Just because stat is not significant doesn’t mean absence of heterogeneity </li></ul></ul></ul><ul><ul><li>High power with many studies </li></ul></ul><ul><ul><ul><li>Heterogeneity detected may not be clinically important </li></ul></ul></ul><ul><ul><li>Use P value cut-off of 0.10 to compensate </li></ul></ul>
    35. 35. I 2 Statistic <ul><li>Instead of testing whether there, assess impact </li></ul><ul><li>I 2 quantifies extent of inconsistency </li></ul><ul><ul><li>Percentage of variability in effect estimates that is due to heterogeneity rather than chance </li></ul></ul>
    36. 36. I 2 Statistic (continued) * Importance of I 2 value depends on: ● magnitude and direction of effects ● strength of evidence of heterogeneity - Chi-squared P value, or - I 2 confidence interval Considerable heterogeneity* 75% to 100% May represent substantial heterogeneity* 50% to 90% May represent moderate heterogeneity* 30% to 60% Might not be important 0% to 40% Guide to Interpretation I 2 value
    37. 37. Sample Forest Plot: Q and I 2
    38. 38. What to do with (Statistical) Heterogeneity <ul><li>Check that data are correct </li></ul><ul><li>Do not do the meta-analysis…may be misleading </li></ul><ul><li>Explore heterogeneity </li></ul><ul><ul><li>Subgroup analyses </li></ul></ul><ul><ul><li>Meta-regression </li></ul></ul><ul><li>Ignore it </li></ul><ul><ul><li>Fixed effects ignore heterogeneity – ignoring may mean an intervention effect that does not actually exist </li></ul></ul>
    39. 39. What to do with (Statistical) Heterogeneity <ul><li>Random effects meta-analysis </li></ul><ul><ul><li>Incorporates heterogeneity but is not a substitution for a thorough investigation </li></ul></ul><ul><li>Exclude studies </li></ul><ul><ul><li>Sensitivity analysis </li></ul></ul>
    40. 40. Subgroup and Meta-regression <ul><ul><li>Chapter 9 </li></ul></ul><ul><ul><li>Observational in nature </li></ul></ul><ul><ul><li>Characteristics used should be prespecified; keep to a minimum </li></ul></ul><ul><ul><li>Conclusions from such analyses should be interpreted with caution </li></ul></ul><ul><ul><li>Subgroups: splitting all studies into groups to make comparisons </li></ul></ul><ul><ul><li>Meta-regression: extension of subgroup analysis, allows investigation of continuous and categorical variables </li></ul></ul>
    41. 41. Subgroup Analysis
    42. 42. Sensitivity Analysis <ul><li>Chapter 9 </li></ul><ul><li>Addresses the question: Are the findings robust to the decisions make in the process of obtaining them? </li></ul><ul><li>Repeats the primary analysis and substitutes alternative decisions for decisions or range of values that were arbitrary or unclear </li></ul><ul><li>Some can be prespecified in the protocol but many issues are identified only during the review process </li></ul><ul><li>Don’t confuse with subgroup analysis </li></ul>