Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Analysis and Interpretation: Overview <ul><li>Analyses </li></ul><ul><ul><li>Narrative:  summary and discussion </li></ul>...
Framework for synthesis <ul><li>Whether narrative or quantitative, a general framework for synthesis: </li></ul><ul><li>Wh...
Why Perform a Meta-analysis? <ul><li>Increases statistical power </li></ul><ul><li>To improve precision </li></ul><ul><li>...
More on Meta-analysis <ul><li>What it is not:  adding up all the patients among trials; trials need to be weighted </li></...
When not Appropriate to do M/a <ul><li>If studies are clinically diverse </li></ul><ul><ul><li>Results may be meaningless ...
Dichotomous Measures <ul><li>Whether individual study or meta-analysis: </li></ul><ul><li>Relative measures:  Risk ratio (...
Risk ratio (RR)  aka relative risk RR =   a / (a+b)  c / (c+d) Risk/ probability/ chance  of the occurrence of an event in...
Sample RR Calculation Death No death RR =   14/133  =  0.11  = 0.13 128/148  0.86 Drug 133 148 Placebo 20 128 119 14
Odds ratio (OR) Intervention Control No event Event OR =  a / b  c / d  Odds of an event occurring to it not occurring for...
Sample OR Calculation Death No death Drug Placebo 133 148 OR =   14/119  =  0.12  = 0.019 128/20  6.4 20 128 119 14
Interpreting (for intervention) Increased odds (harmful) Increased odds  (beneficial) OR>1 (6.4/0.12) Reduced odds (benefi...
RR vs. OR <ul><li>Different measures – people make the mistake of interpreting them to be the same </li></ul><ul><li>Simil...
Closer Look at Odds RR = 0.11 / 0.86 = 0.13 ↑ A rate (11%) OR = 0.12 / 6.4  = 0.019 ↑ ~1:9 ↑ ~7:1
Absolute Effect Measures <ul><li>Relative measures don’t tell you the actual number of participants who benefited </li></u...
Risk Difference (RD) Death No death Actual difference in  risk of events Placebo Drug 133 148 RD = 14/133 – 128/148 = 0.11...
Risk Difference (RD)  (continued) <ul><li>RD = 0, no difference between groups </li></ul><ul><li>RD<0 reduces risk ( ☺  fo...
NNT <ul><li>Expected number of people who need to receive the experimental rather than the comparator intervention for one...
Uncertainty <ul><li>Confidence interval, usually 95% </li></ul><ul><ul><li>Range of values above and below the calculated ...
Which effect measure for meta-analysis? <ul><li>Relative effect measures are, on average, suggested to be more consistent ...
Meta-analysis in RevMan
Meta-analysis in RevMan  (continued) <ul><li>Formulae for calculating effect measures and confidence intervals available o...
Fixed vs Random Effects <ul><li>Fixed effects :  true effect of intervention (magnitude and direction) is the same value i...
Fixed Effects Analysis in Picture View
Random Effects Analysis in Picture View
Random effects in RevMan 5 ←  DerSimonian and Laird  random effects model
Random effects in RevMan 5  (continued) ←  DerSimonian and Laird  random effects model
Sample Forest plot (RR) <ul><li># pts with events & total pts in each group </li></ul>
Meta-analysis for Continuous Data <ul><li>Two effect measures for data with normal distribution:  MD and SMD </li></ul><ul...
Mean Difference (MD) <ul><li>Formerly called weighted mean difference </li></ul><ul><li>When studies use same scale for ou...
Standardized Mean Difference (SMD) <ul><li>Use when trials assess the same outcome but measure in a variety of ways, inclu...
Heterogeneity <ul><li>Any kind of variability among studies </li></ul><ul><li>Clinical:   participants, interventions, out...
    Clinical and Methodologic Heterogeneity  <ul><li>Are differences across studies so great that they should not be combi...
Statistical Heterogeneity <ul><li>To what extent are the results consistent? </li></ul><ul><li>Q test and I 2  statistic <...
Q test <ul><li>Q test:  ‘chi-squared’ statistic </li></ul><ul><ul><li>Care must be taken in interpretation </li></ul></ul>...
I 2  Statistic <ul><li>Instead of testing whether there, assess impact </li></ul><ul><li>I 2  quantifies extent of inconsi...
I 2  Statistic  (continued) * Importance of I 2  value depends on: ●  magnitude and direction of effects ●  strength of ev...
Sample Forest Plot:  Q and I 2
What to do with (Statistical) Heterogeneity <ul><li>Check that data are correct  </li></ul><ul><li>Do not do the meta-anal...
What to do with (Statistical) Heterogeneity <ul><li>Random effects meta-analysis </li></ul><ul><ul><li>Incorporates hetero...
Subgroup and Meta-regression <ul><ul><li>Chapter 9 </li></ul></ul><ul><ul><li>Observational in nature </li></ul></ul><ul><...
Subgroup Analysis
Sensitivity Analysis <ul><li>Chapter 9 </li></ul><ul><li>Addresses the question:  Are the findings robust to the decisions...
Upcoming SlideShare
Loading in …5
×

Analysis and Interpretation

14,923 views

Published on

Cochrane Review author training workshop, January 22-23, 2009 at the University of Calgary Health Sciences Centre

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Analysis and Interpretation

  1. 1. Analysis and Interpretation: Overview <ul><li>Analyses </li></ul><ul><ul><li>Narrative: summary and discussion </li></ul></ul><ul><ul><li>Quantitative: involving statistical analysis (including meta-analysis) </li></ul></ul><ul><li>Meta-analysis should only be used when appropriate </li></ul><ul><li>Inappropriate to define a systematic review as high quality based on whether it contains a meta-analysis </li></ul>
  2. 2. Framework for synthesis <ul><li>Whether narrative or quantitative, a general framework for synthesis: </li></ul><ul><li>What is the direction of effect? </li></ul><ul><li>What is the size of the effect? </li></ul><ul><li>Is the effect consistent across studies? </li></ul><ul><li>What is the strength of evidence for the effect? </li></ul>
  3. 3. Why Perform a Meta-analysis? <ul><li>Increases statistical power </li></ul><ul><li>To improve precision </li></ul><ul><li>Answer questions not posed by individual studies </li></ul><ul><li>Settle controversies from conflicting studies or generate new hypotheses </li></ul><ul><li>Meta-analyses: derive meaningful conclusions from data and help prevent errors in interpretation </li></ul>
  4. 4. More on Meta-analysis <ul><li>What it is not: adding up all the patients among trials; trials need to be weighted </li></ul><ul><li>May be possible to conduct for some comparisons/outcomes in a review and not for others </li></ul><ul><li>Need to determine whether the studies are similar enough to be meta-analyzed </li></ul><ul><li>Need to make a decision as to </li></ul><ul><li>whether it is appropriate! </li></ul>
  5. 5. When not Appropriate to do M/a <ul><li>If studies are clinically diverse </li></ul><ul><ul><li>Results may be meaningless </li></ul></ul><ul><ul><li>Genuine differences may be obscured </li></ul></ul><ul><li>If a mix of comparisons -> determine which need to be assessed separately </li></ul><ul><li>If outcomes too diverse </li></ul><ul><li>If includes studies at risk of bias, these results may be misleading </li></ul><ul><li>Presence of serious publication or reporting biases </li></ul>
  6. 6. Dichotomous Measures <ul><li>Whether individual study or meta-analysis: </li></ul><ul><li>Relative measures: Risk ratio (RR) or Odds ratio (OR) </li></ul><ul><li>Absolute measure: Risk difference (RD) </li></ul><ul><li>Number needed to treat (NNT) </li></ul>
  7. 7. Risk ratio (RR) aka relative risk RR = a / (a+b) c / (c+d) Risk/ probability/ chance of the occurrence of an event in treatment relative to control Intervention Control a+b=n I c+d=n C Event No event d c b a
  8. 8. Sample RR Calculation Death No death RR = 14/133 = 0.11 = 0.13 128/148 0.86 Drug 133 148 Placebo 20 128 119 14
  9. 9. Odds ratio (OR) Intervention Control No event Event OR = a / b c / d Odds of an event occurring to it not occurring for treatment relative to control a+b=n I c+d=n C d c b a
  10. 10. Sample OR Calculation Death No death Drug Placebo 133 148 OR = 14/119 = 0.12 = 0.019 128/20 6.4 20 128 119 14
  11. 11. Interpreting (for intervention) Increased odds (harmful) Increased odds (beneficial) OR>1 (6.4/0.12) Reduced odds (beneficial) Reduced odds (not beneficial) OR<1 (0.12/6.4) No difference No difference OR=1, RR=1 Increased risk (harmful) Increased risk (beneficial) RR>1 (0.86/0.11) Reduced risk (beneficial) Reduced risk (not beneficial) RR<1 (0.11/0.86) Bad outcome (e.g. infection) Good outcome (e.g. remission)
  12. 12. RR vs. OR <ul><li>Different measures – people make the mistake of interpreting them to be the same </li></ul><ul><li>Similar values when events are rare, but differences noted when events are common: </li></ul><ul><ul><li>When Rx increases chances of events, OR>RR </li></ul></ul><ul><ul><li>When Rx decreases chances of events, OR<RR </li></ul></ul><ul><ul><li>In both cases, if OR interpreted as RR, leads to overestimation of the intervention effect! </li></ul></ul><ul><li>RR for an event vs non-event not the same! </li></ul>
  13. 13. Closer Look at Odds RR = 0.11 / 0.86 = 0.13 ↑ A rate (11%) OR = 0.12 / 6.4 = 0.019 ↑ ~1:9 ↑ ~7:1
  14. 14. Absolute Effect Measures <ul><li>Relative measures don’t tell you the actual number of participants who benefited </li></ul><ul><ul><li>RR 2.0….same for 80% vs 40% as for 10% vs 5%...but these are very different event rates! </li></ul></ul>
  15. 15. Risk Difference (RD) Death No death Actual difference in risk of events Placebo Drug 133 148 RD = 14/133 – 128/148 = 0.11 – 0.86 = - 0.75 20 128 119 14
  16. 16. Risk Difference (RD) (continued) <ul><li>RD = 0, no difference between groups </li></ul><ul><li>RD<0 reduces risk ( ☺ for bad outcome, not for good outcome) </li></ul><ul><li>RD>0 increases risk (☺ for good outcome, harmful for bad) </li></ul>
  17. 17. NNT <ul><li>Expected number of people who need to receive the experimental rather than the comparator intervention for one additional person to incur or avoid an event in a give time frame </li></ul><ul><li>If a single study, can calculate from RD </li></ul><ul><li>Cannot be combined in a meta-analysis; need to calculate from another meta-analysis summary statistic </li></ul><ul><li>From a meta-analysis, should be calculated from either OR or RR </li></ul><ul><li>Chapter 12 </li></ul>
  18. 18. Uncertainty <ul><li>Confidence interval, usually 95% </li></ul><ul><ul><li>Range of values above and below the calculated treatment effect within which we can be reasonably certain (e.g., 95% certain) that the real effect lies. </li></ul></ul><ul><ul><li>For RR and OR, results are statistically significant if CI does not include 1 </li></ul></ul><ul><ul><li>For RD, results are statistically significant if CI does not include 0 </li></ul></ul>
  19. 19. Which effect measure for meta-analysis? <ul><li>Relative effect measures are, on average, suggested to be more consistent than absolute measures (empirical evidence) </li></ul><ul><li>Avoid RD unless clear reason to suspect consistency </li></ul><ul><li>Generally recommend: RR or OR, but remember risk of misinterpretion of OR </li></ul>
  20. 20. Meta-analysis in RevMan
  21. 21. Meta-analysis in RevMan (continued) <ul><li>Formulae for calculating effect measures and confidence intervals available on cochrane.org </li></ul><ul><li>Not available in RevMan: meta-regression </li></ul>
  22. 22. Fixed vs Random Effects <ul><li>Fixed effects : true effect of intervention (magnitude and direction) is the same value in every study </li></ul><ul><ul><li>‘ typical intervention effect’ </li></ul></ul><ul><ul><li>No study-to-study variability </li></ul></ul><ul><ul><li>Only within study variability </li></ul></ul><ul><li>Random effects : effects being estimated among studies are not identical but follow some distribution </li></ul><ul><ul><li>studies estimating different, yet related, intervention effects </li></ul></ul><ul><ul><li>estimate and CI: centre of the distribution of effects </li></ul></ul>
  23. 23. Fixed Effects Analysis in Picture View
  24. 24. Random Effects Analysis in Picture View
  25. 25. Random effects in RevMan 5 ← DerSimonian and Laird random effects model
  26. 26. Random effects in RevMan 5 (continued) ← DerSimonian and Laird random effects model
  27. 27. Sample Forest plot (RR) <ul><li># pts with events & total pts in each group </li></ul>
  28. 28. Meta-analysis for Continuous Data <ul><li>Two effect measures for data with normal distribution: MD and SMD </li></ul><ul><li>Data: Sample size, mean, standard deviation (SD) </li></ul><ul><li>Don’t confuse SD with standard error (SE) </li></ul><ul><li>SD = SE x √n </li></ul><ul><li>Fixed or random effects analysis </li></ul><ul><li>For change-from-baseline data: Chapters 7 and 9 </li></ul><ul><li>Skewed data: Chapter 9 </li></ul>
  29. 29. Mean Difference (MD) <ul><li>Formerly called weighted mean difference </li></ul><ul><li>When studies use same scale for outcomes </li></ul>
  30. 30. Standardized Mean Difference (SMD) <ul><li>Use when trials assess the same outcome but measure in a variety of ways, including using different scales </li></ul>
  31. 31. Heterogeneity <ul><li>Any kind of variability among studies </li></ul><ul><li>Clinical: participants, interventions, outcomes </li></ul><ul><ul><li>True intervention effect will be different in different studies </li></ul></ul><ul><li>Methodologic: trial design, quality </li></ul><ul><ul><li>Studies not estimating same quantity, suffer different degrees of bias </li></ul></ul><ul><li>Statistical: from clinical or methodologic…or both! </li></ul><ul><ul><li>Observed effects of intervention are more different than that expected by chance </li></ul></ul><ul><ul><li>In practice, can be difficult to separate the influence of clinical vs methodologic on observed statistical heterogeneity…likely due to both </li></ul></ul>
  32. 32. Clinical and Methodologic Heterogeneity <ul><li>Are differences across studies so great that they should not be combined? </li></ul><ul><li>At protocol stage, specify factors that you plan to investigate as potential causes of heterogeneity </li></ul><ul><li>Be transparent with a priori vs post hoc investigations of heterogeneity in a review </li></ul>
  33. 33. Statistical Heterogeneity <ul><li>To what extent are the results consistent? </li></ul><ul><li>Q test and I 2 statistic </li></ul>
  34. 34. Q test <ul><li>Q test: ‘chi-squared’ statistic </li></ul><ul><ul><li>Care must be taken in interpretation </li></ul></ul><ul><ul><li>Low power with few studies or small sample size </li></ul></ul><ul><ul><ul><li>Just because stat is not significant doesn’t mean absence of heterogeneity </li></ul></ul></ul><ul><ul><li>High power with many studies </li></ul></ul><ul><ul><ul><li>Heterogeneity detected may not be clinically important </li></ul></ul></ul><ul><ul><li>Use P value cut-off of 0.10 to compensate </li></ul></ul>
  35. 35. I 2 Statistic <ul><li>Instead of testing whether there, assess impact </li></ul><ul><li>I 2 quantifies extent of inconsistency </li></ul><ul><ul><li>Percentage of variability in effect estimates that is due to heterogeneity rather than chance </li></ul></ul>
  36. 36. I 2 Statistic (continued) * Importance of I 2 value depends on: ● magnitude and direction of effects ● strength of evidence of heterogeneity - Chi-squared P value, or - I 2 confidence interval Considerable heterogeneity* 75% to 100% May represent substantial heterogeneity* 50% to 90% May represent moderate heterogeneity* 30% to 60% Might not be important 0% to 40% Guide to Interpretation I 2 value
  37. 37. Sample Forest Plot: Q and I 2
  38. 38. What to do with (Statistical) Heterogeneity <ul><li>Check that data are correct </li></ul><ul><li>Do not do the meta-analysis…may be misleading </li></ul><ul><li>Explore heterogeneity </li></ul><ul><ul><li>Subgroup analyses </li></ul></ul><ul><ul><li>Meta-regression </li></ul></ul><ul><li>Ignore it </li></ul><ul><ul><li>Fixed effects ignore heterogeneity – ignoring may mean an intervention effect that does not actually exist </li></ul></ul>
  39. 39. What to do with (Statistical) Heterogeneity <ul><li>Random effects meta-analysis </li></ul><ul><ul><li>Incorporates heterogeneity but is not a substitution for a thorough investigation </li></ul></ul><ul><li>Exclude studies </li></ul><ul><ul><li>Sensitivity analysis </li></ul></ul>
  40. 40. Subgroup and Meta-regression <ul><ul><li>Chapter 9 </li></ul></ul><ul><ul><li>Observational in nature </li></ul></ul><ul><ul><li>Characteristics used should be prespecified; keep to a minimum </li></ul></ul><ul><ul><li>Conclusions from such analyses should be interpreted with caution </li></ul></ul><ul><ul><li>Subgroups: splitting all studies into groups to make comparisons </li></ul></ul><ul><ul><li>Meta-regression: extension of subgroup analysis, allows investigation of continuous and categorical variables </li></ul></ul>
  41. 41. Subgroup Analysis
  42. 42. Sensitivity Analysis <ul><li>Chapter 9 </li></ul><ul><li>Addresses the question: Are the findings robust to the decisions make in the process of obtaining them? </li></ul><ul><li>Repeats the primary analysis and substitutes alternative decisions for decisions or range of values that were arbitrary or unclear </li></ul><ul><li>Some can be prespecified in the protocol but many issues are identified only during the review process </li></ul><ul><li>Don’t confuse with subgroup analysis </li></ul>

×