Your SlideShare is downloading. ×
2010 smg training_cardiff_day2_session4_sterne
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

2010 smg training_cardiff_day2_session4_sterne

868
views

Published on

Published in: Education, Business

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
868
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1Department of SOCIAL MEDICINE University of BRISTOL Risk of bias assessments: analysis and interpretation Jonathan Sterne Department of Social Medicine, University of Bristol, UK
  • 2. 2Department of SOCIAL MEDICINE University of BRISTOL Bias in the results of meta-analyses • Reporting biases – Publication bias – Selective reporting of outcomes – Fertile ground for statistical tests – Now, finally, addressed in the obvious way, by mandatory trial registration
  • 3. 3Department of SOCIAL MEDICINE University of BRISTOL
  • 4. 4Department of SOCIAL MEDICINE University of BRISTOL Bias in the results of meta-analyses • Reporting biases – Publication bias – Selective reporting of outcomes – Fertile ground for statistical tests – Now, finally, addressed in the obvious way, by mandatory trial registration • Biases resulting from flaws in trial conduct
  • 5. 5Department of SOCIAL MEDICINE University of BRISTOL 5 Randomised Controlled Trials (RCTs) • Simple idea … Eligible participants Randomise Intervention group Comparison group Assess outcomes Assess outcomes
  • 6. 6Department of SOCIAL MEDICINE University of BRISTOL 6 • Deceptively simple idea Eligible participants Randomise Intervention group Comparison group Assess outcomes Assess outcomes Bias can be introduced at all stages of the conduct of RCTs Randomised Controlled Trials (RCTs)
  • 7. 7Department of SOCIAL MEDICINE University of BRISTOL Flaws in the conduct of RCTs • Trials provide causal inferences about the effect of the intervention if we randomise sufficient individuals and avoid selection and performance biases • This can be undermined by: – Inadequate generation of randomisation sequence
  • 8. 8Department of SOCIAL MEDICINE University of BRISTOL Flaws in the conduct of RCTs • Trials provide causal inferences about the effect of the intervention if we randomise sufficient individuals and avoid selection and performance biases • This can be undermined by: – Inadequate generation of randomisation sequence – Inadequate concealment of allocation Problems with randomisation may cause selection bias, if participants or healthcare providers can predict treatment allocation
  • 9. 9Department of SOCIAL MEDICINE University of BRISTOL Flaws in the conduct of RCTs • Trials provide causal inferences about the effect of the intervention if we randomise sufficient individuals and avoid selection and performance biases • This can be undermined by: – Inadequate generation of randomisation sequence – Inadequate concealment of allocation – Inadequate blinding • Performance bias • Care of intervention and control groups not comparable • Detection bias • Measurement of outcomes not comparable
  • 10. 10Department of SOCIAL MEDICINE University of BRISTOL Flaws in the conduct of RCTs • Trials provide causal inferences about the effect of the intervention if we randomise sufficient individuals and avoid selection and performance biases • This can be undermined by: – Inadequate generation of randomisation sequence – Inadequate concealment of allocation – Inadequate blinding – Excluding patients, or analysing them in the wrong group
  • 11. 11Department of SOCIAL MEDICINE University of BRISTOL Including biased trials will cause meta- analyses to be biased
  • 12. 12Department of SOCIAL MEDICINE University of BRISTOL 0.2 0.4 0.6 21 Meta-analysis Single large trial Odds Ratio Nitrates in myocardial infarction Inpatient geriatric assessment Magnesium in myocardial infarction Aspirin for pre-eclampsia prevention Intervention Egger et al. BMJ 1997
  • 13. 13Department of SOCIAL MEDICINE University of BRISTOL Including biased trials will cause meta- analyses to be biased • An obvious solution is to score the quality of trials included in the meta-analysis • We could then downweight low quality trials, or exclude trials scoring below a chosen quality threshold
  • 14. 14Department of SOCIAL MEDICINE University of BRISTOL The death of quality scores • 25 known checklists • Between 3 and 34 components • Frequently no definitions of quality • Most components said to be based on “accepted criteria” (Moher et al. Controlled Clinical Trials 1995; 16: 62-73)
  • 15. 15Department of SOCIAL MEDICINE University of BRISTOL “Quality scores are useless and potentially misleading” Greenland Am.J.Epidemiol. 1994;140:290-296 “perhaps the most insidious form of subjectivity masquerading as objectivity is ‘quality scoring’. This practice subjectively merges objective information with arbitrary judgements in a manner that can obscure important sources of heterogeneity among study results”
  • 16. 16Department of SOCIAL MEDICINE University of BRISTOL Relative risk with 95% CI 0·5 0·75 1 1·25 QualityScale1-25 All trials 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Re-analysis using 25 different scales to assess trial quality Endpoint: Deep vein thrombosis Jüni et al. JAMA 1999 282: 1054-1060 High quality Low quality
  • 17. 17Department of SOCIAL MEDICINE University of BRISTOL Empirical evidence of bias Evidence-based critical appraisal
  • 18. 18Department of SOCIAL MEDICINE University of BRISTOL Meta-epidemiology (Naylor, BMJ 1997; 315: 617-619) • Identify a large number of meta-analyses • Record characteristics of individual studies (allocation concealment, blinding, type of publication, language etc.) • Compare treatment effects within each meta-analysis (for example not double blind vs. double blind) • Estimate ratio of odds ratios
  • 19. 19Department of SOCIAL MEDICINE University of BRISTOL Schulz KF, Chalmers I, Hayes RJ, Altman DG. (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273: 408-412. 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 No Yes Double-blind Yes No Exclusions Inadequate/unclear Adequate Sequence generation Inadequate/unclear Adequate Concealment of allocation Ratio of Odds Ratios Treatment effect over-estimation under-estimation Reference Empirical evidence of bias 33 meta-analyses, 250 RCTs
  • 20. 20Department of SOCIAL MEDICINE University of BRISTOL NOTE: Weights are from random effects analysis Overall (I-squared = 88.6%, p = 0.000) Kjaergard, 2001 Als-Neilsen, 2004 Moher, 1998 Study ID Balk, 2002 Egger, 2003 Schulz, 1995 0.79 (0.66, 0.95) 0.60 (0.37, 0.97) 1.02 (0.93, 1.13) 0.63 (0.45, 0.88) ratios (95% CI) 0.95 (0.83, 1.09) 0.79 (0.70, 0.89) 0.66 (0.59, 0.73) 0.79 (0.66, 0.95) 0.60 (0.37, 0.97) 1.02 (0.93, 1.13) 0.63 (0.45, 0.88) ratios (95% CI) Ratio of odds 0.95 (0.83, 1.09) 0.79 (0.70, 0.89) 0.66 (0.59, 0.73) .25 1 4.25 .5 1 2 Ratio of odds ratios Allocation concealment: combined evidence
  • 21. 21Department of SOCIAL MEDICINE University of BRISTOL Analysis of meta-epidemiological studies • Most studies used a logistic regression approach assuming fixed effects within and between meta-analyses – Assumes no between-trial heterogeneity, and that effects of bias are the same in each meta-analysis • Two-stage approach: – Estimate the effect trials characteristics separately in each meta- analysis – Combine estimates across meta-analyses (Sterne et al. Statistics in Medicine 2002; 21: 1513-1524)
  • 22. 22Department of SOCIAL MEDICINE University of BRISTOL Non blinded more beneficial Non blinded less beneficial 0.93 (0.83, 1.04) Ratio of odds ratios (95% CI) Overall (76) No. of trials* 314vs. 432 Comparison (No. of meta-analyses) * Non blinded vs. blinded Variability in bias (P value) 0.11 (p<0.001) The image cannot be displa… Ratio of odds ratios 0.5 0.75 1 1.5 2 1.01 (0.92, 1.10) 0.75 (0.61, 0.93)Subjective outcomes (32) Objective outcomes (44) 210vs. 227 104vs. 205 0.08 (p<0.001) 0.14 (p=0.001)T h The image cannot be displayed. Your computer may not have enough me… Combined analysis of three empirical studies: blinding
  • 23. 23Department of SOCIAL MEDICINE University of BRISTOL Inadequately concealed more beneficial Inadequately concealed less beneficial Ratio of odds ratios 0.5 0.75 1 1.5 2 Ratio of odds ratios (95% CI) 0.83 (0.74, 0.93)Overall (102) No. of trials* 532 vs. 272 Comparison (No. of meta-analyses) * Inadequately/unclearly concealed vs. adequately concealed 0.11 (p<0.001) The image cannot be displa… The image cannot be displayed. Your computer may not have enou 0.91 (0.80, 1.03) 0.69 (0.59, 0.82) Objective outcomes (62) Subjective outcomes (40) 310 vs. 174 222 vs. 98 0.11 (p<0.001) 0.07 (p=0.011) Variability in bias (P value) Combined analysis of three empirical studies: allocation concealment Wood, L., Egger, M., Gluud, L.L., Schulz, K., Jüni, P., Altman, D.G., Gluud, C., Martin, R.M., Wood, A.J.G. and Sterne, J.A.C. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ, 336: 601-605.
  • 24. 24Department of SOCIAL MEDICINE University of BRISTOL Inadequately concealed more beneficial Inadequately concealed less beneficial Ratio of odds ratios 0.5 0.75 1 1.5 2 Ratio of odds ratios (95% CI) 0.83 (0.74, 0.93)Overall (102) No. of trials* 532 vs. 272 Comparison (No. of meta-analyses) * Inadequately/unclearly concealed vs. adequately concealed 0.11 (p<0.001) The image cannot be displa… The image cannot be displayed. Your computer may not have enou 0.91 (0.80, 1.03) 0.69 (0.59, 0.82) Objective outcomes (62) Subjective outcomes (40) 310 vs. 174 222 vs. 98 0.11 (p<0.001) 0.07 (p=0.011) Variability in bias (P value) Combined analysis of three empirical studies: allocation concealment Wood, L., Egger, M., Gluud, L.L., Schulz, K., Jüni, P., Altman, D.G., Gluud, C., Martin, R.M., Wood, A.J.G. and Sterne, J.A.C. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ, 336: 601-605. Emprical evidence on the effect of flaws in trial conduct will always be limited by imperfect reporting of trials
  • 25. 25Department of SOCIAL MEDICINE University of BRISTOL Bias assessment in Cochrane reviews • Project led by Julian Higgins and Doug Altman • “Risk of bias”, not “quality” • Cochrane reviewers are now asked to judge whether there is a risk of bias in the results of the trial – “Yes”: high risk of bias – “No”: low risk of bias 1. Sequence generation (randomisation) 2. Allocation concealment 3. Blinding of participants, personnel and outcomes 4. Incomplete outcome data (attrition and exclusions) 5. Selective outcome reporting 6. Other (including topic-specific, design-specific)
  • 26. 26Department of SOCIAL MEDICINE University of BRISTOL 26 The ROB tool: how to assess items Two components 1. Description of what happened • possibly including ‘done’, ‘probably done’, ‘probably not done’ or ‘not done’ for some items 2. Review authors’ judgement • whether bias unlikely to be introduced through this item (Yes, No, Unclear) Yes = Low risk of bias No = High risk of bias Unclear = unable to make a clear judgement ‘Blinding’ and ‘Incomplete outcome data’ may need separate assessments for different outcomes
  • 27. 27Department of SOCIAL MEDICINE University of BRISTOL 27 Risk of bias table Study: Fisman 1981 Authors’ judgement Description Sequence adequately generated? UNCLEAR “Patients were randomly allocated”. Allocation concealed? UNCLEAR No information. Blinding? All included outcomes Low risk “double blind design”. “Millet… resembles lecithin in appearance… When ground, each substance could be distinguished from the other by hue and taste but staff were not informed as too which was which.” Incomplete outcome data addressed? All included outcomes High risk Data unavailable for meta-analysis. Randomised: lecithin = Not stated, placebo = Not stated, Total = 33. Missing: lecithin = 7 (non-cooperation or diarrhoea = 2; moved to nursing home = 4, death = 2), placebo = 5 (non-cooperation or diarrhoea = 3, death = 2), total missing = 36%.
  • 28. 28Department of SOCIAL MEDICINE University of BRISTOL 28 Study: Fisman 1981 Authors’ judgement Description Free of selective reporting? High risk No quantitative results reported due to lack of effect. It is apparently clear which outcomes were measured. Free of other bias? Low risk No problems apparent. Risk of bias table
  • 29. 29Department of SOCIAL MEDICINE University of BRISTOL Adequate sequence generation Allocation concealment Blinding (Patient-reported outcomes) Blinding (Mortality) Incomplete outcome data addressed (Short-term outcomes (2-6 wks)) Incomplete outcome data addressed (Longer-term outcomes (> 6ks)) Free of selective reporting Free of other bias Yes (Low risk of bias) Unclear No (High risk of bias) Summary of risk of bias for included trials Incorporating bias assessments into reviews So now I’ve dealt with bias in my review? No!
  • 30. 30Department of SOCIAL MEDICINE University of BRISTOL 30 Summarising risk of bias • Reviewers will need to do this: – for an outcome within a study (across bias domains) – for an outcome across studies (for a meta-analysis) • Outcome-level summaries should inform the choice of meta-analytic strategy for that outcome • Meta-analysis-level summaries should inform the interpretation (summary of findings)
  • 31. 31Department of SOCIAL MEDICINE University of BRISTOL Incorporating outcome-level bias assessments into meta-analyses 1. Present all studies and provide a narrative discussion of risk of bias – such an approach is discouraged because • Descriptions of bias are likely to be lost in discussion and conclusions • Results from studies at high risk of bias should be downweighted 2. Primary analysis restricted to studies at low (or low and unclear) risk of bias – Often only a small proportion of trials – Reviewers reluctant to discard information, 3. Present multiple analyses with equal prominence – Confusing for readers and decision-makers
  • 32. 32Department of SOCIAL MEDICINE University of BRISTOL Evaluation of use of ROB tool
  • 33. 33Department of SOCIAL MEDICINE University of BRISTOL Testing for bias within a meta-analysis is unlikely to help
  • 34. 34Department of SOCIAL MEDICINE University of BRISTOLOdds ratio .01 .1 1 10 100 No. of events Treatment Control Australia 1976 11/33 17/31 Europe 1974 6/110 8/113 Europe 1984 8/39 15/40 Germany 1989 2/16 2/16 Germany 1994 1/18 1/18 Hong Kong 1974 1/20 1/20 Japan 1977 4/47 0/41 Switzerland 1975 1/18 3/22 Taiwan 1997 2/21 2/19 USA 1979a 6/16 11/15 USA 1979b 4/7 5/8 USA 1987 27/75 36/76 USA 1994a 2/21 0/20 USA 1994d 6/25 1/14 USA 1996a 46/136 58/89 USA 1997 88/205157/218 Subtotal 215/807317/760 Canada 1977 6/22 9/28 Romania 1976 1/20 0/20 USA 1988 15/126 18/142 USA 1994b 12/37 21/34 USA 1996b 3/10 1/11 Subtotal 37/215 49/235 Overall 252/1022366/995 Example: Clozapine versus neuroleptic medication for schizophrenia Concealment inadequate (H) Concealment adequate (L)
  • 35. 35Department of SOCIAL MEDICINE University of BRISTOL Concealment adequate (L) Concealment inadequate (H) All 0.25 0.50 1.00 2.00 Treatment odds ratio (log scale) Example: Clozapine versus neuroleptic medication for schizophrenia ROR: 0.66 (0.31,1.41)
  • 36. 36Department of SOCIAL MEDICINE University of BRISTOLRatio (inadeqate v adequate concealment of allocation) .01 .05 .1 .5 1 2 5 10 Combined The effects of components of trial quality are usually imprecisely estimated in a single meta-analysis Little hope of adjusting for the effects of trial quality using only the information available in the meta- analysis Data from Schulz et al. (JAMA 1995)
  • 37. 37Department of SOCIAL MEDICINE University of BRISTOL Effects of flaws in the conduct of trials • Change in average intervention effect (bias) – the focus of most previous research • Between-meta-analysis variability in average effect of bias • Increases in between-trial heterogeneity • If we knew that lack of blinding always exaggerated intervention effects by 20% there would be no problem • Bias matters because its effects are uncertain
  • 38. 38Department of SOCIAL MEDICINE University of BRISTOL Consequences of flaws in trial conduct
  • 39. 39Department of SOCIAL MEDICINE University of BRISTOL Consequences of flaws in trial conduct How might we use evidence about the effects of flaws in trial conduct, from meta- epidemiological studies, to combine data from studies at high and low risk of bias in meta-analyses?
  • 40. 40Department of SOCIAL MEDICINE University of BRISTOL Consequences for a single study at high risk of bias • Correct the estimated effect of intervention, for the average bias associated with the flaw(s) in the trial • Increase the trial variance by adding: – the average increase in between-trial heterogeneity – the between-meta-analysis variance in average bias • So, trials at high risk of bias should be downweighted in meta-analyses
  • 41. 41Department of SOCIAL MEDICINE University of BRISTOL Concealment adequate (L) Concealment inadequate (H) All Bias-adjusted 0.25 0.50 1.00 2.00 Treatment odds ratio (log scale) Example: Clozapine versus neuroleptic medication for schizophrenia Relative weight (type H studies)=81% Relative weight (type H studies)=37% w=0.41 ROR: 0.66 (0.31,1.41)
  • 42. 42Department of SOCIAL MEDICINE University of BRISTOL Consequences for meta-analyses • Limits to the informational value of studies at high risk of bias: 1. Even a very large study at high risk of bias has minimum variance corresponding to the sum of variances of increase in heterogeneity and variance in bias 2. Even a meta-analysis of large studies at high risk of bias has minimum variance corresponding to the between-meta-analysis variance in average bias Given current knowledge, the best approach for Cochrane review authors is to restrict meta-analyses to studies at low (or low and unclear) risk of bias
  • 43. 43Department of SOCIAL MEDICINE University of BRISTOL Conclusions • Flaws in the conduct of randomized controlled trials are important because they increase uncertainty • If we want to include potentially biased evidence in a systematic review, then we should downweight and correct for bias, based on empirical evidence on its effects • The best currently available approach for Cochrane reviewers is to present a primary meta-analysis restricted to studies at low (or low and unclear) risk of bias – Bias assessments based on the Risk of Bias Tool seem widely accepted – Improvements to the Handbook, RevMan and training materials may be needed to improve incorporation of bias assessments in reviews and SoF