Handling Bias in Medical Research
Acknowledge
• Cochrane Collaboration’s Bias Methods Group
Overview
• Impact of bias in medical research
• Introduction to bias and systematic error
• Sources of bias in the product...
Potential Impact of bias
The impact that biased estimates of effect have on research is
perhaps obvious but should not be ...
What is bias?
• A bias is a systematic error, or deviation from the truth,
in results or inferences
If you were to repeat ...
Common misconceptions
• Bias may be distinguished from methodological quality
• Although the highest standards of conduct ...
Published
Conceived
Performed
Submitted
Cited
Research agenda bias
Sources of bias in the production and
dissemination of ...
Sackett J Chron Dis 1979
Published
Conceived
Performed
Submitted
Cited
Sources of bias in the production and
dissemination...
Published
Conceived
Performed
Submitted
Cited
Sources of bias in the production and
dissemination of evidence
Reporting Bi...
Further evidence of the presence of bias
In 2010, 235 biases were mapped and characterised
17,265,924 PubMed items were cl...
0.2 0.4 0.6 0.8 21
Meta-analysis
Single large trial
Odds Ratio
(95% Confidence Intervals)
Nitrates in myocardial infarctio...
Meta-epidemiology
• Identify a large number of meta-analyses
• Record characteristic(s) of individual studies (e.g., degre...
Meta-epidemiological studies: meta-meta-analytic method
Pooled ORs calculated for trials with and
without methodological f...
Empirical evidence of bias: RCTs
Schulz JAMA 1995
0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2
No
Yes
Double-blind
Yes
No
Exclusions
In...
Prominent biases in RCTs
Bias Components Evidence
Selection bias Sequence generation
Allocation Concealment
Schulz 1995, M...
Methods to assess bias in primary studies
Cochrane Risk of Bias Tool (RCTs)
Downs and Black (NRS)
Newcastle- Ottawa Scale ...
Single-centre versus multi-centre trials
• New potential (meta)bias (Dechartres, A. 2011)
26% larger effect size in single...
A new concept: Meta-bias
“Meta-epidemiological studies, examining many meta-
analyses on diverse topics, have been finding...
Publication Bias
• This concept is not new (Sterling, TD. 1959)
• Well accepted and more recent processes for assessing
as...
McAuley 2000
Egger 2003
Published vs. unpublished
0.91 (0.85 to 0.97)
Ratio of odds ratios
0.4 0.5 0.6 0.7 0.8 0.9 1 1.2 1...
Funnel plots
• If all studies come from a single underlying
population, this graph should look like an
inverted funnel, wi...
Funnel plot: no evidence of bias
StandardError
Odds ratio
0.1 0.3 1 3
3
2
1
0
100.6
Funnel plots
• In the presence of publication bias some studies
are missing
• asymmetry is present
Light RJ, Pillemer DB. ...
0.1 0.3 1 3 100.6
Asymmetrical
Funnel Plot
Reporting bias present
Odds ratio
StandardError
3
2
1
0
“Small study effect”
There is a tendency for smaller trials in a
meta-analysis to show greater treatment
effects than the ...
Selective reporting
• This concept is not new (Hemminki 1980)
• Recent evidence that this remains a serious issue (Kirkham...
Global attempts to minimise bias:
Trial registration
In 2004 the International committee of Medical Journal
Editors (ICMJE...
• Clinical trials.gov
• WHO portal
Global attempts to minimise bias:
Trial registration
This is a great start, but with the evidence mounting
suggesting that...
Registering Systematic Review Protocols:
An International Initiative
Vancouver, BC, Canada
February 18, 2011
• http://www.crd.york.ac.uk/prospero/
• Launched Feb 2011
• 400+ registered protocols from 27 countries
Thank you for your attention
David Moher: dmoher@ohri.ca
Upcoming SlideShare
Loading in …5
×

David Moher - MedicReS World Congress 2012

454 views
368 views

Published on

Handling Bias in Medical Research
David Moher
Cochrane Collaboration’s Bias Methods Group

Published in: Health & Medicine
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
454
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

David Moher - MedicReS World Congress 2012

  1. 1. Handling Bias in Medical Research
  2. 2. Acknowledge • Cochrane Collaboration’s Bias Methods Group
  3. 3. Overview • Impact of bias in medical research • Introduction to bias and systematic error • Sources of bias in the production and dissemination of evidence • Meta-epidemiology and empirical evidence of bias • Means to assess bias in primary studies • How does this lead to good medical research?
  4. 4. Potential Impact of bias The impact that biased estimates of effect have on research is perhaps obvious but should not be understated Bias leads to: Inaccurate estimate of benefit and harms for an intervention Biases can influence the results of secondary research systematic reviews Both bias and “spin” can directly influence what care patients receive and contribute to quality of care
  5. 5. What is bias? • A bias is a systematic error, or deviation from the truth, in results or inferences If you were to repeat the study over and over again the results would always be inaccurate • Biases can vary in magnitude and direction • There are no current, well accepted means to estimate magnitude of bias
  6. 6. Common misconceptions • Bias may be distinguished from methodological quality • Although the highest standards of conduct may have been implemented the results still may not be believable • Bias should not be confused with impression due to random error (sampling variation) In systematic reviews the quality of reporting of trial reports is fundamental to assessing potential biases (www.consort-statement.org)
  7. 7. Published Conceived Performed Submitted Cited Research agenda bias Sources of bias in the production and dissemination of evidence
  8. 8. Sackett J Chron Dis 1979 Published Conceived Performed Submitted Cited Sources of bias in the production and dissemination of evidence Methodological quality Attrition bias Misclassification bias Berkson bias Missing clinical data bias Confounding bias Non blind diagnosis bias Centripetal bias Non blind test interpretation bias Diagnostic access bias Non-respondent bias Diagnostic purity bias Partial verification bias Diagnostic suspicion bias Performance bias Diagnostic vogue bias Prevalence-incidence bias Exposure suspicion bias Recall bias Family information bias Referral filter bias Interviewer bias Selection bias Membership bias Unmasking bias
  9. 9. Published Conceived Performed Submitted Cited Sources of bias in the production and dissemination of evidence Reporting Biases: Publication Bias Time Lag Bias Language Bias Multiple Publication Bias Outcome reporting bias Citation bias
  10. 10. Further evidence of the presence of bias In 2010, 235 biases were mapped and characterised 17,265,924 PubMed items were clustered by biases and other text, use of terms were mapped over time Forty bias terms were used in the title or abstract of more than 100 articles each; confounding, response bias, sampling bias, observer bias, cognitive bias, race bias and perceptual bias all occur in the literature before 1965; There were shown to be bias ‘cliques’, co-appearing biases and the evolution of terminology was mapped over time Chavalarias, Ioannidis 2010, J Clin Epi
  11. 11. 0.2 0.4 0.6 0.8 21 Meta-analysis Single large trial Odds Ratio (95% Confidence Intervals) Nitrates in myocardial infarction Inpatient geriatric assessment Magnesium in myocardial infarction Aspirin for prevention of pre-eclampsia Intervention: Egger BMJ 1997 Differences are apparent between trials
  12. 12. Meta-epidemiology • Identify a large number of meta-analyses • Record characteristic(s) of individual studies (e.g., degree of allocation concealment, type of publication, language etc.) • Compare treatment effects within each meta-analysis (e.g., adequate allocation concealment versus inadequate) • Estimate ratio of odds ratios comparing high quality and low quality trials (“meta-meta-analysis”) (Naylor, BMJ 1997; 315: 617-619
  13. 13. Meta-epidemiological studies: meta-meta-analytic method Pooled ORs calculated for trials with and without methodological flaw within each meta- analysis, and difference calculated as Ratio of ORs (ROR): ROR = OR with flaw / OR without flaw The RORs are then meta-analysed to estimate overall ROR across all meta-analyses and the between MA variability in bias Flawed studies show more benefit Flawed studies show less benefit ROR ROR (95% CI) ROR ROR
  14. 14. Empirical evidence of bias: RCTs Schulz JAMA 1995 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 No Yes Double-blind Yes No Exclusions Inadequate/unclear Adequate Sequence generation Inadequate/unclear Adequate Concealment of allocation Ratio of Odds Ratios (95% CI) Treatment effect over-estimation under-estimation (favoring experimental intervention) Reference 33 meta-analyses, 250 RCTs
  15. 15. Prominent biases in RCTs Bias Components Evidence Selection bias Sequence generation Allocation Concealment Schulz 1995, Moher 1998, Kjaergard 2001, Siersma 2007; Schulz 1995a Jüni 2001 Performance bias Blinding of participants and personnel Pildal 2007, Wood 2008, Boutron 2006 Detection bias Blinding of outcome assessment Pildal 2007, Wood 2008, Boutron 2006 Attrition bias Incomplete outcome data Tierney 2005, Melander 2003, Porta 2007 Reporting bias Selective reporting Publication bias Hutton 2000, Hahn 2002, Williamson 2005, Chan 2004, Dwan 2011 Rosenthal 1979, Sterling 1959, Scherer 2007, Easterbrook 1991, Dickersin 1993, Hopewell 2008
  16. 16. Methods to assess bias in primary studies Cochrane Risk of Bias Tool (RCTs) Downs and Black (NRS) Newcastle- Ottawa Scale (NRS) MINORS NIH criteria Zaza et al. QUADAS (DTA) PEDro Oxford Centre of evidence-based medicine Deschartres 2011 J Clin Epi
  17. 17. Single-centre versus multi-centre trials • New potential (meta)bias (Dechartres, A. 2011) 26% larger effect size in single-centre trials (adjusting for trial size) • “Is conduct at a single-centre a surrogate for a process measure that would explain the result?” • But which class of studies provides the “truth” ? • Could fall under the auspices of reporting bias: Single centre trials published less frequently More room to shape results positively in small teams
  18. 18. A new concept: Meta-bias “Meta-epidemiological studies, examining many meta- analyses on diverse topics, have been finding risk factors for bias in those meta-analyses- both in the methods that meta-analysts use and in the characteristics of studies included- that are not clearly associated with a bias producing process in an individual study We are calling the differences associated with such non- process-related factors a meta-bias.” (Goodman and Dickersin, 2011)
  19. 19. Publication Bias • This concept is not new (Sterling, TD. 1959) • Well accepted and more recent processes for assessing asymmetry of funnel plots (Sterne, JAC. 2011) • Impacts systematic review directly • Impacts clinical practice guidelines indirectly • Difficult to assess when k < 5
  20. 20. McAuley 2000 Egger 2003 Published vs. unpublished 0.91 (0.85 to 0.97) Ratio of odds ratios 0.4 0.5 0.6 0.7 0.8 0.9 1 1.2 1.4 1.6 1.8 2 Impact of publication bias
  21. 21. Funnel plots • If all studies come from a single underlying population, this graph should look like an inverted funnel, with the effect sizes homing in on the true underlying value as n increases. Light RJ, Pillemer DB. Summing up. The science of reviewing research. Harvard University Press, 1984.
  22. 22. Funnel plot: no evidence of bias StandardError Odds ratio 0.1 0.3 1 3 3 2 1 0 100.6
  23. 23. Funnel plots • In the presence of publication bias some studies are missing • asymmetry is present Light RJ, Pillemer DB. Summing up. The science of reviewing research. Harvard University Press, 1984.
  24. 24. 0.1 0.3 1 3 100.6 Asymmetrical Funnel Plot Reporting bias present Odds ratio StandardError 3 2 1 0
  25. 25. “Small study effect” There is a tendency for smaller trials in a meta-analysis to show greater treatment effects than the larger trials Small study effects need not result from bias Sterne et al. Journal of Clinical Epidemiology 2000
  26. 26. Selective reporting • This concept is not new (Hemminki 1980) • Recent evidence that this remains a serious issue (Kirkham et al PLoS ONE 2010) • New reviews from 3 consecutive issues Cochrane Library • 22% of 288 review/protocol pairings were discrepant in at least one outcome • Currently included in Cochrane RoB Tool; • Limited by the need to review trial protocols, the potential solution is registration mandated by funding agencies
  27. 27. Global attempts to minimise bias: Trial registration In 2004 the International committee of Medical Journal Editors (ICMJE) announced that they will no longer publish trials that were not registered at inception
  28. 28. • Clinical trials.gov • WHO portal
  29. 29. Global attempts to minimise bias: Trial registration This is a great start, but with the evidence mounting suggesting that registration is not optimal
  30. 30. Registering Systematic Review Protocols: An International Initiative Vancouver, BC, Canada February 18, 2011
  31. 31. • http://www.crd.york.ac.uk/prospero/ • Launched Feb 2011 • 400+ registered protocols from 27 countries
  32. 32. Thank you for your attention David Moher: dmoher@ohri.ca

×