Tools for Assessing Risk of Bias
Doug Altman
Centre for Statistics in Medicine
University of Oxford
Acknowledgement
 Cochrane Collaboration Bias Methods Group
Systematic review
 A systematic review attempts to collate all empirical
evidence that fits pre-specified eligibility cri...
Key characteristics of a systematic
review
 A clearly stated set of objectives with an explicit,
reproducible methodology...
Taking account of study quality
 Methodological quality of the primary studies may
affect their results
 The influence o...
Quality scales and checklists
 Many exist (>40)
– vary hugely in complexity and scope
 Many combine trial quality with r...
The Cochrane Collaboration aims to help people
make well-informed decisions about healthcare by
preparing, maintaining and...
Cochrane library
8
Rethinking the approach to assessing
study quality
 2005: Cochrane Collaboration initiative
 Rationale: quality scores a...
Cochrane Risk of Bias (RoB) tool
 The Risk of Bias (RoB) tool was developed over
several years as a way to address concer...
Developing a Risk of bias tool
 In May 2005, 16 statisticians, epidemiologists, and
review authors attended a three day m...
The Risk of Bias Tool
12
http://www.cochrane-handbook.org/
13
BMJ 2011;343:d5928
Risk of bias
General principle of the RoB tool
 Avoidance of bias
Assessment of risk of specific types of bias

 Select...
15
Participants
Treatment A Comparator B
Outcome
Assessment
Outcome
Assessment
Detection bias
Selection bias
Attrition bia...
16
Population
Treatment A Comparator B
Outcome
Assessment
Outcome
Assessment
Detection bias
Selection bias
Attrition bias
...
Domains of the Cochrane
“Risk of Bias” tool
Review authors’ judgement:
 Was the allocation sequence adequately generated?...
Domains of the Cochrane
“Risk of Bias” tool
Review authors’ judgement:
 Was the allocation sequence adequately generated?...
The “Risk of Bias tool” (RoB)
General principles
19
 2 steps
 What was reported
 Extracted text: what was reported in t...
The “Risk of Bias tool” (RoB)
General principles. What was reported?
Sequence generation. Low
risk
Quote: “patients were r...
The “Risk of Bias tool” (RoB)
General principles. Judgment
 High risk of bias
 Bias of sufficient magnitude to have a no...
Attrition bias: Considerations
 How much data is missing from each group?
 Why is data missing in each group?
 How were...
Attrition bias
Low risk of bias
• No missing outcome data
• Reasons for missing data not related to outcome
• Missing data...
Attrition bias
High risk of bias
• Reason for missing data related to outcome, with
either imbalance in numbers or reasons...
The “Risk of Bias tool” (RoB)
General principles
 Reviewers specifically trained
 Independent duplicate assessment with ...
Selective outcome reporting
 Concern is that there is a tendency to selectively
report outcomes that are statistically si...
Risk of bias summary – each study
27
Risk of bias summary – all studies
28
Summary assessment of the risk of bias
across trials
Classification
Low risk of bias Most information is from trials at lo...
Risk of bias for non-randomised studies
Nonrandomised studies of interventions:
 Many issues are the same
 Main concern ...
Conclusions
 Assessing the risk of bias is an essential step for an
appropriate interpretation of systematic reviews and
...
Closing comments
 Good systematic reviews are a major source of
reliable information on the benefits and harms of
health ...
33
Upcoming SlideShare
Loading in …5
×

Doug Altman - MedicReS World Congress 2012

810 views

Published on

Tools for Assessing Risk of Bias
Doug Altman
Centre for Statistics in Medicine
University of Oxford

Published in: Health & Medicine
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
810
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
17
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Doug Altman - MedicReS World Congress 2012

  1. 1. Tools for Assessing Risk of Bias Doug Altman Centre for Statistics in Medicine University of Oxford
  2. 2. Acknowledgement  Cochrane Collaboration Bias Methods Group
  3. 3. Systematic review  A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria to answer a specific research question  It uses explicit, systematic methods that are selected with a view to minimising bias, thus providing reliable findings from which conclusions can be drawn and decisions made 3
  4. 4. Key characteristics of a systematic review  A clearly stated set of objectives with an explicit, reproducible methodology  A systematic search that attempts to identify all studies that would meet the eligibility criteria  An assessment of the validity of the findings of the included studies, such as through the assessment of risk of bias  Systematic presentation and synthesis of the characteristics and findings of the included studies [Liberati et al, BMJ 2007] 4
  5. 5. Taking account of study quality  Methodological quality of the primary studies may affect their results  The influence of study quality should routinely be examined – How?  Quality is difficult to define – design, conduct and analysis of a trial? – clinical relevance? – quality of reporting? – all of these? Focus on Randomised Controlled Trials (RCTs) 5
  6. 6. Quality scales and checklists  Many exist (>40) – vary hugely in complexity and scope  Many combine trial quality with reporting quality and other aspects (e.g. sample size)  Very popular: Jadad et al 1996 – Simple score from 0 to 5 – Serious flaws: combines methodology and reporting  Comparison of high vs low quality trials depends – Results depend strongly on which scale is used [Jüni et al, JAMA 1999] 6
  7. 7. The Cochrane Collaboration aims to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions (and diagnostic test accuracy) 7
  8. 8. Cochrane library 8
  9. 9. Rethinking the approach to assessing study quality  2005: Cochrane Collaboration initiative  Rationale: quality scores are not helpful  Focus should be on aspects with potential for bias, such as – concealment of treatment allocation – blinding of outcome assessment – handling of patient attrition 9
  10. 10. Cochrane Risk of Bias (RoB) tool  The Risk of Bias (RoB) tool was developed over several years as a way to address concerns about the existing method of assessing study “quality” in Cochrane reviews  Based on empirical evidence where possible  It is intended to lead to uniform approach across reviews from all Cochrane review groups 10
  11. 11. Developing a Risk of bias tool  In May 2005, 16 statisticians, epidemiologists, and review authors attended a three day meeting to develop the new tool  Identified items that were truly potential biases rather than sources of heterogeneity or imprecision  Seven areas were discussed leading to a proposed set of criteria for assessing protection from bias – as adequate, inadequate, or unclear 11
  12. 12. The Risk of Bias Tool 12 http://www.cochrane-handbook.org/
  13. 13. 13 BMJ 2011;343:d5928
  14. 14. Risk of bias General principle of the RoB tool  Avoidance of bias Assessment of risk of specific types of bias   Selection bias  Performance bias  Detection bias  Attrition bias Risk of bias summary 14
  15. 15. 15 Participants Treatment A Comparator B Outcome Assessment Outcome Assessment Detection bias Selection bias Attrition bias Bias Performance bias
  16. 16. 16 Population Treatment A Comparator B Outcome Assessment Outcome Assessment Detection bias Selection bias Attrition bias Bias Performance bias Randomisation Blinding of participants and personnel Blinding of outcome assessors
  17. 17. Domains of the Cochrane “Risk of Bias” tool Review authors’ judgement:  Was the allocation sequence adequately generated?  Was allocation adequately concealed?  Was knowledge of the allocated intervention adequately prevented during the study? – Participants – Trial personnel  Were incomplete outcome data adequately addressed?  Are reports of the study free of suggestion of selective outcome reporting?  Was the study apparently free of other problems that could put it at a high risk of bias? 17
  18. 18. Domains of the Cochrane “Risk of Bias” tool Review authors’ judgement:  Was the allocation sequence adequately generated?  Was allocation adequately concealed?  Was knowledge of the allocated intervention adequately prevented during the study? – Participants – Trial personnel  Were incomplete outcome data adequately addressed?  Are reports of the study free of suggestion of selective outcome reporting?  Was the study apparently free of other problems that could put it at a high risk of bias? 18 Separate assessment for each outcome
  19. 19. The “Risk of Bias tool” (RoB) General principles 19  2 steps  What was reported  Extracted text: what was reported in the published report / protocol/ contact with authors  Commentary  Judgment relating to the risk of bias  Low risk of bias  High risk of bias  Unclear (judgment is impossible)
  20. 20. The “Risk of Bias tool” (RoB) General principles. What was reported? Sequence generation. Low risk Quote: “patients were randomly allocated”. Comment: Probably done, since earlier reports from the same investigators clearly describe use of random sequences (Cartwright 1980). Blinding of participants and personnel (performance bias) Low risk Quote: “double blind, double dummy”; “High and low dose tablets or capsules were indistinguishable in all aspects of their outward appearance. For each drug an identically matched placebo was available”. Comment: Probably done Blinding of outcome assessment (detection bias) (Mortality) Low risk Quote: “Obtained from medical records” Comment: review authors do not believe this will introduce bias. etc 20
  21. 21. The “Risk of Bias tool” (RoB) General principles. Judgment  High risk of bias  Bias of sufficient magnitude to have a notable impact on the results  Unclear risk of bias  Insufficient details reported  Appropriate reporting, but the risk of bias is unknown 21
  22. 22. Attrition bias: Considerations  How much data is missing from each group?  Why is data missing in each group?  How were data analysed?  Handling of incomplete outcome data 22
  23. 23. Attrition bias Low risk of bias • No missing outcome data • Reasons for missing data not related to outcome • Missing data balanced across groups, with similar reasons • Missing data not enough to have a clinically relevant impact on the intervention effect estimate • Missing data have been imputed using appropriate methods 23
  24. 24. Attrition bias High risk of bias • Reason for missing data related to outcome, with either imbalance in numbers or reasons • Missing data enough to induce clinically relevant bias in intervention effect estimate • ‘As-treated’ analysis with substantial departure of the intervention received from that assigned at randomization • Inappropriate use of imputation 24
  25. 25. The “Risk of Bias tool” (RoB) General principles  Reviewers specifically trained  Independent duplicate assessment with consensus  Decisions need to be pre-specified in the protocol  Classification of outcomes (subjective / objective)  Blinding: successful blinding procedure  Missing data  Other risk of bias  Contact authors for missing information 25
  26. 26. Selective outcome reporting  Concern is that there is a tendency to selectively report outcomes that are statistically significant – Empirical evidence [Kirkham et al, BMJ 2010]  Challenges in detecting such behaviour  Ideally need study protocol  May be internal evidence from publications or trials register  Usually have to assess this item as “Unclear” 26
  27. 27. Risk of bias summary – each study 27
  28. 28. Risk of bias summary – all studies 28
  29. 29. Summary assessment of the risk of bias across trials Classification Low risk of bias Most information is from trials at low risk of bias Unclear risk of bias Most information is from trials at low or unclear risk of bias High risk of bias The proportion of information from trials at high risk of bias is sufficient to affect the interpretation of results 29
  30. 30. Risk of bias for non-randomised studies Nonrandomised studies of interventions:  Many issues are the same  Main concern is selection bias – how were treatment chosen?  High risk of non-comparable groups  Some tools exist – e.g. “Newcastle-Ottawa”  Cochrane Collaboration is planning to develop a RoB tool Other observational studies – e.g. prognosis, aetiology:  Parallel developments are needed30
  31. 31. Conclusions  Assessing the risk of bias is an essential step for an appropriate interpretation of systematic reviews and meta-analysis  Cochrane Risk of Bias tool includes 7 items to be evaluated for RCTs  Need for transparency  Training and use of the handbook recommendations  Similar approach should be adopted for other study types 31
  32. 32. Closing comments  Good systematic reviews are a major source of reliable information on the benefits and harms of health care interventions  A key aspect is assessment of risk of bias of the primary studies  Cochrane Risk of Bias tool includes 7 items to be evaluated for RCTs – A similar approach is needed for other study types  Should take account of risk of bias in the analysis and interpretation 32
  33. 33. 33

×