This document discusses assessing quality and bias in studies. It defines quality as having two aspects: methodological quality, which relates to the design and conduct of the research, and reporting quality, which is how well the study is written to allow replication. Assessing bias focuses on potential systematic errors that could overestimate or underestimate effects. Several tools are used to evaluate risk of bias, consistency, precision, and publication bias. Guidelines also exist for transparent reporting in different study designs to improve quality. While quality assessment tools are available, they are not universally accepted and may lack external validation. Bias is a systematic error that should be evaluated in each study through domains like selection bias, information bias, and confounding.

1. Assessing quality and bias in studies
Nicholas Ngwili, ILRI
Online Training on Systematic Literature Review, ILRI Nairobi, 5-6 May 2020

2. What is Quality in studies
Includes two complementary aspects
 Methodological quality
 Reporting quality
Not about value of the research or importance of
topic!

3. Methodological quality
• Relates to the design and conduct of research
• Is fundamental to understanding the results and the level of
confidence in the findings
• Largely based around assessment of the risk of bias-
selection, performance, detection, attrition, reporting and oth
er biases

4. Tools
• The Grading of Recommendations Assessment Development
and Evaluation (GRADE) systems used.
• Includes a risk of bias assessment but also assessment of;
o Consistency - consistent with those in the other included
studies
o Imprecision - confidence interval associated with a finding
o publication bias

5. Reporting quality
 How well a piece of scientific work is written-
whether the work can be replicated.
 Poor quality of reporting is common - makes
assessing methodology difficult
 Various approaches to improving the quality of
scientific reporting

6. Examples- guidelines on reporting
 Enhancing the QUAlity and Transparency Of health Research
(EQUATOR) Network specific for study design-
http://www.equator-network.org/
 Consolidated Standards of Reporting Trials (CONSORT)
Statement for RCTs
 STROBE - STrengthening the Reporting of
OBservational studies in Epidemiology

7. Are quality tools valuable?
 Available but not universally accepted- due to
over standardisation of published material.
 Challenging if you do not have all the required
information
 lack of external validation

8. Quality of qualitative research- points to
note
• Triangulation- compares the results from either
two or more different methods of data collection
• Respondent validation
• Clear exposition of methods of data collection
and analysis
• Sensitivity -Personal and intellectual biases

9. Summary /tips
 Look at the individual components of quality,
rather than attempting a binary categorization
into high or low quality
 Present in a transparent and understandable
way

10. What is bias?
Systematic error or deviation from the truth- introduced into sampling or
testing by selecting or encouraging one outcome or answer over others
o systematic reviews depend on included studies
o incorrect studies = misleading reviews
o should I believe the results?
o assess each study for risk of bias
o can’t measure the presence of bias
o Bias may overestimate or underestimate the effect
o look for methods shown to minimize risk

11. Bias is not the same as
 Imprecision - random error due to sampling variation - reflected in the
confidence interval
 Quality - bias can occur in well-conducted studies. Not all methodological
flaws introduce bias
 Reporting - good methods may have been used but not well reported

12. Assessing bias in observational studies
Type of bias
 Selection mechanisms in recruitment of study participants
(selection bias)- lack of randomness
 Selective recall or inconsistent data collection (information
bias), measurement errors
 Confounding

13. Errors
• Observational epidemiological studies measure associations
• Association ≠ Causation
• need to consider possible errors
• Systematic error- conclusion that are systematically different-
selection bias, information bias and confounding
• Random error

14. Bias in e.g cross-sectional studies
• Representativeness of study group
• Sample size
• Ascertainment of exposure
• Management of non respondents
• Comparability of exposed and non exposed
• Ascertainment of outcome
• Statistical test

15. Assessing for bias – chapter 8 of the
Cochrane handbook
 7 evidence-based domains
 review authors’ judgement
 Low risk of bias
× High risk of bias
? Unclear
 support for judgement
 evidence/quotes from the paper or other sources
 review author’s explanation

16. References
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC111
7321/pdf/50.pdf - Assessing quality in qualitative
research.
• https://training.cochrane.org/handbook
• Murphy E, Dingwall R, Greatbatch D, Parker S,
Watson P. Qualitative research methods in health
technology assessment: a review of the literature.
Health Technology Assessment 1998;2.
• Dingwall R, Murphy E, Watson P, Greatbatch D,
Parker S. Catching goldfish: quality in qualitative
research. Journal of Health Services Research and
Policy 1998;3:167-72

17. This presentation is licensed for use under the Creative Commons Attribution 4.0 International Licence.
better lives through livestock
ilri.org
ILRI thanks all donors and organizations which globally support its work through their contributions
to the CGIAR Trust Fund