Assessing quality and bias in studies
Nicholas Ngwili, ILRI
Online Training on Systematic Literature Review, ILRI Nairobi, 5-6 May 2020

What is Quality in studies
Includes two complementary aspects
 Methodological quality
 Reporting quality
Not about value of the research or importance of
topic!

Methodological quality
• Relates to the design and conduct of research
• Is fundamental to understanding the results and the level of
confidence in the findings
• Largely based around assessment of the risk of bias-
selection, performance, detection, attrition, reporting and oth
er biases

Tools
• The Grading of Recommendations Assessment Development
and Evaluation (GRADE) systems used.
• Includes a risk of bias assessment but also assessment of;
o Consistency - consistent with those in the other included
studies
o Imprecision - confidence interval associated with a finding
o publication bias

Reporting quality
 How well a piece of scientific work is written-
whether the work can be replicated.
 Poor quality of reporting is common - makes
assessing methodology difficult
 Various approaches to improving the quality of
scientific reporting

Examples- guidelines on reporting
 Enhancing the QUAlity and Transparency Of health Research
(EQUATOR) Network specific for study design-
http://www.equator-network.org/
 Consolidated Standards of Reporting Trials (CONSORT)
Statement for RCTs
 STROBE - STrengthening the Reporting of
OBservational studies in Epidemiology

Are quality tools valuable?
 Available but not universally accepted- due to
over standardisation of published material.
 Challenging if you do not have all the required
information
 lack of external validation

Quality of qualitative research- points to
note
• Triangulation- compares the results from either
two or more different methods of data collection
• Respondent validation
• Clear exposition of methods of data collection
and analysis
• Sensitivity -Personal and intellectual biases

Summary /tips
 Look at the individual components of quality,
rather than attempting a binary categorization
into high or low quality
 Present in a transparent and understandable
way

What is bias?
Systematic error or deviation from the truth- introduced into sampling or
testing by selecting or encouraging one outcome or answer over others
o systematic reviews depend on included studies
o incorrect studies = misleading reviews
o should I believe the results?
o assess each study for risk of bias
o can’t measure the presence of bias
o Bias may overestimate or underestimate the effect
o look for methods shown to minimize risk

Bias is not the same as
 Imprecision - random error due to sampling variation - reflected in the
confidence interval
 Quality - bias can occur in well-conducted studies. Not all methodological
flaws introduce bias
 Reporting - good methods may have been used but not well reported

Assessing bias in observational studies
Type of bias
 Selection mechanisms in recruitment of study participants
(selection bias)- lack of randomness
 Selective recall or inconsistent data collection (information
bias), measurement errors
 Confounding

Errors
• Observational epidemiological studies measure associations
• Association ≠ Causation
• need to consider possible errors
• Systematic error- conclusion that are systematically different-
selection bias, information bias and confounding
• Random error

Bias in e.g cross-sectional studies
• Representativeness of study group
• Sample size
• Ascertainment of exposure
• Management of non respondents
• Comparability of exposed and non exposed
• Ascertainment of outcome
• Statistical test

Assessing for bias – chapter 8 of the
Cochrane handbook
 7 evidence-based domains
 review authors’ judgement
 Low risk of bias
× High risk of bias
? Unclear
 support for judgement
 evidence/quotes from the paper or other sources
 review author’s explanation

References
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC111
7321/pdf/50.pdf - Assessing quality in qualitative
research.
• https://training.cochrane.org/handbook
• Murphy E, Dingwall R, Greatbatch D, Parker S,
Watson P. Qualitative research methods in health
technology assessment: a review of the literature.
Health Technology Assessment 1998;2.
• Dingwall R, Murphy E, Watson P, Greatbatch D,
Parker S. Catching goldfish: quality in qualitative
research. Journal of Health Services Research and
Policy 1998;3:167-72

This presentation is licensed for use under the Creative Commons Attribution 4.0 International Licence.
better lives through livestock
ilri.org
ILRI thanks all donors and organizations which globally support its work through their contributions
to the CGIAR Trust Fund