Critical Appraisal of Published
Dr. Tarek Amin
Professor of Public Health
• Every day …
– ~ 46 randomized clinical trials are published
– ~ 1000 new Medline articles
– ~ 6,000 new articles in biomedical journals
• Every year …
– ~ 3 million articles published in ~ 30,000
• Most research published in medical journals is
– T poorly done
– Insufficiently relevant to be clinically
• Besieged with too much information to keep
up to date.
• High quality information is often not easy to
Critical appraisal is not
o Negative dismissal of any piece of research
o Assessment on results alone
o Based entirely on statistical analysis
o Undertaken by experts only
?Why critically appraise
• To find out the validity of the study
– Are the methods robust?
• To find out the reliability of the study
– What are the results and are they credible?
• To find out the applicability of the study
– Is it important enough to change my practice?
?What do I need to know
o Awareness of study designs
o Levels of evidence
o CA checklists
o CA resources
Describe the evidence
Comparison with other evidence
Describe the evidence- 1
What relationship being evaluated and what hypothesis was
What were the exposure and the outcome variable?
What was the study design?
Case report, series
Prospective or retrospective cohort study
Cross sectional study or
Describe the evidence- 1
Definition of participants in terms of:
– Source populations
– Time frame
– Eligibility criteria
– Participation rates of the different groups compared
Summary of the main results:
What is the result in terms of association between exposure
Should be possible to express the main result in a simple table
and obtain from the paper the means to calculate the
appropriate measure of association.
2- Internal Validity
o The truthfulness of inferences about the study
o Causal relationship between exposure and
outcome or just an association?
Internal Validity- 2
Two aspects of internal validity
1. Non-causal explanations
2. Causal explanations
Smoking is a risk factor for cancer of the larynx
• we’ d like to quantify the strength of the
association between smoking and laryngeal
cancer, but …
• many smokers are also drinkers (which is also
a risk factor for cancer of the larynx)
• drinking is said to confound the association
between smoking and risk of laryngeal cancer
(Confounding (the formal definition
• The effect of an extraneous variable that
wholly or partially accounts for the apparent
effect of the study exposure, or masks an
underlying true association
A variable is confounder
A variable is a confounder if:
1. It is causally associated with the outcome;
2. It is non-causally associated with the
3. It and the exposure variable are on two
separate causal pathways
A confounding variable is associated with the
exposure and it affects the outcome, but it is not an
intermediate link in the chain of causation between
.exposure and outcome
A relationship between exposure and outcome
identified by chance?
Type I error: null hypothesis is rejected when,
in reality, it is true.
The order of these non-causal explanations is
o Observation (information) bias, analytical
manipulation of the data will not overcome the
o Confounding, then appropriate analysis will
(in most cases) overcome the problem
Five aspects of causal explanations
Is there a correct temporal relationship?
Is the relationship strong?
Is there a dose-response relationship?
Consistency of the association?
Specificity of association
Is there a correct temporal . 1
o The exposure must act before the outcome
o No problem with prospective study designs
o Difficult in retrospective studies
?Is the relationship strong. 2
Larger relative risks (and Odds) are more
likely to reflect causal relationships.
Is there a dose-response. 3
The greater the exposure, the greater the risk of
Consistency of the association. 4
Expected to apply across a wide range of
An association identified in one study that is
consistent with the same association identified
in a different groups of subjects.
Specificity of association. 5
Specificity: exposure produces a specific
outcome (e.g. asbestos and mesothelioma)
External Validity- 3
External validity: can the results be applied to
?populations other than that which was studied
• If the internal validity of a study is poor, the
answer is no
Aspects of external validity:
1. Applied to the eligible population?
2. Applied to the source population?
3. Applied to other relevant populations?
1. Can the results be applied to the eligible population?
– The relationship between the study population (the
population from which samples are taken) and the
eligible population (those that met the study inclusion
criteria but did not take part) should be well
– Non-participation have to be considered carefully as
they are likely to be non-random.
Can the results be applied to the. 2
Whether the association between outcome and
exposure given by the study participants is
likely to apply to other groups
Can the results be applied to other. 3
The difficulties of applying results from one
group of subjects to another will be minimal
for issues of basic physiology and maximal for
effects in which cultural and psycho-social
aspects are dominant
Comparison with other evidence- 4
. Useful to consider a hierarchy of evidence
1. Randomized [clinical] trials
2. Cohort and case-control studies
3. Other comparative studies
4. Descriptive studies, case series, case studies,
Three aspects of comparison should be
1. Results consistent with other evidence?
2. Results plausible biologically?
3. Coherency with the existing knowledge.
?Are the results consistent with other evidence
• Most important characteristic used in the
judgment that an association is causal
• Lack of consistency argues against causality
Is the observed association biologically
• An association is regarded as coherent if it fits
the general features of the distribution of both
the exposure and the outcome under
3. External validity
• – can the results be applied to the eligible population?
• – can the results be applied to the source population?
• – can the results be applied to other relevant populations?
4. Comparison of the results with other evidence
• – are the results consistent with other evidence?
• – are the results plausible biologically?
• – is there coherency with the distribution of the exposure and the
• Can we apply these results to other populations? Are the findings
reported here consistent with other studies that looked at the
– Users’ Guides to Evidence-Based Practice
– A Student’ s Guide to the Medical Literature
– Pearls for Residents: Annotated Critical Appraisal
• Critical Appraisal of Bio-medical Literature
• Critical Appraisal Resources for Assessing Health
and Medical Research
• Bandolier http://www.medicine.ox.ac.uk/bandolier/