1. Critical Appraisal of Published
Medical Research
Dr. Tarek Amin
Professor of Public Health
Cairo University

2. Background
• Every day …
– ~ 46 randomized clinical trials are published
– ~ 1000 new Medline articles
– ~ 6,000 new articles in biomedical journals
• Every year …
– ~ 3 million articles published in ~ 30,000
journals

3. Background
• Most research published in medical journals is
either
– T poorly done
oo
– Insufficiently relevant to be clinically
useful
• Besieged with too much information to keep
up to date.
• High quality information is often not easy to
find.

4. Critical appraisal is not
o Negative dismissal of any piece of research
o Assessment on results alone
o Based entirely on statistical analysis
o Undertaken by experts only

5. ?Why critically appraise
• To find out the validity of the study
– Are the methods robust?
• To find out the reliability of the study
– What are the results and are they credible?
• To find out the applicability of the study
– Is it important enough to change my practice?

6. ?What do I need to know
o Awareness of study designs
o Levels of evidence
o Statistics!!
o CA checklists
o CA resources

7. Roadmap
1.
2.
3.
4.
5.
Describe the evidence
Internal validity
External validity
Comparison with other evidence
Resources

8. Describe the evidence- 1
 What relationship being evaluated and what hypothesis was
tested?
 What were the exposure and the outcome variable?
 What was the study design?

Case report, series

Survey

Clinical trial

Case-control

Prospective or retrospective cohort study

Cross sectional study or
 Others

9. Describe the evidence- 1
 Definition of participants in terms of:
– Source populations
– Time frame
– Eligibility criteria
– Participation rates of the different groups compared
 Summary of the main results:
 What is the result in terms of association between exposure
and outcome?
 Should be possible to express the main result in a simple table
and obtain from the paper the means to calculate the
appropriate measure of association.

10. 2- Internal Validity
o The truthfulness of inferences about the study
population.
o Causal relationship between exposure and
outcome or just an association?

11. Internal Validity- 2
Two aspects of internal validity
1. Non-causal explanations
2. Causal explanations

12. Non-causal explanations
• Bias
• Confounding
• Chance variation

13. Non-causal explanations
• Bias
– Selection bias, surveillance, diagnosis,
referral, non-response, length of stay, survival
bias
– Misclassification bias: recall, interviewer,
improper analysis, etc.,

14. Confounding
Smoking is a risk factor for cancer of the larynx
• we’ d like to quantify the strength of the
association between smoking and laryngeal
cancer, but …
• many smokers are also drinkers (which is also
a risk factor for cancer of the larynx)
• drinking is said to confound the association
between smoking and risk of laryngeal cancer

15. Confounding definition
(Confounding (the formal definition
• The effect of an extraneous variable that
wholly or partially accounts for the apparent
effect of the study exposure, or masks an
underlying true association

16. A variable is confounder
A variable is a confounder if:
1. It is causally associated with the outcome;
and
2. It is non-causally associated with the
exposure; and
3. It and the exposure variable are on two
separate causal pathways

17. .Confounding
A confounding variable is associated with the
exposure and it affects the outcome, but it is not an
intermediate link in the chain of causation between
.exposure and outcome
Oral contraceptive
M
yocardial infarction
Smoking
IUD insertion
Salpingitis
ST
Ds

18. Chance variation
A relationship between exposure and outcome
identified by chance?
Type I error: null hypothesis is rejected when,
in reality, it is true.

19. Non-causal explanations
The order of these non-causal explanations is
:important
o Observation (information) bias, analytical
manipulation of the data will not overcome the
problem
o Confounding, then appropriate analysis will
(in most cases) overcome the problem

20. Five aspects of causal explanations
1.
2.
3.
4.
5.
Is there a correct temporal relationship?
Is the relationship strong?
Is there a dose-response relationship?
Consistency of the association?
Specificity of association

21. Is there a correct temporal . 1
?relationship
o The exposure must act before the outcome
occurs
o No problem with prospective study designs
o Difficult in retrospective studies

22. ?Is the relationship strong. 2
Larger relative risks (and Odds) are more
likely to reflect causal relationships.

23. Is there a dose-response. 3
?relationship
The greater the exposure, the greater the risk of
disease.

24. Consistency of the association. 4
Expected to apply across a wide range of
subjects.
An association identified in one study that is
consistent with the same association identified
in a different groups of subjects.

25. Specificity of association. 5
Specificity: exposure produces a specific
outcome (e.g. asbestos and mesothelioma)

26. External Validity- 3
External validity: can the results be applied to
?populations other than that which was studied
• If the internal validity of a study is poor, the
answer is no
Aspects of external validity:
1. Applied to the eligible population?
2. Applied to the source population?
3. Applied to other relevant populations?

27. 1. Can the results be applied to the eligible population?
– The relationship between the study population (the
population from which samples are taken) and the
eligible population (those that met the study inclusion
criteria but did not take part) should be well
documented.
– Non-participation have to be considered carefully as
they are likely to be non-random.

28. Can the results be applied to the. 2
?source population
Whether the association between outcome and
exposure given by the study participants is
likely to apply to other groups

29. Can the results be applied to other. 3
?relevant populations
The difficulties of applying results from one
group of subjects to another will be minimal
for issues of basic physiology and maximal for
effects in which cultural and psycho-social
aspects are dominant

30. Comparison with other evidence- 4
. Useful to consider a hierarchy of evidence
1. Randomized [clinical] trials
2. Cohort and case-control studies
3. Other comparative studies
4. Descriptive studies, case series, case studies,
clinical experience

31. Three aspects of comparison should be
:considered
1. Results consistent with other evidence?
2. Results plausible biologically?
3. Coherency with the existing knowledge.

32. ?Are the results consistent with other evidence
• Most important characteristic used in the
judgment that an association is causal
• Lack of consistency argues against causality

33. Plausibility
Is the observed association biologically
understandable?

34. Coherency
• An association is regarded as coherent if it fits
the general features of the distribution of both
the exposure and the outcome under
assessment.

35. Summary
1. Describe the evidence
• – type of study, outcome measure, population investigated,
• results
2. Internal validity
• – non-causal explanations
• • bias
• • confounding
• • chance
• – causal explanations
• • temporal relationship
• • strength of relationship
• • dose-response
• • consistency
• • specificity

36. Summary
3. External validity
• – can the results be applied to the eligible population?
• – can the results be applied to the source population?
• – can the results be applied to other relevant populations?
4. Comparison of the results with other evidence
• – are the results consistent with other evidence?
• – are the results plausible biologically?
• – is there coherency with the distribution of the exposure and the
• outcome?
• Can we apply these results to other populations? Are the findings
reported here consistent with other studies that looked at the
same thing?

37. Resources
Web sites
– Users’ Guides to Evidence-Based Practice
http://www.cche.net/usersguides/main.asp
– A Student’ s Guide to the Medical Literature
http://grinch.uchsc.edu/sg/
– Pearls for Residents: Annotated Critical Appraisal
References
http://www.cfpc.ca/English/cfpc/cme/pearls/pearls
%20residents/default.asp?s=1

38. • Critical Appraisal of Bio-medical Literature
http://www.shef.ac.uk/scharr/ir/units/critapp/resource
s.htm
• Critical Appraisal Resources for Assessing Health
and Medical Research
http://www.etsu.edu/health/index_files/harvill_hando
ut.pdf
• Bandolier http://www.medicine.ox.ac.uk/bandolier/

39. Thank you