Why critically appraise?
• Important element of evidence based
medicine
• To weigh up how valid and useful the
research will be
• Research is of variable quality
• Only an estimated 1% is judged
clinically relevant
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
What to look for in an RCT
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
Is the methodology sound?
Publication bias in systematic
reviews
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
Papers with more “interesting” results are
more likely to be:
• Submitted and accepted for publication
• Published in a major journal
• Published in English
• Quoted by authors
• Quoted in newspapers
Have the authors addressed this in their
review?
How are the results presented?
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
• Number needed to treat (NNT)
The number of patients that need to
be treated with en intervention in
order to get one additional positive
outcome. E.g. if the NNT is 4, four
people would need to treated with the
intervention in order for one to gain
benefit.
How are the results presented?
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
• Number needed to treat (NNT)
• Odds Ratio
• Relative risk
Odds and risk
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
Odds of winning
1:9
You versus the rest
Risk of winning
1:10
You versus all the
runners
10 horses
running, you bet on
1 horse
How are the results presented?
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
• Number needed to treat (NNT)
• Odds Ratio
• Relative risk
• Mean difference
This one is fairly self explanatory. It’s
the difference between the mean
values in the treatment group and the
control group.
How are the results presented?
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
• Number needed to treat (NNT)
• Odds Ratio
• Relative risk
• Mean difference
Clarity of the results is key. Are the
results presented clearly?
Forest plots
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
more than 1less than 1 1
A forest plot (or blobbogram)
is often used in systematic
reviews or meta-analyses to
display data from a number
of individual studies
Forest plots
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
more than 1less than 1 1
Line of no effect
Anything to the right of the line
of no effect indicates more of
the effect, anything to the left
is less.
If the outcome is positive you
want to see more of the results
on the right of the graph to
show that the intervention is
effective.
Forest plots
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
more than 1less than 1 1
Line of no effect
Best estimate
Based on an average, the best
estimate is the point at which the
true effect of the treatment is
most likely to lie
Forest plots
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
more than 1less than 1 1
Line of no effect
Best estimate
Confidence interval
Confidence intervals show
the range in which the true
effect of the treatment
could lie.
If the confidence interval
crosses the line of no effect
the result of the treatment
is not statistically significant.
Forest plots
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
more than 1less than 1 1
Line of no effect
Best estimate
Pooled result
Confidence interval
A meta-analysis will give a
pooled result combining the
results of the individual
studies
P-value
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
Could the result have occurred
by chance?
p = 0.001 (1 in 1000)
p = 0.2 (1 in 5)
A p-value of less than 0.05 (1 in 20) is
considered to be statistically significant
How critical appraisal works
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
• Involves answering a short
questionnaire
• We use the CASP questionnaires at
http://www.casp-uk.net/
• Questionnaires available for different
types of research
Summary
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
Validity
Is it
trustworthy?
Results
What does
it say?
Relevance
Will it help?
Further reading
RDF
InformationLiteracy
http://www.library.qmul.ac.uk/
Critical appraisal questionnaires / checklists
• http://www.casp-uk.net/
• http://www.cebm.net/index.aspx?o=1157
• http://www.sign.ac.uk/methodology/checklists.html
General information on critical appraisal
• http://www.medicine.ox.ac.uk/bandolier/painres/downloa
d/whatis/what_is_critical_appraisal.pdf
• http://www.nature.com/nrgastro/journal/v6/n2/full/ncpgast
hep1331.html
Be the first to comment