This document outlines five essential questions to ask when assessing the methodological quality of papers: 1) Was the study original? 2) Whom is the study about? 3) Was the design of the study sensible? 4) Was systematic bias avoided or minimized? 5) Was the study large enough and long enough to make the results credible? It discusses factors to consider for each question when evaluating a study's methods section such as sample size, duration of follow up, and completeness of follow up.

1. How to Read a Paper:
2 – Assessing the Methodological
Quality of Papers
Dr Luke Kane
May 2014
1

2. References
• This lecture was made from T Greenhalgh’s
papers in the BMJ Series “How to Read a
Paper”
• Greenhalgh, T. (1997) “How to read a paper”
British Medical Journal. Web, accessed April-
May 2014 at <http://www.bmj.com/about-
bmj/resources-readers/publications/how-
read-paper>
2

3. Outline
• Five questions to ask yourself when looking at
a paper
3

4. Five Essential Questions
• To decided whether methods used are valid:
1. Was the study original?
2. Whom is the study about?
3. Was the design of the study sensible?
4. Was systematic bias avoided or minimised?
5. Was the study large enough and/or long enough
to make the results credible?
4

5. Q1: Was the study original?
• Only a very small amount of research is done that
is entirely “new”
• You need to ask if the study adds to the literature
• For example:
– Is it bigger or longer than previous ones?
– Is the methodology more rigorous?
– Does it address methodological criticisms of previous
ones?
– Will the results add to a meta-analysis?
– Is the population studied different? Ages? Sex?
5

6. Q2: Whom is the Study about?
• How were the subjects recruited?
– Is it likely to bias the results?
• E.g. how happy are you with services of hospital?
• If you place an ad in a newspaper asking people to
email in their answers then you are only sampling
people that read the newspaper and are motivated to
email you
• Better to ask 1 in 10 people in a waiting room for
example
6

7. Q2: Whom is the Study about?
• Who was included?
– Most trials in the West only include healthy young adults
– Often trials are run on people with severe forms of a
disease – e.g. end stage heart failure
– This could mean that the results are not valid for those
with milder forms
• Who was excluded?
– Many studies exclude people who are illiterate
– People with co-existing illness
– People taking other medications
• Is it fair to extrapolate this to patients taking multiple medications
with numerous diseases?
7

8. Q2: Whom is the Study about?
• Were the subjects studied in “real life”
circumstances?
– Trials can often be run very differently to how
people practise medicine in the real world
– i.e. special equipment
– More time, more information
8

9. Q3: Was the design of the Study
Sensible?
• Critical appraisal:
• Start with two fundamental questions
– What specific intervention was being considered
and what was it being compared with?
– What outcome was measured and how?
– This isn’t always what is written down!
– You need to work this out for yourself
9

10. Q3: Is the design sensible?
• What specific intervention was being
considered and what was it being compared
with?
– Don’t take the written statements at face value
– Authors often misrepresent what they did
– They often overestimate importance
– For example…
10

11. Q3: Is the design sensible?
11

12. Q3: Is the design sensible?
12

13. Q3: Is the design sensible?
13

14. Q3: Is the design Sensible?
• What outcome was measured, how?
– Is the outcome a sensible choice?
If not then why did they choose it?
– For example, the best way to see if a new drug works
is to see if it makes you live longer and whether the
quality of life it gives you is good
– If the trial is not based on the above outcome, what
and why?
– Is it based on a level of an enzyme in the blood which
the pharmaceutical company says is a good indicator
of survival?
14

15. Q4: Was Systematic Bias Avoided or
Minimsed?
• Systematic bias is anything that erroneously
influences the conclusions about groups and
distorts comparisons
• In any study the groups being compared must
be as similar as possible to each other
• Different types of study designs have different
steps to reduce bias:
15

16. Q4: Avoiding Bias in RCTs
• Importance
of random
allocation!
• Need to
check for
bias at
following
stages:
16

17. Q4: Avoiding Bias in Non-randomised
clinial trials
• You need to use common sense to decide if
differences between intervention and control
groups are likely to invalidate differences
ascribed to the intervention
• If non-randomised then "almost always" the
effects are biased
• e.g. testing a new drug between two groups:
one is 75% female, one is 30% female
17

18. Q4: Reducing Bias in Cohort Studies
• Reminder:
Cohort study
• Very difficult to
get two groups
who have same
age, sex,
socioeconomic
status etc with
only difference
the exposure
18

19. Q4: Reducing Bias in Cohort Studies
• In practice, much of the “controlling” in cohort
studies occurs at analysis stage
• Complex statistical adjustment is made for
baseline differences in key variables
• Unless this is done adequately, statistical tests
of probability and confidence intervals will be
dangerously misleading.
• Good example is alcohol and teetotallers

20. Q4: Reducing Bias in Case-Control
Studies
• Process most open to bias is not the
assessment of outcome, but the diagnosis of
the “cases” and the decision as to when the
individual became a case
20

21. Q4: Reducing bias, Was Assessment
Blind?
• Assessment bias from people who assess the
outcome if they know what group (treatment
or control) someone is in
• E.g. I knew that a patient had been
randomised to an active drug to lower blood
pressure rather than to a placebo, I might be
more likely to recheck a reading which was
surprisingly high.
21

22. Q5: : Were preliminary statistical
questions dealt with?
• Three important numbers can often be found
in the methods section of a paper:
1.the size of the sample;
2.the duration of follow up;
3.and the completeness of follow up.
22

23. Q5: Statistical Questions
• Sample size: How big?
• The sample needs to be big enough to have a
high chance of detecting a statistically
significant effect if one exists
• And therefore you can be reasonably sure that
no benefit exists if one is not found in the
study
23

24. Q5: Statistical Questions
• Sample size - how to calculate it
• Two things need to be decided:
• What level between the two groups would constitute a significant
effect?
• What is the mean and standard deviation of the primary outcome
variable
• Then with a statistical nomogram you can work out how large a sample
you need to have a moderate, high, or very high chance of detecting a
true difference between the groups—the power of the study
• It is common for studies to stipulate a power of between 80% and
90%.

25. Q5: statistical Questions, ERRORs
• Underpowered studies are common
• Usually because hard to recruit subjects
• Such studies typically lead to a type II or ß
error
• This is the erroneous conclusion that an
intervention has no effect.
• A type I or α error is the conclusion that a
difference is significant when in fact it is due
to sampling error

26. Q5: Stats, duration of follow up
• Even if sample size was adequate, a study must
continue long enough for the effect of the
intervention to be reflected in the outcome
variable
• A study looking at effect of a new painkiller on
degree of postop pain may only need a follow up
period of 48 hours
• Study of effect of nutritional supplementation in
the preschool years on final adult height, follow
up should be measured in decades

27. Q5:Stats, Completeness of Follow up
• Subjects who drop out ofstudies are less likely to have taken their
tablets as directed, more likely to have missed their checkups, and
more likely to have experienced side effects
• Reasons why patients withdraw:
• Incorrect entry of patient into trial (did not fulfil the entry criteria)
• Suspected adverse reaction to the trial drug
• Loss of motivation
• Loss of patient motivation
• Withdrawal by clinician for clinical reasons (such as concurrent
illness or pregnancy)
• Loss to follow up (patient moves away, etc)
• Death

28. Q5: completeness
• You still need to analyse the people who
dropped out
• This can be done on an intention to treat basis

29. Conclusion
• First essential question to ask about the methods
section of a published paper is: was the study
original?
• The second is: whom is the study about?
• Thirdly, was the design of the study sensible?
• Fourthly, was systematic bias avoided or
minimised?
• Finally, was the study large enough, and
continued for long enough, to make the results
credible?

30. References
• Bowers, D. (2008) Medical Statistics from Scratch: An
Introduction for Health Professionals. USA: Wiley-
Interscience.
• Grant, A. (2014) “Epidemiology for tropical doctors”.
Lecture (S6) from the Diploma of Tropical Medicine &
Hygiene, London School of Hygiene & Tropical
Medicine.
• Greenhalgh, T. (1997) “How to read a paper” British
Medical Journal. Web, accessed April-May 2014 at
<http://www.bmj.com/about-bmj/resources-
readers/publications/how-read-paper>
30