This document discusses meta-analysis and its use and limitations in synthesizing data from multiple studies on a research question. It notes that while meta-analysis provides an objective means of synthesis, it is still susceptible to biases depending on how it is conducted. Key steps in performing a rigorous meta-analysis are outlined, including having a clear research question, documenting literature search methods, extracting study details, assessing heterogeneity and publication bias, and exploring potential moderators of findings. Concerns raised decades ago about the potential for meta-analyses to be "gamed" remain important to consider.

1. Meta-analysis: Mega-
silly or mega-useful?
- Hans Eysenck

2. Research question: Is there
an abnormal cytokine profile
in autism?

3. Yes - 2 studies No - 6 studies
Study 1 Study 3
Study 2 Study 4
Study 5
Study 6
Study 7
Study 8
6 studies indicate ‘no’ so should we conclude there’s no
abnormal cytokine profile in autism?

4. Yes No
Study 1 (n = 200;
clinical diagnosis)
Study 3 (n = 10; self-report)
Study 2 (n = 100;
clinical diagnosis
Study 4 (n = 8; self-report)
Study 5 (n = 13; self-report)
Study 6 (n = 5; self-report)
Study 7 (n = 15; self-report)
Study 8 (n = 17; self-report)
Big differences in study quality but are the 2 ‘yes’ studies
worth more than then 6 ‘no’ studies?

5. Meta-analysis is an objective
and transparent technique to
synthesise data from a
number of related studies.

6. Doing a meta-analysis isn’t
particularly hard, it’s just hard
work.

7. How do you interpret a meta-
analysis?

8. It’s very easy for others to
‘game’ a meta-analysis to get
the outcome they want -
watch out for this.

9. 9 Circles of scientific hell
‘Sins’ that can
influence the
data in your
meta-analysis
Sins that are
often
overlooked in
meta-analysis

10. This letter from Eysenck put meta-analysis on the map -
his concerns are still valid today.

11. 1. How did they search
for articles?

12. 2. What was their
inclusion criteria?

13. Garbage in, garbage out

14. 3. Were the studies
homogenous?

15. 4. Did they
account for
publication
bias?

16. 4. Do the authors have an
agenda to push? Conflicts of
interest even more important
here.

17. How do you do a meta-
analysis?

19. If you want the theory read
these two books

20. 1. Have a good research
question
•Is there a debate in the literature?
•Perhaps a research question is settled but you want to look
at a moderator

21. 2. Pilot your search terms
•Too broad and you’ll be swamped, too narrow and you’ll
miss papers
•Use relevant databases (Pubmed + Embase will have you
covered)
•Also a good ‘feasibility’ check

22. 3. Document everything!
•Can someone reading your paper recreate your analysis?
•This makes your analysis transparent

23. 3. Extract the data
•Can help having an ‘data extraction’ form where you enter
important study details
•Gold standard is having 2 people do this and a third
adjudicating any disagreement

24. There’s a few software
packages you can use;
• Comprehensive meta-analysis (recommended)
• R packages (tricky but more flexibility with figures)
• An excel spreadsheet that comes with Cumming
(2014)

25. You can extract almost any
data to create a common
effect size
• P-values and sample size
• Means and SDs
• Correlation coefficients (‘easiest’ meta-analysis)
• Still not enough info? Contact the author!
•Most authors oblige (it’s a citation!)
•Not likely they’ll have data if older than 10 years

26. The software/package will
calculate common effect
sizes (even if you’re
extracting different types of
data) and then calculate a
summary effect size

27. Forest plot
sub-summary effect
size (i.e., what the
overall impact of one
cytokine?
Overall effect size (i.e.,
what’s the summary of
ALL studies?)

28. Publication bias?
•Are there ‘missing’ studies?
•A scatterplot of standard error against individual effect size
•Large studies tend to have small SE (near top)
•There should be an even spread (especially near the bottom)
Should be
about 4 more
studies here

29. What happens if there’s bias?
•You can impute the missing studies and re-analyse
•If your overall conclusions don’t change with the inclusion of the
studies you’re in the clear
Imputed studies

30. Forest plot
Are these different?

31. Here you can get some
clues as to which factors
are driving a result (i.e., is
this due to one cytokine?)

32. Other common moderator
analyses
• Gender - is this only found in one gender?
• Age - is this stronger/weaker in older people?
• Study quality - what’s the effect of ‘bad’ studies?
• Different types of measures
• Clinical groups - e.g., Bipolar vs. schizophrenia

33. Meta-analysis is a better
approach than a
‘traditional’ narrative
review, in most cases.

34. It’s also possible to do
meta-analysis with brain
imaging data but this is for
another time

35. Questions?
If your thinking of
doing a meta-
analysis I’d be
happy to help!
http://xkcd.com/1447/