This document discusses meta-analysis and research synthesis. It begins by explaining the differences between narrative and systematic literature reviews. It then discusses effect sizes and how they are used to quantitatively assess the magnitude of effects across studies, rather than just determining whether an effect exists. The document defines meta-analysis as a statistical tool that combines the results of multiple studies, usually from a systematic review, to produce a summary effect size. Examples of applications and techniques for analysis beyond just the summary effect size are provided, including investigating heterogeneity and performing subgroup and meta-regression analyses. Steps for conducting a meta-analysis are outlined at the end.

1. Meta-Analysis
and Research
Synthesis
Tomás de la Rosa Macías
tomas.rosa@uca.es
Postdoctoral Researcher
Psychobiology Area. Department of Psychology
Neuropsychopharmacology & Psychobiology Research Group PAIDI-
CTS-510
Cádiz - Spain
https://cts510.uca.es/

2. 1) Narrative and Systematic Reviews
2) What is a Effect Size?
3) What is a Meta-Analysis?
4) Examples of applications
5) Analysis beyond meta-analysis
6) How to meta-analyze
Index

3. From Narrative to Systematic reviews
A single study rarely is enough to make
conclussions about a phenomena. There is a
need for Research Synthesis of Primary Studies
• Specific Research Question
• Pre-established
inclusion/exclusion criteria
• Systematic search and data
extraction
• Broad and non-specific
Research Question
• No criteria for
included/excluded studies
• No set rules for search and
data extraction
Narrative Reviews Systematic Reviews

4. Comparing Between Studies: Towards Quantitative Assesment
”4 studies including data from 340
patients evaluated the effect of [...]
Study A, B and C found significant
diferences between control and
patients [...] D failed to found
significant effect.” or “75% of
studies found diferences”
Other relevant information
about the studies
Study A: p=0.011*
Study B: p<0.001**
Study C: p=0.006*
Study D: p=0.087
Systematic
Review of
Literature
Excluded
studies
Research
Question
Effect of neuropathic
pain on anxiety

5. Seems that neuropathic pain has an effect on
anxiety…but…Howmuch?
How many points inthe scale that measures
anxiety is the difference between Control and
Patients group?
We are looking for a quantitative answer.
NOT Is there a difference? BUT How much is
that difference?
Comparing Between Studies: Towards Quantitative Assesment
Study A: p=0.011*
Study B: p<0.001**
Study C: p=0.006*
Study D: p=0.087
Primary studies report graphics and a p-value.
Canwe measurethe magnitudeoftheeffect
withp-value?

6. CONTROL PATIENTS
H0 Ha
Can we measure the magnitude of the effect with p-value?
Using Effect Size—or Why the P Value Is Not Enough
(Sullivan & Feinn, 2012)
Mean
SD

7. Using Effect Size—or Why the P Value Is Not Enough
(Sullivan & Feinn, 2012)
d
CONTROL PATIENTS
H0
P-value
Z =
𝑑
𝑆𝐷
𝑛
Ha
Can we measure the magnitude of the effect with p-value?

8. Seems that neuropathic pain has an effect on
anxiety…but…Howmuch?
We are looking for a quantitative answer
How many points inthe scale that measures
anxiety are from Control to Patients group?
Comparing Between Studies: Towards Quantitative Assesment
Study A: p=0.011*
Study B: p<0.001**
Study C: p=0.006*
Study D: p=0.087
Primary studies report graphics and a p-value.
Canwe measurethe magnitudeoftheeffect
withp-value?
NO!
P-valueonlygivesyouinformation about
difference being≠0

9. H0 Ha
d
Comparing Between Studies: Towards Quantitative Assesment

10. Why we shouldn’t compare p-values between-studies?
H0 Ha
Cohen’s d
Differences between
means
Hedge’s g
Omega-squared (ω2)
Relation between
continuous variables
Pearson’s p
Dichotomus variables
Odds Ratio
Risk Ratio
Calculating and reporting effect sizes...(Lakens 2013)

11. What is a meta-analysis?
Statistical tool and graph that combines the
results of multiple studies, normally, within a
previous Systematic Review.
The main result is a summary/weighted/true
effect size across all studies

12. What is a meta-analysis?
Gene V. Glass coined the
term and formalized it in 1976
“Statistical significance is the least interesting
thing about the results. You should describe
the results in terms of measures of magnitude
–not just, does a treatment affect people, but
how much does it affect them.”
Gene V. Glass
Statistical tool and graph that combines the
results of multiple studies, normally, within a
previous Systematic Review.
The main result is a summary/weighted/true
effect size across all studies

13. Research Synthesis and the “Pyramid of Evidence”
Evidence Based Medicine (Cochrane Collaboration)
What is a meta-analysis?

14. Studies weight
Name os studies
Experimental
(Mean, SD)
Control
(Mean, SD) Studies Effect Size/CI95%
Summary/True Effect Size+CI95%
What is a meta-analysis? THE FOREST PLOT

15. Effect Size and p-value
Introduction to Meta-Analysis (Borenstein M. 2009)
Cohen’s d

16. Effect Size and p-value
Introduction to Meta-Analysis (Borenstein M. 2009)
Cohen’s d

17. Introduction to Meta-Analysis (Borenstein M. 2009)
Effect Size and p-value
Cohen’s d

18. Effect Size and p-value
C 0.01 ∞ <0.001
With suficient sample size, virtually any
effect could be significant
Untreated Treated
125
127
129
131
133
135
mmHg
**
N = 10000000
p<0.001
d = 1 mmHg
Drug for
hipertension trial
Cohen’s d

19. Single
administration
(24h)
Chronic
administration
Chronic
administration
+
Delay
Single
administration
(3h)
CCI/SNL rodents performance in Von
Frey/Acetone after administration of
Ketamine

20. How effective are second-Generation antipsychotic drugs? A meta-analysis of placebo-controlled trials
(Leucht S. 2009)

21. What to do after we have computed our summary effect?
The same as with primary studies, we sometimes need to lead with
variability (Heterogeneity, I2 ;τ2), trying to guess where it comes from
and avoid that it affects our conclusions.
Analysis beyond meta-analysis

22. Single
administration
(3h)
Some observed heterogeneity could be
explained by moderators
Subgroup analysis is to meta-analysis, what
posthoc is for ANOVA
Analysis beyond meta-analysis

23. When the moderators that
explain our heterogeneity are
continous, we need to perform a
Meta-Regression
Analysis beyond meta-analysis
(Berkey 1995)

24. When the moderators that explain
our heterogeneity are continous,
we need to perform a Meta-
Regression.
Latitude was significantly
associated with the effect size of
the study.
Analysis beyond meta-analysis
(Berkey 1995)

25. How to Meta-Analyze...
1. Select your Effect Size of Interest
2. Retrieve studies through a Systematic Review
3. Extract Mean+SD and n for each study
4. Performe Meta-Analysis and deal with heterogeneity (Subgroup Analysis,
Meta-Regression...).
5. Report your results and draw your conclusions

26. Tomás de la Rosa Macías
tomas.rosa@uca.es
Postdoctoral Researcher
Psychobiology Area. Department of Psychology
Neuropsychopharmacology & Psychobiology Research Group PAIDI-
CTS-510
Cádiz - Spain
https://cts510.uca.es/
Questions?
-Using Effect Size — or Why the P Value Is Not Enough (Sullivan
& Feinn, 2012)
-Introduction to Meta-Analysis (Borenstein M. 2009)
-How effective are second-Generation antipsychotic drugs? A
meta-analysis of placebo-controlled trials (Leucht S. 2009)
-Efficacy of ketamine in relieving neuropathic pain: a systematic
review and meta-analysis of animal studies (Velzen 2021)
-The handbook of research synthesis and meta-analysis (Cooper
2019)
-Calculating and reporting effect sizes to facilitate cumulative
science: a practical primer for t-tests and ANOVAs (Lakens
2013)
-The mass production of redundant, misleading, and conflicted
systematic reviews and meta-analysis (Ioannidis 2016)
SOFTWARE
CMA ($$, Not Reproducible analysis)
JASP (Open Source, R-based, Free)
JAMOVI (Open Source, R-based, Free)
SPSS 28.0 ($$)
Meta XL (Excel based, Free)