Meta-analysis combines the results of multiple studies on a topic to increase power and improve estimates of the size of effects. A meta-meta-analysis examined over 7,500 primary studies and 54,000 effect sizes across 176 meta-analyses. It found small to medium cumulative effect sizes for factors like advertising, pricing, and consumer behavior, demonstrating how meta-analysis can advance knowledge by integrating vast amounts of empirical data. However, meta-analysis also faces challenges like heterogeneous studies, publication bias, and selecting only a non-random sample of a domain.
12. Effect of attitudes on behaviour?
• r=.5
• k=128, N=4,598
• Glasman & Albarracin (2006)
• stronger when
• attitudes are more accessible
• direct experience and/or reported frequently
• attitudes are more stable over time
• confident, based on bhvr relevant information, one-sided info
13. Effect of anti-depressants?
• Cohen’s d < .2
• for HDRS < 23 (mild or moderate), placebo vs ADM (anti-depressant
medication)
• HDRS scale runs 0 to 52
• pax treated with ADMs average 2 points higher on HDRS than
placebo!
• statistical significance vs clinical significance
• https://www.scientificamerican.com/article/antidepressants-do-
they-work-or-dont-they/
19. Meta-meta-analysis of advance in
knowledge
• 176 meta-analyses
• >7,500 primary studies (43 studies per meta-analysis)
published between 1918 and 2012
• >54,000 effect sizes (307 effect sizes per meta-analysis)
• sample of 8,337,096 subjects (based on primary studies of 131
meta-analyses)
• Eisend 2015 JM
26. • research designs
• preferential treatment of experiments (RCT)
• selection bias
• file drawer
• publication bias
• heterogeneity / moderator analyses
• non-random sample of domain
• fixed vs random effects
• collinearity
Contentious issues
Editor's Notes
10 cohort studies
5.8% risk background risk vs 6.9% risk if eating 50g of bacon a day !
+17% risk / 100g of red meat / day
cf men who smoke have 20x risk of lung cancer
SIMULATED 95% CIs of +/- 0.5%
what does it tell us if the CIs are smaller? larger?
statistically significant vs clinically (or practical) significance
statistical significance is a function of sample size, p-value, and effect-size (bigger effects are more likely to be “statistically significant”)
Tversky & Kahneman (1971) “Belief in the law of small numbers” Psych Bull
expt n=20 generated a significant result, z=2.23, p<.05
Well (1991) “The perils of N=1” JCR
Bass (1995) “Empirical generalizations in marketing science: a personal view” Mktg Science
McShane & Gal (2015) “Blinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence” Mgt Science
NHST encouraging dichotomous view
Original study effect size versus replication effect size (correlation coefficients). Diagonal line represents replication effect size equal to original effect size. Dotted line represents replication effect size of 0. Points below the dotted line were effects in the opposite direction of the original. Density plots are separated by significant (blue) and nonsignificant (red) effects.
the stage is set and the play requires hypotheses, test, conclusions
so we create hypotheses to explain the data - after they are collected !
Hamilton Depression Rating Scale (HDRS)
k=6, 718 patients
breakage rate of condoms is 2%
you have more chance of a condom failing than someone clicking on your banner ad!
highlighting that mktrs don’t control powerful effects, but are playing in the ultimate numbers game
the stage is set and the play requires hypotheses, test, conclusions
so we create hypotheses to explain the data - after they are collected !
Go fishing, let your suspicions run wild, choose those which have evidence, accept that there may be some false alarms
