Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015

1. Mr. T (Tetsuya Sakai)
Statistical
Significance,
Power, and
Sample Sizes
A Systematic
Review of
SIGIR and TOIS,
2006‐2015
July 18@ACM SIGIR 2016,
Pisa, Italy.
α/2 α/2

2. TALK OUTLINE
•Motivation
•Survey Method
•Results
•Summary
•Data and Feedback

3. Preliminaries (1)
• In IR experiments, we often compare sample means to
guess if the population means are different.
• We often employ parametric tests (assume specific
population distributions and guess their parameters)
‐ paired and unpaired t‐tests (comparing m=2 means)
‐ ANOVA (comparing m (>2) means)
one‐way, two‐way, two‐way without replication
Are the two population
means equal?
Are the m population
means equal?
scores
EXAMPLE
n topics
m systems
Sample mean for a system

4. Preliminaries (2)
• H0: tentative assumption that all population means are equal
• test statistic: what you compute from observed data – under H0, this
should obey a known distribution (e.g. t‐distribution)
• p‐value: probability of observing what you have observed (or
something more extreme) assuming H0 is true
Null hypothesis
test statistic t0

5. Preliminaries (3)
Reject H0
if p‐value <= α
test statistic t0 t(φ; α)
Accept H0 Reject H0
H0 is true
systems are equivalent
Correct conclusion
(1‐α)
Type I error
α
H0 is false
systems are different
Type II error
β
Correct conclusion
(1‐β)
α/2 α/2
Statistical power:
ability to detect real
differences
Cohen’s five‐eighty convention:
α=5%, 1‐β=80%

6. Bad practices
• Reporting α but not the p‐value (dichotomous thinking)
“Significant at α=0.05” but p‐value=0.05? 0.000001?
• Reporting the p‐value but not the effect size
“p‐value=0.01” but does this reflect a large effect size
or just a large sample size (n)?
p‐value = f( sample_size, effect_size )
large effect size ⇒ small p‐value
large sample size ⇒ small p‐value
magnitude of the
actual difference
Anything can be made statistically significant by using lots of data

7. Statistical Reform in Information Retrieval?
http://sigir.org/files/forum/2014J/2014J_sigirforum_Article_TetsuyaSakai.pdf

8. http://tois.acm.org/authors.cfm
Do people actually follow these instructions? What about SIGIR?

9. Doubts
• Some IR experiments are extremely overpowered –
everything is statistically significant due to large data (e.g.
search engine logs), no matter how small the effect is. Okay
if the effect size is properly reported and if the small effect
size translates to a large profit for companies.
• Some IR experiments are extremely underpowered – missing
real differences due to lack of observations (e.g. a small user
group). Maybe better experimental designs are in order.
n much larger than necessary
n much smaller than necessary

10. TALK OUTLINE
•Motivation
•Survey Method
•Results
•Summary
•Data and Feedback

11. sleeping
30%
email
15%
teaching
10%
supervising
10%
meetings
10%
eating/drinking
10%
twitter/commuting
8%
movies
5%
research
2%
AN AVERAGE WEEKDAY OF
AN AVERAGE JAPANESE PROFESSOR

12. Papers examined (Feb‐Oct 2015)

13. Coding scheme STEP 1
Does it contain a table/figure of means that
probably deserves significance testing?
If YES, record one such table/figure
(representative table).

14. Coding scheme STEP 2
Does it conduct a
significance test?
Categorise according to the
primary test used
(one test type per paper).
pdf searched with Ctrl+F
“statistical” “signi” “test” “ANOVA”

15. Coding scheme STEP 3
Does it claim “significance”?
Record the exact expressions
used.

16. Coding scheme STEP 4
Record whether p‐
values/test statistics
are reported.
If only the p‐value is
reported, deduce test
statistic wherever
possible

17. Coding scheme STEP 5
If test statistic + sample size
are available, conduct power
analysis with R tools
133 papers

18. Coding scheme: Additional classification
Tests for comparing
two systems

19. Significance test categorisation scheme
Covered by my power analysis tools
Tests for comparing two systems

20. R power analysis tools
(based on the library pwr)
• future.sample.unpairedt
• future.sample.pairedt
• future.sample.1wayanova
• future.sample.2waynorep
• future.sample.2wayanova
R code from this
book by Prof Toyoda
modified by Sakai
INPUT: sample size, test statistic,
degrees of freedom
OUTPUT: effect size, achieved power,
future sample size

21. TALK OUTLINE
•Motivation
•Survey Method
•Results
•Summary
•Data and Feedback

22. Coding scheme STEP 2
Does it conduct a
significance test?
Categorise according to the
primary test used
(one test type per paper).
pdf searched with Ctrl+F
“statistical” “signi” “test” “ANOVA”

23. 862 papers with a representative table:
distribution over significance test types
• 35‐37% use the paired t‐test.
• 28‐30% do not report significance test results although they have a representative table.
• 18‐24% use tests other than t‐tests and ANOVA.

24. 862 papers with a representative table:
significance test types over a decade
Two proportions z‐test (proportion2006 minus proportion2015): in both figures,
• the proportion of “no significance test” papers in 2015 is significantly lower than 2006
95%CI: [0.023, 0.353] (SIGIR) and [0.031, 0.329] (SIGIR+TOIS)
• the proportion of “paired t‐test” papers in 2015 is significantly higher than 2006
95%CI: [‐0.357, ‐0.044] (SIGIR) and [‐0.361, ‐0.074] (SIGIR+TOIS)

25. Coding scheme STEP 3
Does it claim “significance”?
Record the exact expressions
used.

26. 167/862=19% of papers claim “significance”
without reporting on significance test results
• “significant improvement” “significantly outperform” even in
abstracts and conclusions
• Highly misleading! Come on, we’re better than this!
IF you actually did a significance test THEN
say “statistically significant” “statistical significance”
ELSE
avoid saying “significant” “significance”

27. Coding scheme: Additional classification
Tests for comparing
two systems

28. 464 papers with a test for comparing two
systems
• 61‐66% use the paired t‐test
• 20‐23% use the Wilcoxon signed rank test
• 4‐5% use the randomisation test
• 3‐4% use the sign test

29. 464 papers with a test for comparing two
systems: across a decade
Two proportions z‐test (proportion2006 minus proportion2015): in both figures,
• the proportion of “Wilcoxon signed rank test” papers in 2015 is significantly lower than 2006
95%CI: [0.065, 0.513] (SIGIR) and [0.073, 0.468] (SIGIR+TOIS)
• the proportion of “paired t‐test” papers in 2015 is significantly higher than 2006
95%CI: [‐0.530, ‐0.048] (SIGIR) and [‐0.489, ‐0.057] (SIGIR+TOIS)

30. Coding scheme STEP 4
Record whether p‐
values/test statistics
are reported.
565 papers
If only the p‐value is
reported, deduce test
statistic wherever
possible

31. Do IR researchers report p‐values and/or test
statistics?
365/565= 65% report neither p‐values nor test statistics.
Situation not improving..

32. (“neither reported” highest in 2013)
SIGIR 2013 PC chairs:
133 for power
analysis (67 used
nonparametric tests
etc.)

33. Coding scheme STEP 5
If test statistic + sample size
are available, conduct power
analysis with R tools
133 papers

34. 133 papers that went through power analysis
overpowered
(power >=0.99)
underpowered
(power <=0.50)
about right
(0.50 < power < 0.99)
Unpaired t‐test 3 1 12
Paired t‐test 23 6 61
One‐way ANOVA 8 1 7
Two‐way ANOVA w/o
replication
2 1 4
Two‐way ANOVA 2 0 2
TOTAL 38 9 86
Yes, the thresholds are arbitrary, but you can do your own analysis using my raw Excel file.

35. Sample size ratio = actual size/future size:
paired/unpaired t‐test (1)

36. Sample size ratio = actual size/future size:
paired/unpaired t‐test (2)
A paper on personalisation from a search engine company (paired t‐test)
t=16.00, n=5,352,460, effect size=0.007, achieved power=1
recommended future sample size=164,107
Effect size very small (though this may translate into substantial profit for a
company)

37. Sample size ratio = actual size/future size:
paired/unpaired t‐test (3)
User experiments, paired t‐test
t=0.95, n=28,
effect size=0.180,
achieved power=0.152
future sample size=244
(similar results for other t‐test
results in the same paper)

38. Sample size ratio = actual size/future size:
ANOVA (1)

39. Sample size ratio = actual size/future size:
ANOVA (2)
Experiments with a commercial social media
application data (one‐way ANOVA)
F=243.42, #levels=3,
sample size per group=2551,
effect size fhat=2.252, achieved power=1,
recommended future sample size per group=52

40. Sample size ratio = actual size/future size:
ANOVA (3) User experiments, two‐way
ANOVA w/o replication
F=0.63, #levels=4,
sample size per group=17,
effect size fhat^2 = 0.039,
achieved power=0.183,
recommended future sample
size per group=75
(similar results for other
ANOVA results in the same
paper)

41. TALK OUTLINE
•Motivation
•Survey Method
•Results
•Summary
•Data and Feedback

42. Main findings (1)
• Of the 862 papers that seem to deserve significance testing,
301 (35%) use the paired t‐test; 255 (30%) lack significance
testing, of which 167 (19%) claim “significance.”
• Of the 464 papers that discuss significance for comparing
two systems, 301 (65%) use the paired t‐test; 97 (21%) use
the Wilcoxon signed rank test.

43. Main findings (2)
• Compared to a decade ago, we are seeing more t‐tests,
fewer Wilcoxon signed rank tests and fewer papers that lack
significance testing.
[Improvements statistically significant according to two‐
proportions z‐tests]
• Of the 565 papers that reported on significance test results,
65% reported neither p‐values nor test statistics.
[Not improving...]

44. Main findings (3)
38/133=29% highly overpowered
(e.g. search engine companies)
9/133=7% highly underpowered
(e.g. small user studies)

45. Recommendations
• Report effect sizes etc., not just p‐values. Especially
important for overpowered experiments because a low p‐
value may just reflect the large sample size.
• Small‐scale user studies: learn about effect sizes and sample
sizes from previous work for well‐designed experiments.
• Better reporting practices = collective knowledge and
progress.

46. TALK OUTLINE
•Motivation
•Survey Method
•Results
•Summary
•Data and Feedback

47. Raw systematic review file:
http://www.f.waseda.jp/tetsuya/data.html
• Raw Excel file containing
systematic review results
• Simple R code for power
calculation

48. Feedback for the systematic review file:
https://twitter.com/IRsysrev

49. Thank you,
See you at