This document discusses several issues with reproducibility and accuracy in scientific research. It summarizes several studies that found: - Only 20-25% of studies in oncology, gynecology, and cardiology could be replicated - Only 11% of "landmark" studies in hematology and oncology could be reproduced - Some individual researchers were found to have published a large number of papers that could not be replicated It also discusses how flexibility in data collection, analysis, and reporting can inadvertently lead researchers to only report results that achieve statistical significance, even if other results were explored. Finally, it notes that "questionable research practices" like these may pose a larger threat to science than outright fraud.

1. gert storms
17 december 2017

2. philip abelson
editor of science
1962-1984
99,9999 % of
[scientific] papers
are accurate and
truthful

4. 15%
10%

5. 67 studies in oncology (47), gynaecology (12),
and cardiovascular studies (8)
only 20-25% replications

6. 53 ‘landmark’ studies in
haematology & oncology
11 % reproduced

7. Yoshitaka Fujii (anesthesia, Japan): 172
Dipak K. Das (heart surgery, USA): 145
John Darsee (medicine, USA): 104
F. Hermann & M. Bruch (medicine, GER): 94
Diederik Stapel (psychology, Ndl): 55

8. Honorable mention:
Jan Hendrik Schön (Germany, Bell labs)
9 papers in Science
7 papers in Nature
28 papers in total
… all published in 2000 & 2001

9. the million $
question !!!

11. Allow a small chance for false positive results:
p < .05 but …
extreme flexibility in
data collection
data peeking
data-analysis
what is an outlier ?
when do you use covariates ?
pre-transform data ?
reportering
report all variables ?
report all tried-out analyses ?
all conditions? all experiments?

12. often many possibilities tested, but only
reported those that ‘worked’ (p < .05)
sometimes no bad intensions, but
a) there are no clear rules
b) we all want significance !
p < .05

13. ”‘usual’ undisarable practices probably larger threat for science
than ‘real’ fraud”
- f0irst anonymous measurement of
‘questionable research practices’ (QRP)
- 2002 survey: NIH sponsored researchers
1768 mid-career (52% response rate)
1479 early-career (43% response rate)
non-response bias ?

15. - first measurement in psychology of
‘questionable research practices’ (QRP)
- electronic questionnaire
- 2155 respondents (response rate 36%)
non-response bias ?

19. American Economic Journal: Applied Economics,
Volume 8, Number 1, January 2016, pp. 1-32(32)

21. Button et al (2013), Nature Neuroscience
median power in neuro sciences = 21%
neuroimaging studies: median 8 %
water maze studies: median 18-31 %

22. Galilei Newton Mendel Bernoulli
Dalton Millikan HewishAlexis Carrel Elckerlyc ??

24. mean = 4
mean = 6

26. theoretical courses don’t have an effect,
but whoever sleeps with the dog gets his flees

27. 30.000
participants
18.000
‘so-so’
150$
12
‘bad guys’
36.000$

29. “Our conclusion, in brief, is that science bears
little resemblance to its conventional portrait.
(…) In the acquisition of new knowledge,
scientists are not guided by logic and objectivity
alone, but also by such nonrational factors as
rhetoric, propaganda, and personal prejudice.
Scientists do not depend solely on rational
thought, and have no monopoly on it. Science
should not be considered the guardian of
rationality in society, but merely one major form
of its cultural expression.”
Broad & Wade, 1982, p. 8-9.