This document discusses the reproducibility crisis in science and potential solutions. It provides references from numerous papers and talks on topics like false positive findings, small sample sizes undermining reliability, questionable research practices, replication in genome-wide association studies, and unintended consequences of open research practices. The references cover pre-registration, cell line contamination issues, impact of journal impact factors, and historical examples like the discredited MMR/autism study.

1. Newsflash! Just out – very good and comprehensive paper on this topic:
Forstmeier, W., Wagenmakers, E.-J., & Parker, T. H. (2016). Detecting and
avoiding likely false-positive findings – a practical guide. Biological Reviews, n/a-
n/a. doi: 10.1111/brv.12315
What is the reproducibility crisis in science and what can we do about it?
D.V. M. Bishop
Talk at Cognition and Brain Sciences Unit, 22nd November 2016
References
Academy of Medical Sciences. 2015. Symposium report: Reproducibility and reliability of
biomedical research: improving research practice. http://www.acmedsci.ac.uk/policy/policy-
projects/reproducibility-and-reliability-of-biomedical-research/
Bem, DJ. 2014. Writing the Empirical Journal Article. In John M. Darley, Mark P. Zanna, and
Henry L. Roediger III (Eds) The Compleat Academic: A Practical Guide for the Beginning Social
Scientist, 2nd Edition. Washington, DC: American Psychological Association, 2004.
Bishop DVM. 2016. Open research practices: unintended consequences and suggestions for
averting them. (Commentary on the Peer Reviewers' Openness Initiative). Open Science 3.
http://rsos.royalsocietypublishing.org/content/3/4/160109
Bustin SA. 2014. The reproducibility of biomedical research: Sleepers awake! Biomolecular
Detection and Quantification 2:35-42.
Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, and Munafo MR. 2013.
Power failure: why small sample size undermines the reliability of neuroscience. Nature
Reviews Neuroscience 14:365-376.
Chabris C, and Simons D. 2010. The invisible gorilla and other ways our intuition deceives us.
London: HarperCollins.
Chambers, C. (2017, forthcoming) The seven deadly sins of psychology. Princeton University
Press.
Collins, F. S., & Tabak, L. A. (2014). NIH plans to enhance reproducibility. Nature, 505(7485),
612-613.
Cramer AOJ, van Ravenzwaaij D, Matzke D, Steingroever H, Wetzels R, Grasman RPPP, Waldorp
LJ, and Wagenmakers E-J. 2016. Hidden multiplicity in exploratory multiway ANOVA:
Prevalence and remedies. Psychonomic Bulletin & Review 23:640-647.
De Groot AD. 2014. The meaning of “significance” for different types of research [translated
from original 1956 paper and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine
Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh,
and Han L. J. van der Maas]. Acta Psychologica 148:188-194.

2. Dickersin K, and Min YI. 1993. NIH clinical trials and publication bias. The Online journal of
current clinical trials Doc No 50.http://europepmc.org/abstract/med/8306005
Dubois J, and Adolphs R. 2016 Building a Science of Individual Differences from fMRI. Trends in
Cognitive Sciences. http://dx.doi.org/10.1016/j.tics.2016.03.014
Fanelli, D. (2010). "Positive" results increase down the hierarchy of the sciences. PLOS One,
5(3). doi: 10.1371/journal.pone.0010068
Gelman A, and Loken E. 2013. The garden of forking paths: Why multiple comparisons can be a
problem, even when there is no 'fishing expedition' or 'p-hacking' and the research
hypothesis was posited ahead of time.
www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
Greenwald AG. 1975. Consequences of prejudice against the null. Psychological Bulletin 82:1-
20.
Ioannidis JPA. 2005. Why most published research findings are false. PLoS Medicine 2:e124.
John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research
practices with incentives for truth telling. Psychol Sci 23: 524–532. doi:
10.1177/0956797611430953
Kerr, N. (1998). HARKing: Hypothesizing after the results are known. Personality and Social
Psychology Review, 2, 196-217.
Kraft, P., Zeggini, E., & Ioannidis, J. P. A. (2009). Replication in Genome-Wide Association
Studies. Statistical Science, 24(4), 561-573. doi: 10.1214/09-sts290
Lazic,S. E. 2016. Experimental Design for Laboratory Biologists : Maximising Information and
Improving Reproducibility
Levenson T. 2015. The Hunt for Vulcan: . . . And How Albert Einstein Destroyed a Planet,
Discovered Relativity, and Deciphered the Universe. New York: Random House. (for summary
see: http://news.nationalgeographic.com/2015/11/151104-newton-einstein-gravity-vulcan-
planets-mercury-astronomy-theory-of-relativity-ngbooktalk/)
Macleod MR, McLean AL, Kyriakopoulou A, Serghiou S, de Wilde A, Sherratt N, Hirst T,
Hemblade R, Bahor Z, Nunes-Fonseca C et al. 2015. Risk of Bias in Reports of In Vivo
Research: A Focus for Improvement. PLOS Biology 13.
MacLeod RAF, Dirks WG, Matsuo Y, Kaufmann M, Milch H, and Drexler HG. 1999. Widespread
intraspecies cross-contamination of human tumor cell lines arising at source. International
Journal of Cancer 83:555-563.
MacCoun R., Perlmutter S. 2015 Hide results to seek the truth. Nature 526, 187-189
Masters JR. 2012. Cell-line authentication: End the scandal of false cell lines. Nature 492:186-
186.
McNutt, M. (2014). The measure of research merit. Science, 346(6214), 1155-1155. doi:
10.1126/science.aaa3796

3. Munafo, M., Noble, S., Browne, W. J., Brunner, D., Button, K., Ferreira, J., . . . Blumenstein, R.
(2014). Scientific rigor and the art of motorcycle maintenance. Nat Biotech, 32(9), 871-873.
doi: 10.1038/nbt.3004
Nardone RM, MacLeod RAF, and Capes-Davis A. 2016. Cancer: Authenticate new xenograft
models. Nature 532:313-313.
Newcombe, R. G. (1987). Towards a reduction in publication bias. British Medical Journal, 295,
656-659
Rosenthal R. 1979. The file drawer problem and tolerance for null results. Psychological Bulletin
86:638-641.
Ryan J. 2008. Understanding and managing cell culture contamination: Technical bulletin.
www.level.com.tw/html/ezcatfiles/vipweb20/img/img/20297/contamination-COR.pdf
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating
research. British Medical Journal, 314(7079), 497. doi: 10.1136/bmj.314.7079.497
Zilles K, and Amunts K. 2010. Centenary of Brodmann's map - conception and fate. Nature
Reviews Neuroscience 11:139-145.
Other sources
Blog on cell line contamination:
http://retractionwatch.com/2015/12/08/hela-is-the-tip-of-the-contamination-iceberg-guest-
post-from-cell-culture-scientist/
My blog: http://deevybee.blogspot.co.uk/2012/11/bishopblog-catalogue-updated-24th-
nov.html
See sections on Statistics and Academic Life
Pre-registration: Rationale and how-to-do it, with FAQs: https://osf.io/8mpji/wiki/home/
MMR/autism debacle: http://www.snopes.com/medical/disease/cdcwhistleblower.asp
Account of Archie Cochrane: from Ben Goldacre’s blog:
http://www.badscience.net/2010/04/righteous-mischief-from-archie-cochrane/
Chris Chambers on Guardian blog: https://www.theguardian.com/science/head-
quarters/2014/may/20/psychology-registration-revolution
Recent piece on Merck’s threat: https://www.technologyreview.com/s/601348/merck-wants-
its-money-back-if-university-research-is-wrong/
David Colquhoun on journal impact factor: http://www.dcscience.net/colquhoun-nature-
impact-2003.pdf

4. John P. A. Ioannidis https://www.youtube.com/watch?v=xGLF6olIZYY
1st BHA Annual Special Lecture
Lakens, D. Improving your statistical inferences: https://www.coursera.org/learn/statistical-
inferences