Reproducibility problems afflict many sciences, including psychology. The problems are, to some extent, rooted in the criteria for and process of scientific publication. In response, many journals, funders and professional societies have begun incentivising change. For example, study preregistration, although traditionally used only by clinical trials researchers, is becoming more common. In this seminar, you will learn how it is now used even in basic experimental psychology, and how you can take advantage of preregistration and other new practices to smooth your path to publication and dissemination of your work. Bring your laptop (optional), walk with me through preregistering a study, and also learn how sites such as Open Science Framework facilitate project management and collaboration. One object of this seminar is to spark discussion of how we can all make our already wonderful system of science even better.

1. WSU_August_2016
The next MERC (MARCS Monday Afternoon Research Colloquium), will be held Monday 1 August, 2016 @ 1pm in Bankstown
Campus, Bldg 3.G.55. Lunch included before at 12pm
MARC
BLOCKBUSTER MARC
Assoc Prof Alex Holcombe
Title: Reproducibility, preregistration, et cetera: Making good science even better
Abstract:
Reproducibility problems afflict many sciences, including psychology.The problems are, to some extent, rooted in the criteria for
and process of scientific publication. In response, many journals, funders and professional societies have begun incentivising change.
For example, study preregistration, although traditionally used only by clinical trials researchers, is becoming more common. In this
seminar, you will learn how it is now used even in basic experimental psychology, and how you can take advantage of
preregistration and other new practices to smooth your path to publication and dissemination of your work. Bring your laptop
(optional), walk with me through preregistering a study, and also learn how sites such as Open Science Framework facilitate project
management and collaboration. One object of this seminar is to spark discussion of how we can all make our already wonderful
system of science even better.
Bio:
Associate Professor Alex Holcombe has been active in several initiatives related to publishing innovations and open science,
beginning in 2006 when he joined the founding advisory board of the journal PLoS ONE. He has been involved with
PsychFiledrawer.org (co-founder), CurateScience.org, and the openness article badges (https://osf.io/tvyxz/) adopted by
Psychological Science and others to reward open practices.Two years ago, he co-founded a new article type, the Registered
Replication Report, at the journal Perspectives on Psychological Science.

2. WSU_August_2016
Reproducibility,
preregistration, etc.:
Making good science even better
Alex.Holcombe@sydney.edu.au
@ceptional
http://www.slideshare.net/holcombea/

3. 3
Slide courtesy Chris Chambers

4. 4
Thanks Deborah Aphthorp

5. 5
Slide courtesy Chris Chambers

6. 6
P-hacking
Simmons et al. (2011), Psychological Science
• Collecting more data after
seeing whether results were
significant
• Option to report either of
two DVs
• Control via ANCOVA for
gender or not
• Drop or not one of 3
conditions
“False-positive psychology:
Undisclosed flexibility in data
collection and analysis allows
presenting anything as significant.”
Simmons et al. (2011), Psychological Science
60% false positive rate
p > .05

7. Schönbrodt, F. D. (2015). p-hacker:Train your p-hacking skills!
http://shinyapps.org/apps/p-hacker/.

8. Slide from Tom Hardwicke

9. 9
Slide courtesy Chris Chambers

10. Behavior
Psychologists
reporting
Failed to report all dependent measures 67%
Selectively reported studies that “worked” 49%
Excluded data after looking at impact of doing so 42%
Claimed to have predicted an unexpected finding 32%
Failed to report all conditions 26%
John, Loewenstein, & Prelec (2012). Psychological Science, DOI: 10.1177/0956797611430953
Motivated reasoning

11. WSU_August_2016
"This one's Psych Science. 1998. Put
your money on ‘will replicate’?”
"What, you think I was born
yesterday?”
"Fine," muttered the bookie.
#AcademicNoir

12. WSU_August_2016
12

13. WSU_August_2016
13

14. WSU_August_2016
"Whatever happened to him, anyway? He always had great results."
“Office of Research Integrity got him,” I replied.
We all laughed.
CC-BY Alex Holcombe
https://twitter.com/ceptional/status/663850868396044288
#AcademicNoir

15. WSU_August_2016

16. Behavior
Psychologists
reporting
Failed to report all dependent measures 67%
Selectively reported studies that “worked” 49%
Excluded data after looking at impact of doing so 42%
Claimed to have predicted an unexpected finding 32%
Failed to report all conditions 26%
John, Loewenstein, & Prelec (2012). Psychological Science, DOI: 10.1177/0956797611430953
Preregistration

17. It’ll soon shake your windows
And rattle your walls
For the times, they are a’changin’
23 July 2016
Stopping rule and power analyses expected.
The International Committee of Medical Journal Editors
(ICMJE) announces that their journals will not publish
reports of trials unless they have been registered.
2004

18. Preregistration https://osf.io/c8xhy/
• General methods
• Participant recruitment, selection
criteria, and exclusion criteria
• Power analysis
• Data analysis plan
• Statistical tests
• Outlier exclusion criteria

19. 19
MAKE P-VALUES
INTERPRETABLE
AGAIN!
PRE-REGISTER
YOUR
ANALYSIS PLAN

20. 20
Pre-registration made easy
AsPredicted.org

21. WSU_August_2016
21
Concerns
1. Pre-registration does not intend to prevent researchers from conducting
exploratory analysis or prevent serendipitous findings
2. It is solely to ensure that the distinction between exploratory and
confirmatory findings is clear and transparent

22. For the times, they are a’changin’
Edwards, Lindman, & Savage (1963)
my other concern is more difficult to
address. I has to do with your very small
samples…
Publication of low power studies is simply
not good for the field in the long run -
they inflate both the rate of false
negatives and false positives in the
literature (e.g., Ellis, 2010).

23. WSU_August_2016
23
https://osf.io/tvyxz/wiki/home/
100% P-hacking Free
Here, check our
numbers.
Here’s how you
can replicate our
result.
CC-ZERO Alex Holcombe
For the times, they are a’changin’

24. WSU_August_2016
24
For the times, they are a’changin’

25. WSU_August_2016
25
Concerns

26. WSU_August_2016
26
New article formats,

27. WSU_August_2016
27
courtesy Chris Chambers (Cardiff)

28. WSU_August_2016
Any one of you go it alone, he’ll say you messed up. But if
we first get him to approve the protocol, and then all run
the replication together…

29. WSU_August_2016
Noeffect
Hagger, M., & Chatzisarantis, N. (2016).A Multilab Preregistered Replication of the Ego-Depletion Effect. Perspectives on
Psychological Science, 11 (4), 546-573
http://blogs.discovermagazine.com/neuroskeptic/2016/07/31/end-of-ego-depletion/#.V55g9pN96IZ