The document summarizes registered reports, an alternative publication format that aims to address reproducibility issues. It discusses: 1) The standard publication process and reproducibility crisis in science due to biases like publication bias, low statistical power, p-hacking, and HARKing. 2) What registered reports are - a two-stage peer review process where the proposed methods and analyses are peer-reviewed before data collection. This removes biases driven by study outcomes. 3) Why registered reports are gaining popularity - they can increase reproducibility, computational reproducibility, and study quality while reducing biases compared to standard publications. 4) An example of an author's experience submitting a registered report to be peer-reviewed in stage

1. Journal Articles Are Not
the Only Fruit:
Registered Reports
John Tyson-Carr
HLJTYSON@Liverpool.ac.uk

2. Summary
1. The Average Publication Process
2. The Reproducibility Crisis
3. What is a Registered Report?
4. Why Registered Reports?
5. An Example
6. Misconceptions, Tips and Career
Implications 2

3. 1. The Average Publication Process
The current status quo for scientific research
3

4. The Average Publication Process
4
1 3 5
6
4
2
Data Collection Submit Manuscript Produce Final Manuscript
Prepare Manuscript Editor Decision &
Standard Peer Review
Process
Publication

5. “ Are there flaws in the
standard publication
process?
5

6. 2. The Reproducibility Crisis
Is most research false?
6

7. The Reproducibility Crisis
▪ Why Most Published Research Findings are False
(Ioannidis, 2005)
▪ Is Most Published Research Wrong (Veritasium,
https://www.youtube.com/watch?v=42QuXLucH3Q)
▪ Reproducibility Project - Open Science Collaboration (2015)
▫ One-hundred psychology studies were replicated
▫ Original studies had 97% with p < .05
▫ Replicated studies had 36% with p < .05
7

8. Significance of results
between original and
replicated study is often
vastly different.
Figure from Open Science
Collaboration (2015)
8
The Reproducibility Crisis

9. 9
Is there a
reproducibility
crisis?
Figure from
Baker (2016)

10. 10
The Four
Horsemen of the
Irreproducibility
Crisis (Bishop,
2019)

11. Horse 1 - Publication Bias
▪ Non-significant results
often seen as “failed
experiments”
▪ Less likely to be written
up, less likely to be
accepted by journals
11

12. Horse 2 – Low Statistical
Power
▪ Reduced chance of
detecting true effects
▪ Reduced chance that
observed effect is a true
effect
▪ Median power in
neuroscience estimated
to be between 8% and
31% (Button et al., 2013) 12
True
Effect
1.2
Study 1
1
Study 2
1.2
Study 3
1.6
20% Power

13. Horse 3 – P-Hacking
▪ With enough flexibility in
analysis procedures, you
can make anything
statistically significant
▪ The 2012 IG Nobel Prize –
Observing “meaningful”
brain activity in a dead
salmon
13

14. Horse 4 – HARKing
▪ Hypothesising After
Results Known
▪ We construct theories to
match the data
▪ Likely to introduce type 1
errors into theories
▪ Figure from Munafo et al.
(2017)
14

15. 3. What is a Registered Report
An alternative to standard journal articles
15

16. What is a Registered Report?
▪ Peer review follows not only writing up of results, but also
preceding the data collection
▪ The research process, including collection and analysis, is
declared in its entirety before collection begins
16

17. What is a Registered Report?
The Two-Stage Process
▪ The usual editorial triage
▪ In stage 1, the proposed
research is assessed
▪ Following in-principle
acceptance (IPA), the
extent to which you did
what you said you were
going to do is assessed
17

18. Stage 1
Title, Abstract, Introduction, Methods (including
proposed analyses) & Pilot Data (optional)
Reviewers assess:
▪ Scientific validity of question
▪ Logic, rationale, and plausibility of hypotheses
▪ Soundness and feasibility of methodology and
analysis pipeline (including statistical power analysis
where appropriate)
▪ Whether detail allows replication of procedures and
analysis
▪ Whether outcome-neutral tests have been specified
sufficiently to test hypotheses, including positive
controls and quality checks
What is a Registered Report?
The Two-Stage Process
Stage 2
Largely resembles a standard manuscript
Reviewers assess:
▪ Whether the data are able to test hypotheses by
satisfying approved outcome-neutral conditions
▪ Whether introduction, rationale and hypotheses are
same as approved from stage 1
▪ Whether authors adhered to registered reports
guidelines
▪ Whether any unregistered, post hoc analyses are
justified, methodologically sound and informative
▪ Whether conclusions are justified given the data
18

19. “ Please note that
editorial decisions will
not be based on the
perceived
importance, novelty,
or clarity of the
results.
19

20. What is a Registered Report?
Other Considerations
▪ Everything needs to be in place prior to stage 1 (ethics,
resources, etc.)
▪ Anticipated timelines following IPA will be expected
▪ All data and codes must be made publicly available (for e.g., on
Open Science Framework), at least in most journals that offer RR
format.
▪ If you withdraw after IPA, summary of the pre-registered study
will still be made publicly available
▪ Can include section containing already collected and analysed
pilot data
▪ Secondary analysis of data is also allowed
20

21. What are
you signing
up for?
A commitment to carry out theory driven
research in a statistically rigorous way 21

22. 4. Why Registered Reports?
Are they the answer?
22

23. Why Registered Reports?
Fighting the Four Horsemen
Publication Bias
Published regardless of
results
Low Statistical Power
Power analysis
declared in advance
P-Hacking
Declaring all analyses in
advance, and the
parameters for those
analyses, stops us from
torturing the data
HARKing
Hypotheses declared in
advance
23

24. Why Registered Reports?
Fighting Bias
A study by Scheel at al. (2021)
revealed:
▪ 4% of standard reports failed
to confirm hypothesis
▪ 56% of registered reports
failed to confirm hypothesis
Figure from Scheel et al. (2021)
24

25. Why Registered Reports?
Fighting Computational Irreproducibility
A study by Hardwicke at al.
(2018) revealed that journals
that encourage public code
result in greater
computational reproducibility
Figure from Hardwicke at al.
(2018)
25

26. Why Registered Reports?
Fighting Poor Study Quality
A study by Soderberg at al. (2018)
required that 353 scientists rate
RRs and non-RRs on 19
characteristics such as novelty,
importance and rigour
RRs numerically outperformed non-
RRs on every criterion, whilst being
statistically indistinguishable in
terms of novelty and creativity
Figure from Soderberg et al. (2018)
26

27. 5. An Example
Cortex Stage 1 Experience
27

28. An Example
▪ Currently at the end of the
first round of reviewer
comments for stage 1
▪ Secondary analysis of data
28

29. An Example
▪ Our previous work revealed,
using modern techniques, a
novel brain response to
symmetry (Tyson-Carr et al.,
2021)
▪ With an idea of what this brain
response represents, we
formulated a set of hypotheses
to test
29

30. An Example
▪ During this project, a catalogue
of data was being compiled
comprising data from ~2215
participants (Makin et al., under
review; https://osf.io/2sncj/)
▪ Rather than design an
experiment to test our
hypotheses, we selected a
subset of eligible data from this
repository to reanalyse
30

31. An Example
▪ With the hypotheses formed, we
produced an analysis pipeline that
tested the hypotheses, including for
each hypothesis:
▫ Inclusion criteria
▫ Proposed analyses and
parameters
▫ Power analyses (with alpha of
.02 and 90%, as per journal
guidelines)
▪ Since we cannot analyse the data
until IPA, we used a set of already
analysed data to form “pilot data”
31

32. 32
Decision Trees
Exhaustive Criteria
to Remove
Researcher
Degrees of
Freedom
PCA to identify
number of brain
sources
If ≥ 2, fit two
sources in visual
cortex
PCA to identify
remaining brain
sources
If ≥ 1, fit further
sources
sequentially
If == 0, finalise
model
If residual variance <
40%, export model
and include data
If residual variance
≥ 40%, exclude
data
If < 2, exclude
data

33. An Example
Reviewer Comments
▪ General suggestions for improvements to
statistical analyses
▪ Improved ways of illustrating data
▪ More detail about the analysis procedures
▪ Suggestions to remove researcher degrees of
freedom following IPA
▪ These comments will differ to stage 1 registered
reports which are not secondary analyses 33

34. 6. Misconceptions, Tips and
Career Implications
Some practical considerations
34

35. Misconceptions, Tips and Career Implications
The Misconceptions
Some misconceptions outlined by Chambers & Tzavella (2022)
Misconception
RRs hamper exploratory research
Reality
Exploratory analysis section in stage 2 report is permitted
35

36. Misconceptions, Tips and Career Implications
The Misconceptions
Some misconceptions outlined by Chambers & Tzavella (2022)
Misconception
RRs can be easily gamed by “post-registering”
Reality
Must certify that no data collection has taken place, and
editors/reviewers usually ask for at least some minor changes
to procedure 36

37. Misconceptions, Tips and Career Implications
The Misconceptions
Some misconceptions outlined by Chambers & Tzavella (2022)
Misconception
Fixed plans that you define cannot be changed
Reality
Editors and reviewers can be consulted if changes need to be
made to registered protocol
37

38. Misconceptions, Tips and Career Implications
The Misconceptions
Some misconceptions outlined by Chambers & Tzavella (2022)
Misconception
RRs slow the pace of research
Reality
Rejection rate is much lower for RRs (10% at Cortex for stage 1,
0% at stage 2), and rejections in standard articles are common
due to unattractive results or an unfixable flaw (something
which would likely be fixed if protocol was peer-reviewed)
38

39. Misconceptions, Tips and Career Implications
Some Tips
Some tips outlined by Chambers & Tzavella (2022)
Tip
Assess feasibility and validity of analyses beforehand to
minimise deviations following IPA
39

40. Misconceptions, Tips and Career Implications
Some Tips
Some tips outlined by Chambers & Tzavella (2022)
Tip
Pilot studies are highly recommended
40

41. Misconceptions, Tips and Career Implications
Some Tips
Some tips outlined by Chambers & Tzavella (2022)
Tip
If you are struggling to define an exhaustive protocol for all
contingencies, consider using blinded analysis (see Dutilh et
al., 2017)
41

42. Misconceptions, Tips and Career Implications
Some Tips
Some personal tips.
Tip
Do not underestimate the specificity required
42

43. Misconceptions, Tips and Career Implications
Some Tips
Some personal tips.
Tip
For early career researchers, use it as an opportunity to
remove that level of abstraction behind the techniques we
usually use
43

44. Misconceptions, Tips and Career Implications
Career Implications
▪ Short-term pain, long-term gain
▫ GitHub
▫ Repositories
▫ Data availability principles
▫ Code availability
▪ Our careers ultimately benefit from improved science
▪ Kicking away the ladder?
44

45. Start
Somewhere
Open science practices do not have to be
all or nothing – a step in the right direction
is a step nonetheless 45

46. 46
THANKS!
Any questions?
You can find me at:
▪ LinkedIn – John Tyson-Carr
▪ HLJTYSON@liverpool.ac.uk