Presented at the Open Scholarship Prize Competition 2021, organised by Open Scholarship Community Galway. Video of the presentation: https://nuigalway.mediaspace.kaltura.com/media/OSW2021A+OSCG+Open+Scholarship+Prize+-+The+Final!/1_d7ekd3d3/121659351#t=56:08

1. Towards reproducible research
and maximally-open data
Pablo Bernabeu
OSCG Open Scholarship Prize Competition, 14th May 2021

2. Psycholinguistics
> Conceptual processing
Reanalysis of data from
Hutchison et al. (2013)
How does the brain process the
meaning of words?
Statistical regularities in language
as well as perceptual, motor and
emotional information.
How does this process vary in
different contexts and across
different people?
2

3. Open information at each stage of research
Design: preregistration of theoretical background and methodological
protocol.
Development: procedural issues (corrections, errors, other changes)
that bear on the final materials.
Completion: data collection software, raw data, processed data, final
data set, analysis code and results.
S
h
a
r
e
d
i
n
m
y
P
h
D
S
h
a
r
e
d
i
n
m
y
M
A
3
Application of
open science
My research
Community: open-source code workshops and software

4. Open data
Bernabeu et al. (2017), Bernabeu (2018)
• All materials from the completion
stage: experiment administration
software, raw data, processed
data, final data sets, analysis code
and results.
• Development stage proceedings
reported.
• Readme files describing the data
sets and linking to resources such
as data dashboards.
4

5. Maximally-open data
Bernabeu et al. (2017), Bernabeu (2018)
• R-based web applications open to scientists and the general public.
• Easy visualization of the variance and inspection of procedural aspects such as
trimming, adjustments, changes. Quicker usage of the data (see blog post).
5

6. Preregistration, power analysis and open data
Chen et al. (2018)
• Prereg.: https://psyarxiv.com/t2pjv/
• Video demonstration of the lab
procedure: https://osf.io/h36wr/
• Data: https://osf.io/waf48/
6

7. Preregistration
and open data
Bernabeu et al. (2021)
• Specification of the theoretical
background and the methodological
protocol for a forthcoming study.
• Integration of FAIR data from several,
large-sample studies
• Large, secondary data = valuable
alternative to small, noisy samples
(see Loken & Gelman, 2017).
7

8. Power analysis: How many participants required?
For next study, the preregistration will include a power analysis based on two large-
sample pilots (combined, FAIR data sets), using power curves based on Monte Carlo
simulations (simr R package).
Preliminary curves below (pending more simulations for a greater accuracy).
Y axis = power for a certain effect; X axis = 1 to 312 participants.
8

9. R workshops
Workshops and presentations on data
visualisation and analysis in R since 2018,
mostly in the context of a fellowship from
the Software Sustainability Institute.
• http://pablobernabeu.github.io/#workshops
• https://github.com/pablobernabeu/Data-is-present
Blogging
Several blog posts on psycholinguistic research,
open science and statistics.
http://pablobernabeu.github.io/blog
9

10. More open-source web applications
for research and teaching
Experimental data simulation WebVTT caption transcription
https://github.com/pablobernabeu/Experimental-data-simulation https://github.com/pablobernabeu/VTT-Transcription-App
10

11. Concluding thoughts
• Attainable for early-career researcher: individual and community applications
Some win-win benefits
• Open science is a framework, rather than an all-or-nothing result.
Design stage: my preregistrations could be even more precise (see Bakker et al., 2020).
Development stage: my procedures could be even more open.
Completion stage: my materials could be even more easily accessible.
• Let’s not eschew studies that report adjustments or errors.
If errare humanum est, by definition (see Bakker et al., 2020), how many spotless studies should
there naturally be in journals?
• Reward structures (e.g., promotion) still often prioritise number of publications.
11

12. References
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor,
D., & Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12),
e3000937. https://doi.org/10.1371/journal.pbio.3000937
Bernabeu, P. (2018). Dutch modality exclusivity norms for 336 properties and 411 concepts. PsyArXiv.
https://psyarxiv.com/s2c5h
Bernabeu, P., Lynott, D., & Connell, L. (2021). Preregistration: The interplay between linguistic and embodied
systems in conceptual processing. OSF. https://osf.io/ftydw/
Bernabeu, P., Willems, R. M., & Louwerse, M. M. (2017). Modality switch effects emerge early and increase
throughout conceptual processing: evidence from ERPs. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. J.
Davelaar (Eds.), Proceedings of the 39th Annual Conference of the Cognitive Science Society (pp. 1629-1634).
Austin, TX: Cognitive Science Society. https://mindmodeling.org/cogsci2017/papers/0318
Chen, S., Szabelska, A., Chartier, C. R., Kekecs, Z., Lynott, D., Bernabeu, P., … Schmidt, K. (2018). Investigating object
orientation effects across 14 languages. PsyArXiv. https://doi.org/10.31234/osf.io/t2pjv/
Hutchison, K. A., Balota, D. A., Neely, J. H., Cortese, M. J., Cohen-Shikora, E. R., Tse, C.-S., Yap, M. J., Bengson, J. J.,
Niemeyer, D., & Buchanan, E. (2013). The semantic priming project. Behavior Research Methods, 45, 1099–1114.
https://doi.org/10.3758/s13428-012-0304-z
Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584-585.
https://doi.org/10.1126/science.aal3618
12

13. Thank you to OSCG, the sponsors and the audience!
Also, thank you to my mentors and everyone else who has
contributed to my research.
13