Reprodutibilidade em resultados de pesquisa (Olavo Bohrer Amaral)
Olavo B. Amaral
Reunião de Editores Científicos
and scientific publication
• We are used to hearing that scientific articles are the most
reliable source of evidence to which we have access.
• That said, we have surprisingly little empirical evidence for
An established paradigm
A basic flaw
• When reading a scientific article, we do not know the
original hypotheses, the number of comparisons and the
number of analysis options.
Like a Tinder profile
• This is not all that different from other ‘non-scientific’
syntheses of reality.
• Traditional peer review conflates the evaluation of methods
and results (and of data and stories).
• Beautiful stories with lots of shaky data tend to trump solid,
- Number of
- Coherence of results
- Novelty of theory
- Potential impact
- Experimental design
- Protocol registration
- Data availability
- Statistical rigor
Doomed to fail
• Irreproducibility is the natural outcome in a system in which
impact and novelty count but truth does not.
Wrong time, wrong place
• Even when peer review does work, it comes at the wrong
time (after results are in) and behind closed doors.
The death of prepublication peer review?
• It is time to discuss whether prepublication peer review
should not give way to different forms of quality control.
Time to experiment
• Do we lose that much by giving up on prepublication peer
Quality of reporting of methods/results in
preprints vs. peer-reviewed articles
bioRxiv 59.4% vs. 63.9% PubMed
p = 0.019, r2 = 0.036
Carneiro et al., in prep.
Actual peer review
• We urgently need better data on reproducibility in various
• 71 lab-strong initiative to reproduce 50-100 experiments from
Brazilian life sciences articles over the last 20 years.
• Each experiment reproduced in 3 labs to estimate interlab
No need to fear change
• Unlike with democracy, we have not really tried out anything
different for the last century.
Barriers are cultural, not scientific
• Peer review has become more important for science as a form
of branding than of actual quality control.
• The current model of article peer review as a barrier to
publication based is more harmful than helpful for science.
• Changing the way we publish science to make it more
reproducible and accessible is an ethical imperative.
• These changes must be accompanied by changes in how we
• The barriers to change are cultural rather than financial or
The tide is changing
• It is time to decide whether to cling to a sinking ship or to
help building new ones.