The document discusses problems with the current scientific publishing system, including inefficiency, opacity, sluggishness, and redundancy. It also notes issues with incompleteness, as there is little incentive to publish replications or non-replications. This leads to the "file drawer problem" where unpublished results are not made available. Suggested solutions include journals that publish replications, pre-registering study designs, and posting peer reviews and commentaries after publication to improve transparency and completeness.
1. The broadest problem in science:
Our publishing system
Alex.Holcombe@sydney.edu.au
School of Psychology
http://www.slideshare.net/holcombea/
@ceptional
1
3. Incomplete
The File-Drawer Problem
•Little career incentive to publish
a non-replication or a replication
•Very difficult to publish a non-
unpublished replication or replication
results •Most journals only publish
files papers that “make a novel
contribution”
•Reviewers/editors tend to hold
non-replicating manuscript to
http://www.flickr.com/photos/nickperez/2569423078 t. magnum higher standard than original.
•Bem example
3
4. Incomplete
The File-Drawer Problem
Tower of
unpublished Corollary 4: The greater the
results flexibility in designs, definitions,
outcomes, and analytical modes in
files a scientific field, the less likely the
research findings are to be true.
Flexibility increases the potential for
transforming what would be “negative”
results into “positive” t most
results.
ha
oan nidis t
ree with I
hile we ag .”
http://www.flickr.com/photos/nickperez/2569423078 t. lse..
m ary, w are famagnum Corollary 6: The hotter a scientific field (with more
“I n sum findings scientific teams involved), the less likely the research
rch
resea findings are to be true.
4
5. Barriers to publishing replications and failed-
replications
• No glory in publishing a replication
• Few journals publish replications
• usually uphill battle even with
those that do
• The wrath of the original researcher
5
6. File-drawer fixes
• Journals that don’t reject
replications for being
uninteresting or unimportant
◦ • ✔
• Pre-registration of study designs
and analysis methods
◦ • ◦
• Brief reporting of replications
◦ ✔ ◦
6
10. File-drawer fixes
• Journals that don’t reject
replications for being
uninteresting or unimportant
◦ • ✔
• Pre-registration of study designs
and analysis methods
◦ • ◦
• Brief reporting of replications
◦ ✔ ◦
10
11. More efficient, more complete
ROIM- peer Review of Intro & Methods
1. Authors plan a replication study
2. They submit an introduction and methods section
3. It is sent to reviewers, including the targeted author
4. The editor decides whether to accept/reject, based on:
1. Reviewer comments regarding the proposed protocol
2. Importance of the study, judged by argument in the
introduction, number of citations of original, reviewer
comments
• • ✔
5. The Intro, Method and analysis plan, and reviewer comments
are posted on the journal website
6. When the results come in, the authors write a conventional
results and discussion section and that together with the raw
data are posted, yielding the complete publication
1. some sort of minimal peer review needed for that. What
exactly?
11
12. ROIM- peer Review of Intro & Methods
• Original author sort-of signed off on it, so can’t
complain / hate the replication authors as much.
• Good way to start for a new PhD project, anyone
planning to build on some already-published results
• Reduce the incentive to publish flashy, headline-
grabbing but unreliable studies?
• • ✔
12
13. Incomplete
post-publication
peer review
When a new paper
appears, readers often spot logical flaws,
experimental weaknesses, questionable
assumptions or alternative interpretations.
Yet individual criticisms may not be
considered important enough to warrant
publication. Even major criticisms are
unlikely to appear until months or years
later, and are often overlooked in the
haystacks of the literature.
13
14. post-publication
peer review
The people who
DON’T notice the
problems with a paper
cite it, the people who
DO, don’t. So people
not in the area never
find out how flawed a
paper is.
The article in question was published in the
Proceedings of the National Academy of Sciences,
and at the time of writing has had 270 citations.
I did a spot check of fifty of those citing articles to
see if any had noted problems with the paper: only
one of them did so. The others repeated the
authors’ conclusions
http://deevybee.blogspot.com.au/
2012/03/time-for-neuroimaging-to-
clean-up-its.html
14
15. post-publication
peer review
Absence of it means we
don’t have many
indicators of quality of
individual articles.
Reinforces reliance on
poor measures like
journal impact factor.
15
16. I have a dream.. our papers will be judged not
by the impact factor of their journal, but by the
quality of their content
16
17. I have a dream that one day,
our papers will be judged not by the
impact factor of their journal, but by
the quality of their content
from Peter Binfield’s talk
Impact Factor
announced (4.3)
17
18. ROIM- peer Review of Intro & Methods
• Original author sort-of signed off on it, so can’t complain / hate
the replication authors as much.
• Good way to start for a new PhD project, anyone planning to
build on some already-published results
• Reduce the incentive to publish flashy, headline-grabbing but
unreliable studies?
• • ✔
• How to incorporate post-publication commentary?
18
19. ROIM- peer Review of Intro & Methods
• What publisher to publish it?
19
20. Scientist meets publisher: the video
Academic knowledge is boxed
in by expensive journals.
http://www.youtube.com/watch?v=GMIY_4t-DR0
20
21. • Free
Why Open Access? • Full-text
• Online access
•Academia marginalising itself
The countries we work with can’t
•More impact! afford journals; they’re already
paying an arm and a leg for
textbooks -Sir John Daniel
•We do our research to
benefit anyone interested, not
some exclusive club The academic community
is only hurting itself, and
•Many scholars, doctors, its long term public
patients, engineers, support, by keeping its
policymakers (and esp. in poor knowledge behind high
countries/small universities) subscription walls -
Andrew Carr
can’t get access
21
22. OA HULK WANTS TO KNOW WHO TO
OCCUPY!
ELSEVIER!? ACS!? HARPERCOLLINS!?
YOU NAME IT, OA HULK WILL OCCUPY AND
SMASH!
Open Access “Hulk”
22
23. Institutional Subscription Cost, 2012
Experimental Brain Research $13,670
Journal of Radioanalytic and Nuclear Chemistry $19,826
Journal of Mathematical Sciences $17,880
Journal of Materials Science $16,699
$3983 USD per article for Elsevier
$1350 USD per article for PLoS ONE
Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html
23
24. $3983 USD per article for Elsevier
$1350 USD per article for PLoS ONE
Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html
24
26. How to help
•Deposit your manuscripts in the university repository
(http://ses.library.usyd.edu.au/
•Even with closed journals, you often have the right to
deposit your final version (e.g. Word document before
typeset by publisher)
•Best for university and its funders if research outputs open
access; indeed it’s mandated by:
•NIH, Wellcome Trust
Let me know i
•Princeton, Harvard
•Queensland University of Technology
•Support open-access publishing models
•PLoS, BioMed Central, eLife
•I don’t recommend paying for open-access “choice” in
closed corporate-published journals
•Support innovations that address problems of inefficiency
and incompleteness
26
27. Comprehensive solution?
Open Science
•As data comes in, uploaded
automatically to web
•Electronic lab notebook
•Papers written via open
collaborative documents on the web
27
29. Industry suffers less from such metrics, but it is nevertheless surprising
that industry were so heavily involved in this project. For example, of
the roughly 100 comments since January 2010 on The Synaptic
Leap website, around 60 came from readers not involved in the This stimu
kernel project at Sydney, and of those approximately 42 came
from industry, 16 from academia. Besides the input described above
to the resolution experiments, a different company contributed samples
of PZQ enantiomers isolated by chromatography for analytical
purposes, and another company is currently determining the phase
diagram of PZQamine. Why would companies choose to be involved,
particularly in a project in neglected tropical diseases where there is
little profit margin and no new intellectual property available? One can Mat Todd
appeal to human nature — we see a problem we can help solve, and
we find it impossible to resist stepping in, p
29
