GigaScience EiC Laurie Goodman's talk at the Council of Science Editors Annual Meeting on "Laurie Goodman at #CSE2014: Reproducibility: It's going to cost you time and effort, but it's our job!". 4th MAy 2014

1. Reproducibility
Laurie Goodman, PhD
Editor-in-Chief, GigaScience
Laurie@gigasciencejournal.com
Mostly Time & Effort
It’s Going to Cost You
But It’s Our Job

2. Growing Issue: increasing number of retractions
>15X increase in last decade
Strong correlation of “retraction index” with
higher impact factor
1. Science publishing: The trouble with retractions http://www.nature.com/news/2011/111005/full/478026a.html
2. Retracted Science and the Retraction Index ▿ http://iai.asm.org/content/79/10/3855.abstract?
At current % increase by 2045 as
many papers published as
retracted!

3. Problem: Growing Replication Gap
1. Ioannidis et al., (2009). Repeatability of published microarray gene expression analyses. Nature Genetics 41: 14
2. Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8)
Out of 18 microarray papers, results
from 10 could not be reproduced

4. The problems with publishing
• Scholarly articles are merely advertisement of scholarship .
The actual scholarly artefacts, i.e. the data and
computational methods, which support the scholarship,
remain largely inaccessible --- Jon B. Buckheit and David L.
Donoho, WaveLab and reproducible research, 1995
• Core scientific statements or assertions are intertwined and
hidden in the conventional scholarly narratives
• Lack of transparency, lack of credit for anything other than
“regular” dead tree publication

5. GigaSolution: deconstructing the paper
Need to credit and reward:
• Data/software availability
• Metadata/curation
• Interoperability
• Availability of workflows
• Transparent analyses
Data
Metadata
Methods
Analyses
Every Component
Has a Citable DOI

6. How are we supporting reproducibility?
Data Sets in
GigaDB
Analyses in
GigaGalaxy
Paper in
GigaScience
Open-access journal Data Publishing Platform
Data Analysis Platform

7. Example in Neuroscience
• Neuroscience Data are
not typically shared
• Data AND Tools are not
typically made
available to the
reviewers
• Journal Editors think
Reviewers will not want
to review data
GigaScience 2014, 3:3 doi:10.1186/2047-217X-3-3

8. Example in Neuroscience
• Neuroscience Data are not typically shared
• Author Dr. Stephen Eglen said: “One way of encouraging neuroscientists to
share their data is to provide some form of academic credit.”
• We hosted with a DOI: 366 recordings from 12 electrophysiology datasets
• GigaDB is included in Thompson Reuters Data Citation Index
• Data AND Tools are not typically made available to the reviewers
• We made manuscript, data and tools all available to the reviewers.
• We make sure to include reviewers who are able to properly assess the data
itself and rerun the tools
• To reduce burdens- we sometimes select a reviewer who ONLY looks at the
data.
• Journal Editors think Reviewers will not want to review data
• What Reviewer Dr. Thomas Wachtler said: “The paper by Eglen and
colleagues is a shining example of openness in that it enables replicating the
results almost as easily as by pressing a button.”
• What Reviewer Dr. Christophe Pouzat said: “In addition to making the
presented research trustworthy, the reproducible research paradigm
definitely makes the reviewers job more fun!”

9. Reviewer Comments on the Process
http://www.biomedcentral.com/biome/christophe-pouzat-and-thomas-wachtler-on-reproducible-research-in-neuroscience/
http://blogs.biomedcentral.com/gigablog/2014/04/16/qa-on-dynamic-documents/

10. Reviewer Take Home Message
http://www.biomedcentral.com/biome/christophe-pouzat-and-thomas-wachtler-on-reproducible-research-in-neuroscience/
http://blogs.biomedcentral.com/gigablog/2014/04/16/qa-on-dynamic-documents/

11. But This Can’t Be
Done For Wet Bench!!!
And Make All Data Available
Take steps to put all the images and raw data in a repository,
and to use biobanking where possible
Address Insufficient-Method
Syndrome:
-Complete, Easily found,
Searchable, and Updatable

12. Lessons learned:
• It is possible to recreate a result from a paper
• Reproducibility is COSTLY. How much are you
willing to spend?
• Learn a huge amount about the study, and provides lots
of information not present in the paper
• Much easier to do this before rather than after
publication

13. It’s Time to Move Beyond
Dead Trees
18121665 1869

14. Thanks to:
editorial@gigasciencejournal.com
database@gigasciencejournal.com
@gigascience
facebook.com/GigaScience
blogs.openaccesscentral.com/blogs/gigablog/
Contact us:
Scott Edmunds, Executive Editor
Nicole Nogoy, Commissioning Editor
Peter Li, Lead Data Manager
Chris Hunter, Lead BioCurator
Rob Davidson, Data Scientist
Xiao (Jesse) Si Zhe, Database Developer
Amye Kenall, Journal Development Manager
Follow us:
www.gigasciencejournal.com
www.gigadb.org