This document discusses the author's inability to fully reproduce the computational experiments from their own past published work investigating the tuberculosis drugome. It provides context on the original experiments and analyses challenges in reproducing the work. Key findings include that missing details, parameter values and intermediate data in the original publication confound reproducibility for all skill levels. The document also raises questions about defining and incentivizing reproducibility as well as the role of tools, policies and publishers.

1. I Cannot Reproduce The Work
From My Own Laboratory
Philip E. Bourne
University of California San Diego
pbourne@ucsd.edu
D. Garijo, S. Kinnings, L. Xie, L. Xie, P.E. Bourne & Y. Gil 2013
Quantifying Reproducibility in Computational Biology:
The Case of the Tuberculosis Drugome. PLOS ONE,8(11): e80278

2. The Case of the Tuberculosis Drugome
Similarities between the binding sites of M.tb proteins (blue),
and binding sites containing approved drugs (red)
Kinnings et al 2010 PLoS Comp Biol 6(11): e1000976

3. Characteristics of the Original and
Current Experiment
• Original and Current:
– Purely in silico
– Uses a combination of public databases and open
source software by us and others
• Original:
– http://funsite.sdsc.edu/drugome/TB/
• Current:
– Recast in the Wings workflow system

4. Considered the Ability to Reproduce
by Four Classes of User
• REP-AUTHOR – original author of the work
• REP-EXPERT – domain expert – can reproduce
even with incomplete methods described
• REP-NOVICE – basic domain (bioinformatics)
expertise
• REP-MINIMAL – researcher with no domain
expertise

5. Rule #1: A Conceptual Overview of the
Method Should Always Be Included

6. Time to Reproduce the Method

7. Some Findings
• Reproducibility is a relative term eg p-value by novices
• The scripts reveal features of the method not found in
the paper and should be published/accessible
• Missing parameter values confound reproducibility
• Missing intermediate data confounds reproducibility
• Changing public data and software confounds exact
reproducibility – more versioning is required as is more
intermediate data

8. Some Thoughts 1/2
• Reproducibility has an associated cost : benefit
ratio
• Is there benefit to pre- vs post-publication
making of reproducibility
• Thus do we really care enough about
reproducibility?
• How much do workflows increase productivity?
• Tools help but policy change is required. What
should that policy be?

9. Some Thoughts 2/2
• If I take your experiment and make it
reproducible should I be rewarded? If yes how
much? Isn’t this like taking your data and
putting it in a database?
• Should the funders mandate reproducibility?
• Should publishers begin to accept workflows
and virtual machines?