I Cannot Reproduce The Work
From My Own Laboratory
Philip E. Bourne
University of California San Diego
pbourne@ucsd.edu

D...
The Case of the Tuberculosis Drugome

Similarities between the binding sites of M.tb proteins (blue),
and binding sites co...
Characteristics of the Original and
Current Experiment
• Original and Current:
– Purely in silico
– Uses a combination of ...
Considered the Ability to Reproduce
by Four Classes of User
• REP-AUTHOR – original author of the work
• REP-EXPERT – doma...
Rule #1: A Conceptual Overview of the
Method Should Always Be Included
Time to Reproduce the Method
Some Findings
• Reproducibility is a relative term eg p-value by novices
• The scripts reveal features of the method not f...
Some Thoughts 1/2
• Reproducibility has an associated cost : benefit
ratio
• Is there benefit to pre- vs post-publication
...
Some Thoughts 2/2
• If I take your experiment and make it
reproducible should I be rewarded? If yes how
much? Isn’t this l...
Upcoming SlideShare
Loading in …5
×

I Cannot Reproduce The Work From My Own Laboratory

1,274 views
1,137 views

Published on

Presented at the EBI Data and Literature Integration Workshop, UK, December 11, 2013.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,274
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

I Cannot Reproduce The Work From My Own Laboratory

  1. 1. I Cannot Reproduce The Work From My Own Laboratory Philip E. Bourne University of California San Diego pbourne@ucsd.edu D. Garijo, S. Kinnings, L. Xie, L. Xie, P.E. Bourne & Y. Gil 2013 Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome. PLOS ONE,8(11): e80278
  2. 2. The Case of the Tuberculosis Drugome Similarities between the binding sites of M.tb proteins (blue), and binding sites containing approved drugs (red) Kinnings et al 2010 PLoS Comp Biol 6(11): e1000976
  3. 3. Characteristics of the Original and Current Experiment • Original and Current: – Purely in silico – Uses a combination of public databases and open source software by us and others • Original: – http://funsite.sdsc.edu/drugome/TB/ • Current: – Recast in the Wings workflow system
  4. 4. Considered the Ability to Reproduce by Four Classes of User • REP-AUTHOR – original author of the work • REP-EXPERT – domain expert – can reproduce even with incomplete methods described • REP-NOVICE – basic domain (bioinformatics) expertise • REP-MINIMAL – researcher with no domain expertise
  5. 5. Rule #1: A Conceptual Overview of the Method Should Always Be Included
  6. 6. Time to Reproduce the Method
  7. 7. Some Findings • Reproducibility is a relative term eg p-value by novices • The scripts reveal features of the method not found in the paper and should be published/accessible • Missing parameter values confound reproducibility • Missing intermediate data confounds reproducibility • Changing public data and software confounds exact reproducibility – more versioning is required as is more intermediate data
  8. 8. Some Thoughts 1/2 • Reproducibility has an associated cost : benefit ratio • Is there benefit to pre- vs post-publication making of reproducibility • Thus do we really care enough about reproducibility? • How much do workflows increase productivity? • Tools help but policy change is required. What should that policy be?
  9. 9. Some Thoughts 2/2 • If I take your experiment and make it reproducible should I be rewarded? If yes how much? Isn’t this like taking your data and putting it in a database? • Should the funders mandate reproducibility? • Should publishers begin to accept workflows and virtual machines?

×