icanhascheezburger.com
Results
may vary
reproducibility.
science.
software.
Professor Carole Goble
The University of Manch...
“An article about computational
science in a scientific publication
is not the scholarship itself, it is
merely advertisin...
http://www.nature.com/nature/focus/reproducibility/index.html
Corbyn, Nature Oct 2012fraud
“I can’t immediately reproduce the
research in my own laboratory. It
took an estimated 280 ho...
Reporting (publishing)
availability
documentation
Replication Gap
1. Ioannidis et al., 2009. Repeatability of published microarray gene expression analyses. Nature Genetics...
Stodden V, Guo P, Ma Z (2013) Toward Reproducible Computational Research: An
Empirical Analysis of Data and Code Policy Ad...
10 Simple Rules for Reproducible
Computational Research
1. For Every Result, Keep Track of How It Was
Produced
2. Avoid Ma...
republic of science*
regulation of science
institution
core facilities
libraries
*Merton’s four norms of scientific behavi...
recomputation.org
sciencecodemanifesto.org
meta-manifesto
• all X should be available and assessable
forever and ever
• the copyright of X should be clear
• X should...
re-compute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle...
Scientific publications have at least
two goals:
(i) to announce a result and
(ii) to convince readers that the
result is ...
Computational Research Virtual Witnessing
Methods
(techniques, algorithms, spec. of the steps)
Instruments
(codes, service...
reusereproduce
repeat replicate
same experiment
same lab
same experiment
different lab
same experiment
different set up
di...
DesignDesign
ExecutionExecution
Result AnalysisResult Analysis
CollectionCollection
PublishPublish
Peer
Review
Peer
Review...
portability
variability sameness
availability
open
description
intelligibility
[Adapted Freire, 2013]
preservation
packagi...
BioSTIF
method
instruments and laboratory
Workflows:
capture the steps
standardised pipelines
repetition & comparison
reco...
Provenance
the link between computation and results
Record
static verifiable record
partially repeat/reproduce
Track
track...
http://nbviewer.ipython.org/urls/raw.githubusercontent.com/myGrid/DataHackLeiden/alan/Player_example.ipynb?create=1
Workfl...
Open, citable workflows
[Scott Edmunds]
Integrative Framework
galaxyproject.org/
portability
variability sameness
availability
open
description
intelligibility
[Adapted Freire, 2013]
preservation
packagi...
Reporting dimension
Authoring
Exec. Papers
Link docs to experiment
Sweave
Provenance
Track,Version
Replay
Workflows, makef...
Aggregated Assets Infrastructures
Sharing and interlinking multi-stewarded
Methods, Models, Data…
Data
Model
Article
Exter...
made reproducible
[Pettifer, Attwood]
http://getutopia.com
portability
variability sameness
availability
open
description
intelligibility
[Adapted Freire, 2013]
preservation
packagi...
Archiving & Porting Dimension
host
service
Open Store
Sci as a Service
Integrative fws
Preservation
Recompute, limited
ins...
specialist codes
libraries, platforms, tools
services
(cloud)
hosted
services
commodity
platforms
data collections
catalog...
“lets copy the box that the
internet is in”
Archive
Isolation
• Independent
• Self contained
• Single ownership
• Freehold...
Closed codes/services, method
obscurity, manual steps
Joppa et al SCIENCE 340 2013, Morin et al SCIENCE 336 2012
Mitigate
...
The Reproducibility Window
all experiments become less reproducible over time
• The how, why and what
• plan to preserve
•...
The Reproducibility Window
The explicit documentation of designed-in
and anticipated variation
Reproducibility = Hard Work
Data sets
Analyses
Linked to
Linked to
DOI
DOI
Open-Paper
Open-Review
DOI:10.1186/2047-217X-1-...
DesignDesign
ExecutionExecution
Result AnalysisResult Analysis
CollectionCollection
PublishPublish
Peer
Review
Peer
Review...
Software sustainability
Software practices
Software deposition
Long term access to
software
Credit for software
Software J...
The Neylon Equation
Process =
Interest
Friction
x
Number
people
reach
Cameron Neylon, BOSC 2013, http://cameronneylon.net/...
productivity
reproducibility
personal
side effect
public
side effect
From make reproducible to
born reproducible
ramps
Research is
like software.
Release
research.
Jennifer Schopf, Treating Data Like Software: A Case for Production Quality D...
Research Objects
• Bundles and relate multi-hosted digital resources of a scientific experiment or
investigation using sta...
Research Objects for…..
Preservation
Archiving
Exchange & Communication
Release-based Publishing
Credit
Recombination/Remi...
identification
aggregation
annotation
dependencies
provenance
checklists
versioning
RO Core Conventions
encoded using stan...
RO Extensions
code
workflows
data
experiments
biology astronomy
NGS
SysBio
Mass Spec
Discipline
Asset type
Howard Ratner, Chair STM Future Labs Committee, CEO EVP Nature Publishing
Group, Director of Development for CHORUS (Clear...
Victoria Stodden, AMP 2011 http://www.stodden.net/AMP2011/,
Special Issue Reproducible Research Computing in Science and E...
http://sciencecodemanifesto.org/http://matt.might.net/articles/crapl/
Technical stuff is the easy stuff
Social
Matters
Organisation
MetricsCulture
Process
[Adapted, Daron Green]
meta-manifesto
all X should be available and assessable forever
the copyright of X should be clear
X should have citable, ...
• myGrid
– http://www.mygrid.org.uk
• Taverna
– http://www.taverna.org.uk
• myExperiment
– http://www.myexperiment.org
• B...
Acknowledgements
• David De Roure
• Tim Clark
• Sean Bechhofer
• Robert Stevens
• Christine Borgman
• Victoria Stodden
• M...
Results may vary: Collaborations Workshop, Oxford 2014
Results may vary: Collaborations Workshop, Oxford 2014
Upcoming SlideShare
Loading in …5
×

Results may vary: Collaborations Workshop, Oxford 2014

1,298 views

Published on

Thoughts on computational science reproducibility with a focus on software. Given at the Software Sustainability Institute's 2014 Collaborations Workshop

Published in: Science
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,298
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
22
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • how, why and what matters
    benchmarks for codes
    plan to preserve
    repair on demand
    description persists
    use frameworks
    partial replication
    approximate reproduction
    verification
  • Multidimensional paper
  • hand-wringing, weeping, wailing, gnashing of teeth.
    Nature checklist.
    Science requirements for data and code availability.
    attacks on authors, editors, reviewers, publishers, funders, and just about everyone.
  • Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124
    http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328
  • Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome
    Daniel Garijo,
    Sarah Kinnings,
    Li Xie,
    Lei Xie,
    Yinliang Zhang,
    Philip E. Bourne mail,
    Yolanda Gil mail
    Published: November 27, 2013
    DOI: 10.1371/journal.pone.0080278
  • Added afterwards.
    1. Required as condition of publication, certain exceptions permitted (e.g. preserving confidentiality of human subjects)
    2. Required but may not affect editorial/publication decisions
    3. Explicitly encouraged/addressed; may be reviewed and/or hosted
    4. Implied
    5. No mention
    59% of papers in the 50 highest-IF journals comply with (often weak) data sharing rules.
    Alsheikh-Ali et al Public Availability of Published Research Data in High-Impact Journals. PLoS ONE 6(9) 2011
  • Pressure from top, pressure from below
    Squeeze
    http://pantonprinciples.org/
  • anyone anything anytime
    publication access, data, models, source codes, resources, transparent methods, standards, formats, identifiers, apis, licenses, education, policies
    “accessible, intelligible, assessable, reusable”
  • The letter or the spirit of the experiment
    indirect and direct reproducibility
    Reproduce the same affect? Or same result?
    Concept drift towards bottom.
    As an old ontologist I wanted an ontology or a framework or some sort of property based classification.
  • / Minute Taking
    It examines the debate between Robert Boyle and Thomas Hobbes over Boyle's air-pump experiments in the 1660s. In 2005, Shapin and Schaffer were awarded the Erasmus Prize for this work.
  • Preservation - Lots of copies keeps stuff safe
    Stability dimension
    Add two more dimensions to our classification of themes
    A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System
    Overlap of course
    Static vs dynamic.
    GRANULARITY
    This model for audit and target of your systems
    overcoming data type silos
    public integrative data sets
    transparency matters
    cloud
    Recomputation.org
    Reproducibility by ExecutionRun It
    Reproducibility by InspectionRead It
    Availability – coverage
    Gathered: scattered across resources, across the paper and supplementary materials
    Availability of dependencies: Know and have all necessary elements
    Change management: Data? Services? Methods? Prevent, Detect, Repair.
    Execution and Making Environments: Skills/Infrastructure to run it: Portability and the Execution Platform (which can be people…), Skills/Infrastructure for authoring and reading
    Description: Explicit: How, Why, What, Where, Who, When, Comprehensive: Just Enough, Comprehensible: Independent understanding
    Documentation vs Bits (VMs) reproducibility
    Learn/understand (reproduce and validate, reproduce using different codes) vs Run (reuse, validate, repeat, reproduce under different configs/settings)
  • Example of an extreme of the software issue
    Multi-code experiments
    platform
    libraries, plugins
    Infrastructure
    components, services
    infrastructure
  • Simplify
    Track
    Versions and retractions
    Error propagation
    Contributions and credits
    Fix
    Workflow repair, alternate component discovery, Black box annotation
    Rerun and Replay
    Partial reproducibility: Replay some of the workflow
    A verifiable, reviewable trace in people terms
    Analyse
    Calculate data quality & trust,
    Decide what data to keep or release
    Compare to find differences and discrepancies
    S. Woodman, H. Hiden, P. Watson,  P. Missier Achieving Reproducibility by Combining Provenance with Service and Workflow Versioning. In: The 6th Workshop on Workflows in Support of Large-Scale Science. 2011, Seattle
  • Client package (currently under development, will be available via Python Package Index (PyPI) for installation for all major platforms (Linux, Mac, Windows)
    Allows for calling Taverna Workflows available via Taverna Player
    List of available workflows can be retrieved from the BioVel Portal (Taverna Player)
    Users can enter the input values using Ipython Notebook (these values can be then results of the code previously run in the Notebook
    The outputs from running the workflow (the results) are returned to the Notebook and processed further
    The full workflow run and the overall process (provenance) can be saved in the Ipython Notebook format
    For an example (static for now), see
    http://nbviewer.ipython.org/urls/raw.githubusercontent.com/myGrid/DataHackLeiden/alan/Player_example.ipynb?create=1
  • Used by gigscience
  • Preservation - Lots of copies keeps stuff safe
    Stability dimension
    Add two more dimensions to our classification of themes
    A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System
    Overlap of course
    Static vs dynamic.
    GRANULARITY
    This model for audit and target of your systems
    overcoming data type silos
    public integrative data sets
    transparency matters
    cloud
    Recomputation.org
    Reproducibility by ExecutionRun It
    Reproducibility by InspectionRead It
    Availability – coverage
    Gathered: scattered across resources, across the paper and supplementary materials
    Availability of dependencies: Know and have all necessary elements
    Change management: Data? Services? Methods? Prevent, Detect, Repair.
    Execution and Making Environments: Skills/Infrastructure to run it: Portability and the Execution Platform (which can be people…), Skills/Infrastructure for authoring and reading
    Description: Explicit: How, Why, What, Where, Who, When, Comprehensive: Just Enough, Comprehensible: Independent understanding
    Documentation vs Bits (VMs) reproducibility
    Learn/understand (reproduce and validate, reproduce using different codes) vs Run (reuse, validate, repeat, reproduce under different configs/settings)
  • Instrumented desktop or server tools
  • Variety:
    common metadata models
    rich metadata collection
    ecosystem
    Validity:
    auto record of experiment set-up, citable and shareable descriptions
    curation, publication,
    mixed stewardship
    third part availability
    model executability
    citability, QC/QA. trust.
    Social issues of understanding the culture of risk, reward, sharing and reporting.
  • This article by Phil Bourne et al doesn’t have any data sets deposited in repositories, but does include data in tables in the PDF, which are also available in the XML provided by PLoS. Here, Utopia has spotted that there’s a table of data (notice the little blue table icon to the left of the table). Clicking on the icon opens a window with a simple ‘spreadsheet’ of the data extracted from the paper, which you can then export in CSV to a proper spreadsheet of your choice. You can also scatter-plot the data to get a quick-and-dirty overview of what’s in the table.
  • Preservation - Lots of copies keeps stuff safe
    Stability dimension
    Add two more dimensions to our classification of themes
    A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System
    Overlap of course
    Static vs dynamic.
    GRANULARITY
    This model for audit and target of your systems
    overcoming data type silos
    public integrative data sets
    transparency matters
    cloud
    Recomputation.org
    Reproducibility by ExecutionRun It
    Reproducibility by InspectionRead It
    Availability – coverage
    Gathered: scattered across resources, across the paper and supplementary materials
    Availability of dependencies: Know and have all necessary elements
    Change management: Data? Services? Methods? Prevent, Detect, Repair.
    Execution and Making Environments: Skills/Infrastructure to run it: Portability and the Execution Platform (which can be people…), Skills/Infrastructure for authoring and reading
    Description: Explicit: How, Why, What, Where, Who, When, Comprehensive: Just Enough, Comprehensible: Independent understanding
    Documentation vs Bits (VMs) reproducibility
    Learn/understand (reproduce and validate, reproduce using different codes) vs Run (reuse, validate, repeat, reproduce under different configs/settings)
  • Recomputation not reproducibility
    ID it to Cite It: ORCID (people), DOI (data, models, tools ...)
    Tracking: local helper systems to instrument and track provenance
    Science as a Service: Virtual Machines, Cloud Appliances, Hosted platforms deploys on your behalf, no installations, common platforms
    Libraries and Repositories: with rich documentation
    Publish: executable papers, companion web sites, embedded electronic lab notebooks, active publications
    Explication of experimental mechanics: pipelines, workflows, script systems with documentation, common tools
  • the reproducibility ecosystem
    For peer and author
    complicated and scattered - super fragmentation – supplementary materials, multi-hosted, multi-stewarded.
    we must use the right platforms for the right tools
    The trials and tribulations of review
    Its Complicated
    www.biostars.org/
    Apache
    Service based ScienceScience as a Service
  • A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System
    Zhao, Gomez-Perez, Belhajjame, Klyne, Garcia-Cuesta, Garrido, Hettne, Roos, De Roure and Goble. Why workflows break - Understanding and combating decay in Taverna workflows, 8th Intl Conf e-Science 2012
    Reproducibility success is proportional to the number of dependent components and your control over them”
    Many reasons why.
    Change / Availability
    Updates to public datasets, changes to services / codes
    Availability/Access to components / execution environment
    Platform differences on simulations, code ports
    Volatile third-party resources (50%): Not available, available but inaccessible, changed
    Prevent, Detect, Repair
  • The only equation I have in the talk.
  • Added after LISC
  • ENCODE threads
    exchange between tools and researchers
    bundles and relates digital resources of a scientific experiment or investigation using standard mechanisms
  • http://www.youtube.com/watch?v=p-W4iLjLTrQ&list=PLC44A300051D052E5
    Our collaboration with ISA/GigaScience/nanopublication is finally being written up and will be submitted to ECCB this Friday. We will upload a copy to Arxiv after the deadline. - We will continue our workshop at ISMB, with BioMED Central. And Kaitlin will also join us on the Panel. You can find more details about agenda and panel planning in other emails.
    Posted on December 11, 2013 by Kaitlin Thaney
    Part of the Science Lab’s mission is to work with other community members to build technical prototypes that move science on the web forward. In particular, we want to show that many problems can be solved by making existing tools and technology work together, rather than by starting from scratch.The reason behind that is two-fold: (1) most of the stuff needed to change behaviors already exists in some form and (2) the smartest minds are usually outside of your organization.
    Our newest project extends our existing work around “code as a research object”, exploring how we can better integrate code and scientific software into the scholarly workflow. The project will test building a bridge that will allow users to push code from their GitHub repository to figshare, providing a Digital Object Identifier for the code (a gold standard of sorts in science, allowing persistent reference linking). We will also be working on a “best practice” standard (think a MIAME standard for code), so that each research object has sufficient documentation to make it possible to meaningfully use.
    The project will be a collaboration of the Science Lab with Arfon Smith (Github; co-founder Zooniverse) and Mark Hahnel and his team at figshare.
    Why code?
    Scientific research is becoming increasingly reliant on software. But despite there being an ever-increasing amount of the academic process described in code, research communities do not yet treat these products as a fundamental component or  “first-class research object” (see our background post here for more). Up until recent years, the sole “research object” in discussion was the published paper, the main means of packaging together the data, methods and research to communicate findings. The web is changing that, making it easier to unpack the components such as data and code for the community to remix, reuse, and build upon.
    A number of scientists are pushing the envelope, testing out new ways of bundling their code, data and methods together. But outside of copy and pasting lines of code into a paper or, if we’re lucky, having it included in a supplementary information file alongside a paper, the code is still often separated from the documentation needed for others to meaningfully use it to validate and reproduce experiments. And that’s if it’s shared openly at all.
    Code can go a long way in helping academia move toward the holy grail that is reproducibility. Unfortunately, academics whose main research output is the code they produce, often cannot get the recognition they deserve for creating it. There is also a problem with versioning:  citing a paper written about software (as is common practice), gives no indication of which version, or release in GitHub terms, was used to generate the results.
    What we’re testing
    figshare and GitHub are two of the leading repositories for data and code (figshare for data; GitHub for code). Open data repositories like figshare have led the way in recent years in changing our practices in relation to data, championing the idea of data as a first-class research object. figshare and others such as Harvard’s Dataverse and Dryad have helped change how we think of data as part of the research process, providing citable endpoints for the data itself that the community trusts (DOIs), as well as clear licensing and making it easy to download, remix, and reuse information. One of the main objectives here is that the exact code used in particular investigations, can be accessed by anyone and persists in the form it was in when cited.
    This project will test whether having a means of linking code repositories to those commonly used for data will allow for software and code to be better incorporated into existing credit systems (by having persistent identifiers for code snapshots) and how seamless we can make these workflows for academics using tools they are already familiar with. We’ve seen this tested with data over recent years, with sharing of detailed research data associated with increased citation rates (Piwowar, 2007). This culture shift of publishing more of the product of research is an ongoing process and we’re keen to see software and code elevated to the same status as the academic manuscript.
    We believe that by having the code closely associated with the data it executes on (or generates) will help reduce the barriers when trying to reproduce and build upon the work of others. This is already being tested in the reverse, with computational scientists nesting their data with the code in GitHub (Carl and Ethan, for example). We want to find out if formally linking the two to help ease that pain will change behavior.
    We are also looking to foster a culture of reuse with academic code. While we know there are lots of variables in this space, we are actively soliciting feedback from the community to help determine best practices for licensing and workflows.
    How to get involved
    (UPDATE: Want to help us test? Instead of sending us an email, how about adding yourself to this issue in our GitHub repository? More about that here.)
    Mark and Arfon will be joining us for our next Mozilla Science Lab community call on December 12, 2013. Join us to hear more about the project. Have a question you’d like to ask? Add it to the etherpad!
    We’re also looking for computational researchers and publishers to help us test out the implementation. Shoot us an email if you’d like to participate.
    Posted in Uncategorized.
  • Its people!!!
  • from make reproducible to born reproducible
    tools/repositories needed, maintained and incorporated into working practices
    researchers will need to adapt their practices, be trained to reproduce,
    cost and responsibility should be transparent, planned for, accounted and borne collectively
    we all should start small, be imperfect but take action. Today.
    spreading the cost
    cradle to grave reproducibility
    tools, processes, standards
    combine making & reporting
    just enough, imperfect
    cost in
    train up and support
    planning
    We cannot sacrifice the youth
    Protect them….a new generation
    Ecosystem of support tools navigation
  • Results may vary: Collaborations Workshop, Oxford 2014

    1. 1. icanhascheezburger.com Results may vary reproducibility. science. software. Professor Carole Goble The University of Manchester, UK The Software Sustainability Institute carole.goble@manchester.ac.uk @caroleannegoble Collaborations Workshop, Oxford, 26 March 2014
    2. 2. “An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment, [the complete data] and the complete set of instructions which generated the figures.” David Donoho, “Wavelab and Reproducible Research,” 1995 datasets data collections algorithms configurations tools and apps codes workflows scripts code libraries services, system software infrastructure, compilers hardware Morin et al Shining Light into Black Boxes Science 13 April 2012: 336(6078) 159-160 Ince et al The case for open computer progra Nature 482, 2012
    3. 3. http://www.nature.com/nature/focus/reproducibility/index.html
    4. 4. Corbyn, Nature Oct 2012fraud “I can’t immediately reproduce the research in my own laboratory. It took an estimated 280 hours for an average user to approximately reproduce the paper. Data/software versions. Workflows are maturing and becoming helpful” disorganisation Phil Bourne Garijo et al. 2013 Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome PLOS ONE, DOI: 10.1371/journal.pone.0080278. inherent
    5. 5. Reporting (publishing) availability documentation
    6. 6. Replication Gap 1. Ioannidis et al., 2009. Repeatability of published microarray gene expression analyses. Nature Genetics 41: 14 2. Science publishing: The trouble with retractions http://www.nature.com/news/2011/111005/full/478026a.html 3. Bjorn Brembs: Open Access and the looming crisis in science https://theconversation.com/open-access-and-the-looming-crisis-in-science-14950 Out of 18 microarray papers, results from 10 could not be reproduced Out of 18 microarray papers, results from 10 could not be reproduced
    7. 7. Stodden V, Guo P, Ma Z (2013) Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals. PLoS ONE 8(6): e67111. doi:10.1371/journal.pone.0067111 Required as condition of publication Required but may not affect decisions Explicitly encouraged may be reviewed and/or hosted Implied No mention Required as condition of publication Required but may not affect decisions Explicitly encouraged may be reviewed and/or hostedImplied No mention 170 journals, 2011-2012
    8. 8. 10 Simple Rules for Reproducible Computational Research 1. For Every Result, Keep Track of How It Was Produced 2. Avoid Manual Data Manipulation Steps 3. Archive the Exact Versions of All External Programs Used 4. Version Control All Custom Scripts 5. Record All Intermediate Results, When Possible in Standardized Formats 6. For Analyses That Include Randomness, Note Underlying Random Seeds 7. Always Store Raw Data behind Plots 8. Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected 9. Connect Textual Statements to Underlying Results 10. Provide Public Access to Scripts, Runs, and Results Citation: Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 Record Everything Automate Everything
    9. 9. republic of science* regulation of science institution core facilities libraries *Merton’s four norms of scientific behaviour (1942) public services
    10. 10. recomputation.org sciencecodemanifesto.org
    11. 11. meta-manifesto • all X should be available and assessable forever and ever • the copyright of X should be clear • X should have citable, versioned identifiers • researchers using X should visibly credit X’s creators • credit should be assessable and count in all assessments • X should be curated, available, linked to all necessary materials, and intelligible
    12. 12. re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle conceptual replication “show A is true by doing B rather than doing A again” verify but not falsify [Yong, Nature 485, 2012] regenerate the figure redo
    13. 13. Scientific publications have at least two goals: (i) to announce a result and (ii) to convince readers that the result is correct ….. papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension Jill Mesirov Accessible Reproducible Research Science 22 Jan 2010: 327(5964): 415-416 DOI: 10.1126/science.1179653 Virtual Witnessing* *Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer.
    14. 14. Computational Research Virtual Witnessing Methods (techniques, algorithms, spec. of the steps) Instruments (codes, services, scripts, underlying libraries) Laboratory (sw and hw infrastructure, systems software, integrative platforms) Materials (datasets, parameters, algorithm seeds) Experiment Setup
    15. 15. reusereproduce repeat replicate same experiment same lab same experiment different lab same experiment different set up different experiment some of same test Drummond C Replicability is not Reproducibility: Nor is it Good Science, online Peng RD, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
    16. 16. DesignDesign ExecutionExecution Result AnalysisResult Analysis CollectionCollection PublishPublish Peer Review Peer Review Peer Reuse Peer Reuse PredictionPrediction Can I repeat & defend my method? Can I review / reproduce and compare my results / method with your results / method? Can I review / replicate and certify your method? Can I transfer your results into my research and reuse this method? * Adapted from Mesirov, J. Accessible Reproducible Research Science 327(5964), 415-416 (2010)
    17. 17. portability variability sameness availability open description intelligibility [Adapted Freire, 2013] preservation packaging gather dependencies capture steps track & keep results gather dependencies capture steps track & keep results A Reproducibility Framework Reporting dimension Archive dimension versioning
    18. 18. BioSTIF method instruments and laboratory Workflows: capture the steps standardised pipelines repetition & comparison record experiment & set-up provenance collection reporting embedded player variant reuse infrastructure shield localised / distributed in-house / external multi-code experiments materials http://www.taverna.org.uk
    19. 19. Provenance the link between computation and results Record static verifiable record partially repeat/reproduce Track track changes carry citation select data to keep/release Analytics repair calc data quality/trust compare diffs/discrepancies W3C PROV standard d1 S0 d2 S1 w S2 y S4 df d1' S0 d2 S1 z w S'2 y' S4 df' (i) Trace A (ii) Trace B PDIFF: comparing provenance traces to diagnose divergence across experimental results [Woodman et al, 2011]
    20. 20. http://nbviewer.ipython.org/urls/raw.githubusercontent.com/myGrid/DataHackLeiden/alan/Player_example.ipynb?create=1 Workflows: sharing and reporting
    21. 21. Open, citable workflows [Scott Edmunds]
    22. 22. Integrative Framework galaxyproject.org/
    23. 23. portability variability sameness availability open description intelligibility [Adapted Freire, 2013] preservation packaging gather dependencies capture steps track & keep results gather dependencies capture steps track & keep results A Reproducibility Framework Reporting dimension Archive dimension versioning
    24. 24. Reporting dimension Authoring Exec. Papers Link docs to experiment Sweave Provenance Track,Version Replay Workflows, makefiles service Sci as a Service Integrative fws Read & Run, Co-location No installation host Open Store Descriptive read, White Box Archived record
    25. 25. Aggregated Assets Infrastructures Sharing and interlinking multi-stewarded Methods, Models, Data… Data Model Article External Databases http://www.seek4science.org Metadata http://www.isatools.org
    26. 26. made reproducible [Pettifer, Attwood] http://getutopia.com
    27. 27. portability variability sameness availability open description intelligibility [Adapted Freire, 2013] preservation packaging gather dependencies capture steps track & keep results gather dependencies capture steps track & keep results A Reproducibility Framework Reporting dimension Archive dimension versioning
    28. 28. Archiving & Porting Dimension host service Open Store Sci as a Service Integrative fws Preservation Recompute, limited installation, Black Box Byte execution Descriptive read, White Box Archived record Read & Run, Co-location No installation ReproZipPackaging Porting White Box, Installation Archived record
    29. 29. specialist codes libraries, platforms, tools services (cloud) hosted services commodity platforms data collections catalogues software repositories my data my process my codes integrative frameworks gateways
    30. 30. “lets copy the box that the internet is in” Archive Isolation • Independent • Self contained • Single ownership • Freehold • Fixed • Self described Active Ecosystem • Dependent • Distributed • Multi-ownership • Tenancy • Changeable / variable • Multi-described
    31. 31. Closed codes/services, method obscurity, manual steps Joppa et al SCIENCE 340 2013, Morin et al SCIENCE 336 2012 Mitigate Detect Repair Zhao, Gomez-Perez, Belhajjame, Klyne, Garcia-Cuesta, Garrido, Hettne, Roos, De Roure and Goble. Why workflows break - Understanding and combating decay in Taverna workflows, 8th Intl Conf e-Science 2012
    32. 32. The Reproducibility Window all experiments become less reproducible over time • The how, why and what • plan to preserve • prepare to repair • description persists • common frameworks • partial replication • approximate reproduction • verification • benchmarks for codes Reproducibility by Invocation Run It Reproducibility by Inspection Read It
    33. 33. The Reproducibility Window The explicit documentation of designed-in and anticipated variation
    34. 34. Reproducibility = Hard Work Data sets Analyses Linked to Linked to DOI DOI Open-Paper Open-Review DOI:10.1186/2047-217X-1-18 >11000 accesses Open-Code 8 reviewers tested data in ftp server & named reports published DOI:10.5524/100044 Open-Pipelines Open-Workflows DOI:10.5524/100038 Open-Data 78GB CC0 data Code in sourceforge under GPLv3: http://soapdenovo2.sourceforge.net/>5000 downloads Enabled code to being picked apart by bloggers in wiki http://homolog.us/wiki/index.php?title=SOAPdenovo2 [Scott Edmunds]
    35. 35. DesignDesign ExecutionExecution Result AnalysisResult Analysis CollectionCollection PublishPublish Peer Review Peer Review Peer Reuse Peer Reuse PredictionPrediction * Adapted from Mesirov, J. Accessible Reproducible Research Science 327(5964), 415-416 (2010) Reproducible Research Environment Integrated infrastructure for producing and working with reproducible research. Reproducible Research Publication Environment Distributing and reviewing; credit; licensing etc. From make reproducible to born reproducible
    36. 36. Software sustainability Software practices Software deposition Long term access to software Credit for software Software Journals Licensing Open Source Software Best Practices for Scientific Computing http://arxiv.org/abs/1210.0530 Stodden, Reproducible Research Standard, Intl J Comm Law & Policy, 13 2009 From make reproducible to born reproducible From make reproducible to born reproducible
    37. 37. The Neylon Equation Process = Interest Friction x Number people reach Cameron Neylon, BOSC 2013, http://cameronneylon.net/ From make reproducible to born reproducible
    38. 38. productivity reproducibility personal side effect public side effect From make reproducible to born reproducible ramps
    39. 39. Research is like software. Release research. Jennifer Schopf, Treating Data Like Software: A Case for Production Quality Data, JCDL 2012 From make reproducible to born reproducible
    40. 40. Research Objects • Bundles and relate multi-hosted digital resources of a scientific experiment or investigation using standard mechanisms • Exchange, Releasing paradigm for publishing http://www.researchobject.org/
    41. 41. Research Objects for….. Preservation Archiving Exchange & Communication Release-based Publishing Credit Recombination/Remix Reproducibility, Computation Training
    42. 42. identification aggregation annotation dependencies provenance checklists versioning RO Core Conventions encoded using standards Minim Information Model Ontology W3C PROV PAV, VoID Git OAI-ORE W3C OAM DOI, ORCID, PURL
    43. 43. RO Extensions code workflows data experiments biology astronomy NGS SysBio Mass Spec Discipline Asset type
    44. 44. Howard Ratner, Chair STM Future Labs Committee, CEO EVP Nature Publishing Group, Director of Development for CHORUS (Clearinghouse for the Open Research of US) STM Innovations Seminar 2012 http://www.youtube.com/watch?v=p-W4iLjLTrQ&list=PLC44A300051D052E5
    45. 45. Victoria Stodden, AMP 2011 http://www.stodden.net/AMP2011/, Special Issue Reproducible Research Computing in Science and Engineering July/August 2012, 14(4) Howison and Herbsleb (2013) "Incentives and Integration In Scientific Software Production" CSCW 2013.
    46. 46. http://sciencecodemanifesto.org/http://matt.might.net/articles/crapl/
    47. 47. Technical stuff is the easy stuff Social Matters Organisation MetricsCulture Process [Adapted, Daron Green]
    48. 48. meta-manifesto all X should be available and assessable forever the copyright of X should be clear X should have citable, versioned identifiers researchers using X should visibly credit X’s creators credit should be assessable and count in all assessments X should be curated, available, linked to all necessary materials, and intelligible • reproducibility spectrum • descriptive reproducibility • papers -> research objects • make reproducible -> born reproducible • ramp up tools -> working practice • adapt and train -> researchers • cost & responsibility -> transparent, accountable and collective • dominants -> society, culture and policy • take action, be imperfect
    49. 49. • myGrid – http://www.mygrid.org.uk • Taverna – http://www.taverna.org.uk • myExperiment – http://www.myexperiment.org • BioCatalogue – http://www.biocatalogue.org • Biodiversity Catalogue – http://www.biodiversitycatalogue.org • Seek – http://www.seek4science.org • Rightfield – http://www.rightfield.org.uk • Open PHACTS – http://www.openphacts.org • Wf4ever – http://www.wf4ever-project.org • Software Sustainability Institute – http://www.software.ac.uk • BioVeL – http://www.biovel.eu • Force11 – http://www.force11.org
    50. 50. Acknowledgements • David De Roure • Tim Clark • Sean Bechhofer • Robert Stevens • Christine Borgman • Victoria Stodden • Marco Roos • Jose Enrique Ruiz del Mazo • Oscar Corcho • Ian Cottam • Steve Pettifer • Magnus Rattray • Chris Evelo • Katy Wolstencroft • Robin Williams • Pinar Alper • C. Titus Brown • Greg Wilson • Kristian Garza • Wf4ever, SysMO, BioVel, UTOPIA and myGrid teams • Juliana Freire • Jill Mesirov • Simon Cockell • Paolo Missier • Paul Watson • Gerhard Klimeck • Matthias Obst • Jun Zhao • Pinar Alper • Daniel Garijo • Yolanda Gil • James Taylor • Alex Pico • Sean Eddy • Cameron Neylon • Barend Mons • Kristina Hettne • Stian Soiland-Reyes • Rebecca Lawrence

    ×