Downloaded 48 times
![Repeat, Replicate, Robust
Why the differences?
Reproduce
[CTitus Brown]
https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html](https://image.slidesharecdn.com/whatisreproducibility-goble-clean-160406120003/85/What-is-reproducibility-goble-clean-14-320.jpg?cb=1459944348)
![Levels of Computational Reproducibility
Coverage: how
much of an
experiment is
reproducible
OriginalExperimentSimilarExperimentDifferentExperiment
Portability
Depth: how much of an experiment is available
Binaries +
Data
Source Code /
Workflow
+ Data
Binaries +
Data +
Dependencies
Source Code /
Workflow
+ Data +
Dependencies
Virtual Machine
Binaries +
Data +
Dependencies
Virtual Machine
Source Code /
Workflow
+ Data +
Dependencies
Figures +
Data
[Freire, 2014]
Minimum:
data and source
code available
under terms
that permit
inspection and
execution.](https://image.slidesharecdn.com/whatisreproducibility-goble-clean-160406120003/85/What-is-reproducibility-goble-clean-31-320.jpg?cb=1459944348)
![Repeatable Environments
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
[C.Titus Brown*]
Metadata Objects : Reproducible Reporting, Exchange
Checklist
ProvenanceTracking
Versioning
Dependencies
container](https://image.slidesharecdn.com/whatisreproducibility-goble-clean-160406120003/85/What-is-reproducibility-goble-clean-32-320.jpg?cb=1459944348)
Uploaded byCarole Goble
What is reproducibility goble-clean
What is Reproducibility? “When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.”[1]. Its the same with Reproducible. Reusable. Repeatable. Recomputable. Replicable. Rerunnable. Regeneratable. Reviewable. Its R* mayhem. Or pride [2]. Does it matter? At least it does for computational science. Different shades of "reproducible" matter in the publishing workflow, depending on whether you are testing for robustness (rerun), defence (repeat), certification (replicate), comparison (reproduce) or transferring between researchers (reuse). Different forms of "R" make different demands on the completeness, depth and portability of research [3]. If we view computational tools (software, scripts) as instruments - “data scopes” rather than “telescopes” or “microscopes” – then we need to be clear when we talk about reproducible computational experiments about whether we are rerunning with the same set up on the same (preserved) instrument (say a Virtual Machine), or reproducing the instrument to replicate the experiment (say a description of an algorithm recoded) or repairing the instrument so we can reuse it for some other experiment (say replacing a defunct web service or a deprecated library). Confused? In this talk I will discuss the R* brouhaha and its practical consequences for computational data driven science. [1] Lewis Carroll, Through the Looking-Glass (1872) [2] David De Roure, More Rs than Pirates http://www.scilogs.com/eresearch/more-rs-than-pirates/ [3] Juliana Freire, Philippe Bonnet, Dennis Shasha, Computational reproducibility: state-of-the-art, challenges, and database research opportunities SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data: 593-596, ACM New York, NY, USA, doi>10.1145/2213836.2213908
More Related Content
More from Carole Goble
What is reproducibility goble-clean
- 1. What is Reproducibility? TheR* Brouhaha Professor Carole Goble The University of Manchester, UK Software Sustainability Institute UK carole.goble@manchester.ac.uk AlanTuring Institute Symposium Reproducibility, Sustainability and Preservation , 6-7 April 2016, Oxford, UK
- 2. “When I usea word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.” Carroll,Through the Looking Glass re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle redo robustness tolerance verificationcompliancevalidation assurance remix
- 3.
- 4. Even in ComputerScience
- 5. http://www.dagstuhl.de/16041 24. – 29.January 2016, Dagstuhl Seminar 16041 Reproducibility of Data-Oriented Experiments in e-Science
- 6.
- 7. 10 Simple Rulesfor Reproducible Computational Research 1. For Every Result, Keep Track of How It Was Produced 2. Avoid Manual Data Manipulation Steps 3. Archive the Exact Versions of All External Programs Used 4. Version Control All Custom Scripts 5. Record All Intermediate Results, When Possible in Standardized Formats 6. For Analyses That Include Randomness, Note Underlying Random Seeds 7. Always Store Raw Data behind Plots 8. Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected 9. Connect Textual Statements to Underlying Results 10. Provide Public Access to Scripts, Runs, and Results Sandve GK, Nekrutenko A,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 Record Everything Automate Everything
- 8. Scientific publications goals: (i)announce a result and (ii) convince readers that the result is correct. Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension. Papers in computational science should describe the results and provide the complete software development environment, data and set of instructions which generated the figures. Virtual Witnessing* *Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer. Jill Mesirov David Donoho
- 9. Datasets, Data collections Standardoperating procedures Software, algorithms Configurations, Tools and apps, services Codes, code libraries Workflows, scripts System software Infrastructure Compilers, hardware
- 10. Analogy: The Lab datascience | data-driven science 1. Philosophy 2. Practice
- 11. Witnessing “Datascopes” Input Data Software OutputData Config Parameters Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, , ref resources Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
- 12. Model Driven Science– can I rerun my model? Model Sweeps,What are the sensitivities?
- 13.
- 14. Repeat, Replicate, Robust Whythe differences? Reproduce [CTitus Brown] https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html
- 15.
- 16. Computational analyses? Repeatability: Sameness Same result 1 Lab 1experiment Reproducibility: Similarity Similar result > 1 Lab > 1 experiment
- 17. “an experiment isreproducible until another laboratory tries to repeat it.” Alexander Kohn
- 18. reviewers want additional work statisticianwants more runs analysis needs to be repeated post-doc leaves, student arrives new data, revised data updated versions of algorithms/codes sample was contaminated
- 19. Measuring Information Gainfrom Reproducibility Research goal Method/Alg. Platform/Exec Env Data Parameters Input data Actors Information Gain Implementation/Code No change Change Don’t care https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/ http://www.dagstuhl.de/16041
- 20. Taxonomy of actionstowards improving reproducibility in Computer Science. https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/ http://www.dagstuhl.de/16041
- 21.
- 22. Methods techniques, algorithms, spec. ofthe steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment Input Data Software Output Data Config Parameters “Datascope” Entropy -> Preservation “Replication / Reproducibility Window”
- 23. Change: Science, methods, datasets Questionsdon’t change, answers do. Materials unavailable one offs, streams, stochastics, sensitivities, licensing scale, non-portable data Change: Instruments break, labs decay Active ref datasets and services Platforms & resources unavailable supercomputer scale non-portable software “Datascope” Entropy -> Preservation “Replication / Reproducibility Window” Archived vs Active Environment Isolated vs Open Distributed Ecosystem
- 24. T1 T2 Evolving Reference KnowledgeBases e.g. UNIPROT
- 25. Repeat harder thanReproduce? Repeating the experiment or the set up? • When the environment is active
- 26. “Datascope” Entropy ->Preservation “Replication / Reproducibility Window” Form? Function? Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
- 27. How? Preserve byReporting, Reproduce by Reading ProvenanceTraces, Notebooks, Rich Metadata Archived Record Methods Materials Instruments Laboratory
- 28. How? Preserve by Reporting,Reproduce by Reading ProvenanceTraces, Notebooks, Rich Metadata Archived Record standards, common metadata Provenance Workflows, Scripts ELNs Methods Materials Instruments Laboratory
- 29. How? Preserve byMaintaining, Repairing,VMs Reproduce by Running, Emulating, Reconstructing Active Instrument Methods Materials Instruments Laboratory
- 30. How? Preserve byMaintaining, Repairing,VMs Reproduce by Running, Emulating, Reconstructing Active Instrument, Byte level Methods Materials Instruments Laboratory
- 31. Levels of ComputationalReproducibility Coverage: how much of an experiment is reproducible OriginalExperimentSimilarExperimentDifferentExperiment Portability Depth: how much of an experiment is available Binaries + Data Source Code / Workflow + Data Binaries + Data + Dependencies Source Code / Workflow + Data + Dependencies Virtual Machine Binaries + Data + Dependencies Virtual Machine Source Code / Workflow + Data + Dependencies Figures + Data [Freire, 2014] Minimum: data and source code available under terms that permit inspection and execution.
- 32. Repeatable Environments *https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html [C.Titus Brown*] MetadataObjects : Reproducible Reporting, Exchange Checklist ProvenanceTracking Versioning Dependencies container
- 33. provenance dependencies steps, features transparency portability robustness preservation access description available standards commonAPIs licensing identifiers standards, common metadata change variation sensitivity versioning packaging So, What is Reproducibility? Being FAIR
- 34. So, What isReproducibility? Being FAIR
- 35. What is Reproducibility? Why,When, Where, Who for, Who by, How Special thanks to • C Titus Brown • Juliana Freire • David De Roure • Stian Soiland-Reyes • Barend Mons • Tim Clark • Daniel Garijo • Wf4Ever and Research Object teams • Dagstuhl Seminar 16041 • Force11 http://www.force11.org