Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

What is reproducibility goble-clean

1,828 views

Published on

What is Reproducibility? “When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.”[1]. Its the same with Reproducible. Reusable. Repeatable. Recomputable. Replicable. Rerunnable. Regeneratable. Reviewable. Its R* mayhem. Or pride [2]. Does it matter? At least it does for computational science. Different shades of "reproducible" matter in the publishing workflow, depending on whether you are testing for robustness (rerun), defence (repeat), certification (replicate), comparison (reproduce) or transferring between researchers (reuse). Different forms of "R" make different demands on the completeness, depth and portability of research [3].
If we view computational tools (software, scripts) as instruments - “data scopes” rather than “telescopes” or “microscopes” – then we need to be clear when we talk about reproducible computational experiments about whether we are rerunning with the same set up on the same (preserved) instrument (say a Virtual Machine), or reproducing the instrument to replicate the experiment (say a description of an algorithm recoded) or repairing the instrument so we can reuse it for some other experiment (say replacing a defunct web service or a deprecated library). Confused? In this talk I will discuss the R* brouhaha and its practical consequences for computational data driven science.
[1] Lewis Carroll, Through the Looking-Glass (1872)
[2] David De Roure, More Rs than Pirates http://www.scilogs.com/eresearch/more-rs-than-pirates/
[3] Juliana Freire, Philippe Bonnet, Dennis Shasha, Computational reproducibility: state-of-the-art, challenges, and database research opportunities SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data: 593-596, ACM New York, NY, USA, doi>10.1145/2213836.2213908

Published in: Science

What is reproducibility goble-clean

  1. 1. What is Reproducibility? The R* Brouhaha Professor Carole Goble The University of Manchester, UK Software Sustainability Institute UK carole.goble@manchester.ac.uk AlanTuring Institute Symposium Reproducibility, Sustainability and Preservation , 6-7 April 2016, Oxford, UK
  2. 2. “When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.” Carroll,Through the Looking Glass re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle redo robustness tolerance verificationcompliancevalidation assurance remix
  3. 3. Reproducibility of Reproducibility Research
  4. 4. Even in Computer Science
  5. 5. http://www.dagstuhl.de/16041 24. – 29. January 2016, Dagstuhl Seminar 16041 Reproducibility of Data-Oriented Experiments in e-Science
  6. 6. Computational Workflow
  7. 7. 10 Simple Rules for Reproducible Computational Research 1. For Every Result, Keep Track of How It Was Produced 2. Avoid Manual Data Manipulation Steps 3. Archive the Exact Versions of All External Programs Used 4. Version Control All Custom Scripts 5. Record All Intermediate Results, When Possible in Standardized Formats 6. For Analyses That Include Randomness, Note Underlying Random Seeds 7. Always Store Raw Data behind Plots 8. Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected 9. Connect Textual Statements to Underlying Results 10. Provide Public Access to Scripts, Runs, and Results Sandve GK, Nekrutenko A,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 Record Everything Automate Everything
  8. 8. Scientific publications goals: (i) announce a result and (ii) convince readers that the result is correct. Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension. Papers in computational science should describe the results and provide the complete software development environment, data and set of instructions which generated the figures. Virtual Witnessing* *Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer. Jill Mesirov David Donoho
  9. 9. Datasets, Data collections Standard operating procedures Software, algorithms Configurations, Tools and apps, services Codes, code libraries Workflows, scripts System software Infrastructure Compilers, hardware
  10. 10. Analogy: The Lab data science | data-driven science 1. Philosophy 2. Practice
  11. 11. Witnessing “Datascopes” Input Data Software Output Data Config Parameters Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, , ref resources Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
  12. 12. Model Driven Science – can I rerun my model? Model Sweeps,What are the sensitivities?
  13. 13. “Micro” Reproducibility “Macro” Reproducibility
  14. 14. Repeat, Replicate, Robust Why the differences? Reproduce [CTitus Brown] https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html
  15. 15. Computational analyses Productivity Track differences
  16. 16. Computational analyses? Repeatability: Sameness Same result 1 Lab 1 experiment Reproducibility: Similarity Similar result > 1 Lab > 1 experiment
  17. 17. “an experiment is reproducible until another laboratory tries to repeat it.” Alexander Kohn
  18. 18. reviewers want additional work statistician wants more runs analysis needs to be repeated post-doc leaves, student arrives new data, revised data updated versions of algorithms/codes sample was contaminated
  19. 19. Measuring Information Gain from Reproducibility Research goal Method/Alg. Platform/Exec Env Data Parameters Input data Actors Information Gain Implementation/Code No change Change Don’t care https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/ http://www.dagstuhl.de/16041
  20. 20. Taxonomy of actions towards improving reproducibility in Computer Science. https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/ http://www.dagstuhl.de/16041
  21. 21. Practice
  22. 22. Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment Input Data Software Output Data Config Parameters “Datascope” Entropy -> Preservation “Replication / Reproducibility Window”
  23. 23. Change: Science, methods, datasets Questions don’t change, answers do. Materials unavailable one offs, streams, stochastics, sensitivities, licensing scale, non-portable data Change: Instruments break, labs decay Active ref datasets and services Platforms & resources unavailable supercomputer scale non-portable software “Datascope” Entropy -> Preservation “Replication / Reproducibility Window” Archived vs Active Environment Isolated vs Open Distributed Ecosystem
  24. 24. T1 T2 Evolving Reference Knowledge Bases e.g. UNIPROT
  25. 25. Repeat harder than Reproduce? Repeating the experiment or the set up? • When the environment is active
  26. 26. “Datascope” Entropy -> Preservation “Replication / Reproducibility Window” Form? Function? Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
  27. 27. How? Preserve by Reporting, Reproduce by Reading ProvenanceTraces, Notebooks, Rich Metadata Archived Record Methods Materials Instruments Laboratory
  28. 28. How? Preserve by Reporting, Reproduce by Reading ProvenanceTraces, Notebooks, Rich Metadata Archived Record standards, common metadata Provenance Workflows, Scripts ELNs Methods Materials Instruments Laboratory
  29. 29. How? Preserve by Maintaining, Repairing,VMs Reproduce by Running, Emulating, Reconstructing Active Instrument Methods Materials Instruments Laboratory
  30. 30. How? Preserve by Maintaining, Repairing,VMs Reproduce by Running, Emulating, Reconstructing Active Instrument, Byte level Methods Materials Instruments Laboratory
  31. 31. Levels of Computational Reproducibility Coverage: how much of an experiment is reproducible OriginalExperimentSimilarExperimentDifferentExperiment Portability Depth: how much of an experiment is available Binaries + Data Source Code / Workflow + Data Binaries + Data + Dependencies Source Code / Workflow + Data + Dependencies Virtual Machine Binaries + Data + Dependencies Virtual Machine Source Code / Workflow + Data + Dependencies Figures + Data [Freire, 2014] Minimum: data and source code available under terms that permit inspection and execution.
  32. 32. Repeatable Environments *https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html [C.Titus Brown*] Metadata Objects : Reproducible Reporting, Exchange Checklist ProvenanceTracking Versioning Dependencies container
  33. 33. provenance dependencies steps, features transparency portability robustness preservation access description available standards common APIs licensing identifiers standards, common metadata change variation sensitivity versioning packaging So, What is Reproducibility? Being FAIR
  34. 34. So, What is Reproducibility? Being FAIR
  35. 35. What is Reproducibility? Why, When, Where, Who for, Who by, How Special thanks to • C Titus Brown • Juliana Freire • David De Roure • Stian Soiland-Reyes • Barend Mons • Tim Clark • Daniel Garijo • Wf4Ever and Research Object teams • Dagstuhl Seminar 16041 • Force11 http://www.force11.org

×