What is Reproducibility? “When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.”[1]. Its the same with Reproducible. Reusable. Repeatable. Recomputable. Replicable. Rerunnable. Regeneratable. Reviewable. Its R* mayhem. Or pride [2]. Does it matter? At least it does for computational science. Different shades of "reproducible" matter in the publishing workflow, depending on whether you are testing for robustness (rerun), defence (repeat), certification (replicate), comparison (reproduce) or transferring between researchers (reuse). Different forms of "R" make different demands on the completeness, depth and portability of research [3]. If we view computational tools (software, scripts) as instruments - “data scopes” rather than “telescopes” or “microscopes” – then we need to be clear when we talk about reproducible computational experiments about whether we are rerunning with the same set up on the same (preserved) instrument (say a Virtual Machine), or reproducing the instrument to replicate the experiment (say a description of an algorithm recoded) or repairing the instrument so we can reuse it for some other experiment (say replacing a defunct web service or a deprecated library). Confused? In this talk I will discuss the R* brouhaha and its practical consequences for computational data driven science. [1] Lewis Carroll, Through the Looking-Glass (1872) [2] David De Roure, More Rs than Pirates http://www.scilogs.com/eresearch/more-rs-than-pirates/ [3] Juliana Freire, Philippe Bonnet, Dennis Shasha, Computational reproducibility: state-of-the-art, challenges, and database research opportunities SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data: 593-596, ACM New York, NY, USA, doi>10.1145/2213836.2213908

1. What is Reproducibility?
The R* Brouhaha
Professor Carole Goble
The University of Manchester, UK
Software Sustainability Institute UK
carole.goble@manchester.ac.uk
AlanTuring Institute Symposium Reproducibility, Sustainability and Preservation , 6-7 April 2016, Oxford, UK

2. “When I use a word," Humpty Dumpty
said in rather a scornful tone, "it means
just what I choose it to mean - neither
more nor less.”
Carroll,Through the Looking Glass
re-compute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle
redo
robustness
tolerance
verificationcompliancevalidation assurance
remix

3. Reproducibility of
Reproducibility Research

4. Even in Computer Science

5. http://www.dagstuhl.de/16041
24. – 29. January 2016, Dagstuhl Seminar 16041
Reproducibility of Data-Oriented Experiments in e-Science

6. Computational
Workflow

7. 10 Simple Rules for Reproducible
Computational Research
1. For Every Result, Keep Track of How It Was
Produced
2. Avoid Manual Data Manipulation Steps
3. Archive the Exact Versions of All External
Programs Used
4. Version Control All Custom Scripts
5. Record All Intermediate Results, When Possible in
Standardized Formats
6. For Analyses That Include Randomness, Note
Underlying Random Seeds
7. Always Store Raw Data behind Plots
8. Generate Hierarchical Analysis Output, Allowing
Layers of Increasing Detail to Be Inspected
9. Connect Textual Statements to Underlying
Results
10. Provide Public Access to Scripts, Runs, and
Results
Sandve GK, Nekrutenko A,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible
Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285
Record
Everything
Automate
Everything

8. Scientific publications goals:
(i) announce a result and
(ii) convince readers that the result
is correct.
Papers in experimental science
should describe the results and
provide a clear enough protocol to
allow successful repetition and
extension.
Papers in computational science
should describe the results and
provide the complete software
development environment, data
and set of instructions which
generated the figures.
Virtual Witnessing*
*Leviathan and the Air-Pump: Hobbes, Boyle, and the
Experimental Life (1985) Shapin and Schaffer.
Jill Mesirov
David Donoho

9. Datasets, Data collections
Standard operating procedures
Software, algorithms
Configurations,
Tools and apps, services
Codes, code libraries
Workflows, scripts
System software
Infrastructure
Compilers, hardware

10. Analogy: The Lab
data science | data-driven science
1. Philosophy
2. Practice

11. Witnessing “Datascopes”
Input Data
Software
Output Data
Config
Parameters
Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, , ref resources
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment

12. Model Driven Science – can I rerun my model?
Model Sweeps,What are the sensitivities?

13. “Micro” Reproducibility
“Macro” Reproducibility

14. Repeat, Replicate, Robust
Why the differences?
Reproduce
[CTitus Brown]
https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html

15. Computational
analyses
Productivity
Track differences

16. Computational
analyses?
Repeatability:
Sameness
Same result
1 Lab
1 experiment
Reproducibility:
Similarity
Similar result
> 1 Lab
> 1 experiment

17. “an experiment is reproducible until
another laboratory tries to repeat it.”
Alexander Kohn

18. reviewers want additional
work
statistician wants more
runs
analysis needs to be
repeated
post-doc leaves, student
arrives
new data, revised data
updated versions of
algorithms/codes
sample was
contaminated

19. Measuring Information Gain from Reproducibility
Research goal
Method/Alg.
Platform/Exec Env
Data Parameters
Input data
Actors
Information Gain
Implementation/Code
No change
Change
Don’t care
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/
http://www.dagstuhl.de/16041

20. Taxonomy of actions towards
improving reproducibility in
Computer Science.
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/
http://www.dagstuhl.de/16041

21. Practice

22. Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, ref datasets
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment
Input Data
Software
Output Data
Config
Parameters
“Datascope” Entropy -> Preservation
“Replication / Reproducibility Window”

23. Change:
Science, methods, datasets
Questions don’t change, answers do.
Materials unavailable
one offs, streams,
stochastics, sensitivities,
licensing
scale, non-portable data
Change:
Instruments break, labs decay
Active ref datasets and services
Platforms & resources unavailable
supercomputer scale
non-portable software
“Datascope” Entropy -> Preservation
“Replication / Reproducibility Window”
Archived vs Active Environment
Isolated vs Open Distributed Ecosystem

24. T1 T2
Evolving Reference
Knowledge Bases
e.g. UNIPROT

25. Repeat harder than Reproduce?
Repeating the experiment or the set up?
• When the environment is active

26. “Datascope” Entropy -> Preservation
“Replication / Reproducibility Window”
Form?
Function?
Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, ref datasets
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment

27. How? Preserve by Reporting, Reproduce by Reading
ProvenanceTraces, Notebooks, Rich Metadata
Archived Record
Methods
Materials
Instruments
Laboratory

28. How?
Preserve by Reporting, Reproduce by Reading
ProvenanceTraces, Notebooks, Rich Metadata
Archived Record
standards, common metadata
Provenance
Workflows,
Scripts
ELNs
Methods
Materials
Instruments
Laboratory

29. How? Preserve by Maintaining, Repairing,VMs
Reproduce by Running, Emulating, Reconstructing
Active Instrument
Methods
Materials
Instruments
Laboratory

30. How? Preserve by Maintaining, Repairing,VMs
Reproduce by Running, Emulating, Reconstructing
Active Instrument, Byte level
Methods
Materials
Instruments
Laboratory

31. Levels of Computational Reproducibility
Coverage: how
much of an
experiment is
reproducible
OriginalExperimentSimilarExperimentDifferentExperiment
Portability
Depth: how much of an experiment is available
Binaries +
Data
Source Code /
Workflow
+ Data
Binaries +
Data +
Dependencies
Source Code /
Workflow
+ Data +
Dependencies
Virtual Machine
Binaries +
Data +
Dependencies
Virtual Machine
Source Code /
Workflow
+ Data +
Dependencies
Figures +
Data
[Freire, 2014]
Minimum:
data and source
code available
under terms
that permit
inspection and
execution.

32. Repeatable Environments
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
[C.Titus Brown*]
Metadata Objects : Reproducible Reporting, Exchange
Checklist
ProvenanceTracking
Versioning
Dependencies
container

33. provenance
dependencies
steps, features
transparency
portability
robustness
preservation
access description
available
standards
common APIs
licensing
identifiers
standards,
common metadata
change
variation sensitivity
versioning
packaging
So, What is Reproducibility? Being FAIR

34. So, What is Reproducibility? Being FAIR

35. What is Reproducibility?
Why, When, Where, Who for, Who by, How
Special thanks to
• C Titus Brown
• Juliana Freire
• David De Roure
• Stian Soiland-Reyes
• Barend Mons
• Tim Clark
• Daniel Garijo
• Wf4Ever and Research Object teams
• Dagstuhl Seminar 16041
• Force11 http://www.force11.org