Making computations reproducible
1 / 31
Scientiﬁc computations tend to involve a number of experiments
under different conditions.
It is important to manage computational experiments so that their
results are reproducible.
In this talk we introduce 3 rules to make computations reproducible.
2 / 31
3 / 31
A lab notebook is indispensable for experimental research in natural
science. One of its role is to make experiments reproducible.
Why not for computational research?
......Lack of reproducibility means lack of reliability.
5 / 31
Common problems in computational experiments:
I confused which results is got under which condition.
I overwrote previous results without intent.
I used inconsistent data to get invalid results.
Not a few problems are caused due an inappropriate management
6 / 31
To archive all results of each experiment with
all information required to reproduce them
so that we can retrieve and restore easily
in a systematic and costless way.
7 / 31
What is introduced in this talk is not a established methodology,
but a collection of ﬁeld techniques. Same with wording.
In this talk, we will not deal with
documentation or test
publishing of a paper
release of OSS
8 / 31
We distinguish the following elements which affect reproducibility of
Algorithm an algorithm coded into a program
implemented by yourself, calling external library, ...
Data input and output data, intermediate data to reuse
Environment software and hardware environment
external library, server conﬁguration, platform, ...
10 / 31
Give an Identiﬁer to each element and archive them.
Record a machine-readable Recipe
with a human-readable comments.
Make every manipulation Mechanized.
11 / 31
......Give an Identiﬁer to each element and archive them.
use version control system
give a name to distinguish data kind
give a version to distinguish concrete content
ﬁnd information of platform
ﬁnd a version (optionally build parameters) of a library
Keep in mind to track all elements during the whole process:
every code under version control
no data without an identiﬁer
no temporary environment
12 / 31
Record a machine-readable Recipe
with a human-readable comments.
A recipe should include all information
required to reproduce the results of an experiment
(other than contents of Algorithm, Data and Environment
stored in other place.)
A recipe should be machine-readable to re-conduct the experiment.
A recipe should include a human-readable comment
on purpose and/or meanings of the experiment.
A recipe should be generated automatically by tracking
13 / 31
Typically a recipe include the following information:
in which order
which data is processed
by which algorithm
under which environment
with which Parameter
Typically a recipe consists of the followings:
a script ﬁle to run the whole process
a conﬁguration ﬁle which speciﬁes parameters and identiﬁers
a text ﬁle of comments
14 / 31
......Make every manipulation Mechanized.
Run the whole process of an experiment by a single operation.
No manual manipulation of data.
No manual compilation of source codes.
Automated provision of an environment.
15 / 31
complement: Tentative experiment
Too large archive detracts substantive signiﬁcant of reproducibility.
For tentative experiments with ephemeral results,
it is not necessarily required to record.
test of codes
trial on tiny data
If there is a possibility to get a result which might be used, referred
or looked up afterward, then it should be recorded.
16 / 31
complement: Reuse of intermediate data
In order to reuse intermediate data, utilize an identiﬁer.
Explicitly specify intermediate data to reuse by an identiﬁer.
Automatically detect available intermediate data
based on dependency.
17 / 31
Use a version control system to manage source codes
such as Git and Mercurial.
It is easy to record a revision and uncommitted changes
at each experiment.
(Learn inside of VCS if you need more ﬂexible management.)
19 / 31
Give appropriate names to directories and ﬁles,
then a resolved absolute path can be used as an identiﬁer.
If no meaningful word is thought up, use time-stamp or hash.
DB or other API
A pair of URI and query of which results are constant
can be used as an identiﬁer.
If API behaves randomly, keep the results at hand (w/time-stamp).
20 / 31
Use PyPa tools (virtualenv, setuptools and pip) or Conda/enstaller.
It is an alternative to utilize CDE.
Use platform, a standard library of Python
Use Ansible or other conﬁguration management tool,
and Vagrant or other provisioning tool.
21 / 31
A tool for developing, building and managing software stacks.
An software stack is described by YAML.
We can create, copy, move and remove software stacks.
$ git checkout stack.yml
$ hit build stack.yaml
22 / 31
Recipe: conﬁguration ﬁle
A conﬁguration in recipe should be of a machine-readable format.
Use ConﬁgParser, PyYAML or json module
to read/write parameters in INI, YAML or JSON format.
A receipt should include the followings:
command line argument
23 / 31
Recipe: script ﬁle
A script in recipe should run the whole process
by a single operation.
There are several alternatives to realize such a script:
utilize a build tool (such as Autotools, Scons, and maf)
utilize a job-ﬂow tool (such as Ruffus, Luigi)
write a small script by hand (e.g. run.py)
24 / 31
“maf is a waf extension for writing computational experiments.”
Conduct computational experiments as build processes.
Focus on machine learning:
run programs with each conﬁguration
aggregate and visualize their results
25 / 31
Recipe: automatic generation
Do it yourself, or use Sumatra.
“Sumatra: automated tracking of scientiﬁc computations”
recording information about experiments, linking to data ﬁles
command line & web interface
integration with LATEX/Sphinx
$ smt run --executable=python --main=main.py
$ smt comment "..."
$ smt info
$ smt repeat
26 / 31
We have introduced 3 rules to manage computational experiments
so that their results are reproducible.
However, our method is just a makeshift patchwork of ﬁeld
We need a tool to manage experiments
in more integrated, systematic and sophisticated manner
for reproducible computations.
28 / 31
 G. K. Sandve, A. Nekrutenko, J. Taylor, E. Hovig, “Ten Simple Rules
for Reproducible Computational Research,” PLoS Comput. Biol.
9(10): e1003285 (2013). doi:10.1371/journal.pcbi.1003285
 V. Stodden, F. Leisch, R. Peng, “Implementing Reproducible
Research,” Open Science Framework (2014). osf.io/s9tya
30 / 31
back to outline
Revision: f2b0e97 (2014-08-03)
31 / 31