Metadata to support reproducibility. What does that mean? What do we need to do? How do we do it?
Will run through the approach that was taken, and some of the vocabs and standards that are being used to do it.
What’s the purpose of publication? Publications intended to present results/positions, along with arguments that reinforce those positions. Reproducability reinforces the validity of our positions. May require us to include much more information than can be included in a paper:in particular, data sets and methods.
Understanding the different roles that are involved in supporting the scientific lifecycle and experimental process.
One of the key issue is that HTTP URIs serve multiple purposes. They are identifiers, but also serve as a mechanism for locating or accessing the content. PIDs, on the other hand tend to involve a resolution or redirection process which guides us to the content. Commonly that resolution ends up on a landing page though – for example DOIs usually resolve to a web page, that may then provide embedded links to further resources.
We can consider ROs as non-information resources (things who’s distinguishing characteristics can’t be conveyed in a message). On resolving the ID for such a thing we get descriptive metadata about it (but not the thing itself). This is a common pattern used for Linked Data resources.
Herbert proposes a bidirectional mapping between PIDs and the HTTP URIs that provide access to the informaiton about them. So we can go from PID to stuff, and from stuff to the PID that it is about.
Approaches like Memento could then be applied to support versioning.
I don’t think there are necessarily any deep problems lurking here – it’s more about the way in which services are set up and establishing convention and practice.
Local folder/file structures – experiences with our astronomy users. Use github for version management. Local tooling produces metadata descriptions.
Example RO in zenodo
Example RO in figshare. Cf Code as a research object.
Work by Dani Garijo of UPM. Web page generated from metadata about papers. RO includes information about the materials provided.
Systems biology bundling. Experiments in mapping between COMBINE archives and ROs.
Metadata for Research Objects
Making Metadata Work, ISKO
London, 23rd June 2014
• Publications are about argumentation: Convince
the reader of the validity of a position
– Reproducible Results System: facilitates
enactment and publication of reproducible
• Results are reinforced by reproducability
– Explicit representation of method.
• Verifiability as a key factor in scientific discovery.
J. Mesirov Accessible Reproducible Research Science 327(5964), p.415-416,
Stodden et. al. Reproducible Research: Addressing the Need for Data and
Code Sharing in Computational Science Computing in Science and
Engineering 12(5), p.8-13, 2010 doi:10.1109/MCSE.2010.113
C.Goble et. al. Accelerating Scientists’ Knowledge Turns
Communications in Computer and Information Science Volume 348,
2013, pp 3-25 doi:10.1007/978-3-642-37186-8_1
» Scientific workflows are at the heart of
› Enable automation of scientific
› Support experimental
› Encourage best practices
» There is then a need to preserve
› Scientific development based on
method reuse and repurpose
› Conservation is key
» Workflow preservation is a
› Representation of complex
› Decay analysis, diagnosis, and
› Social Objects that can be
inspected, reused, repurposed
Preservation of scientific workflows in
Multi-step computational process
Repeatable and comparative
Transparent, precise, citable
Accurate provenance logs
Reusable protocols, know-how,
Can I review /
Can I defend
Can I reuse /
Context: Semantic Web and Linked
• SW: Explicit machine-readable representation of information
• LD: A set of best practices for publishing
and connecting data on the Web
1. Use URIs to name things
2. Use dereferencable HTTP URIs
3. Provide useful content on
lookup using standards
4. Include links to other stuff
• An aggregation object that bundles together experimental
resources that are essential to a computational scientific study
– data used
– results produced in an experiment study;
– (computational) methods employed to
produce and analyse that data;
– people involved in the investigation.
• Plus annotation information that provides additional
information about both the bundle itself and the resources of
• Three principles underlie the approach:
– Referring to resources
(and the aggregation itself)
– Describing the aggregation structure
and its constituent parts
– Associating information with aggregated resources.
• Mechanisms for referring to the resources that are aggregated
within a Research Object
– Web Resources
• ORCID IDs
• HTTP URIs provide both access and identification
• PIDs: Persistent Identifiers (e.g.DOIs) tend to resolve to
human-readable landing pages
– With embedded links to further (possibly machine-
• ROs seen as non-information resources with descriptive
– Standard patterns for Linked Data resources
• Bidirectional mappings between URIs and PIDs
• Versioning through, e.g. Memento
H. Van de Sompel et. al. Persistent Identifiers for Scholarly Assets
and the Web: The Need for an Unambiguous Mapping 9th
International Digital Curation Conference
• Open Archives Initiation Object Reuse and Exchange (OAI
ORE) is a standard for describing aggregations of web
• Uses a Resource Map to describe the aggregated resources
• Proxies allow for statements about the resources within the
– Capturing context and viewpoints
• Several concrete serialisations
– RDF/XML, Atom, RDFa
• Open Annotation specification is a community developed data
model for annotation of web resources
• Developed by the W3C Open Annotation Community Group
• Allows for “stand-off” annotations
– Annotation as a first class citizen
• Developed to fit with Web Architecture
• Essential to the understanding and interpretation of the
scientific outcomes captured by a Research Object as well as
the reuse of the resources within it.
– Provenance information about the experiments, the study
or any other experimental resources
– Evolution information about the Research Object and its
– Descriptions of computational methods
– Dependency information or settings
about the experiment executions
Core & Extensions
• Core model provides support for aggregation and annotation
• Extensions provide additional vocabularies for domain specific
• Workflow Provenance
– Information capturing workflow executions
• Workflow Description
– Abstractions describing Processes, inputs and outputs
• Research Object Evolution
– Information describing change and “snapshots”
preservation and access to preserved ROs as depicted in Figure 6. Optionally, an external repository may
used to support the frequently evolving research objects. The repositories may be housed in a single
multiple physical repositories, and use the same or differing technologies (e.g. a repository may use a dig
preservation solution for the Preservation Repository and specialized digital library solution for the Acce
Repository). Additionally, as the Preservation Repository does not have the same interactive u
requirements as the access and live repositories, it could be implemented with slower (or offline) stora
Figure 6. Conceptual Archival System Storage Architecture.
ROs and OAIS
• ROs as Information Packages in OAIS
• myExperiment as live/access repository
• ROHUB as archival repository
SCAPE: Planning and Watch
• SCAPE project concerned with Digital Preservation.
• Planning and Watch infrastructure to helpmmonitor
the state of a repository and co-ordinate appropriate actions
• Driven by policies.
myExperiment and RODL
Data source monitoring,
Wf4Ever: Monitoring and Watch
• Ideas applied to workflow preservation
• Survey of 92 Taverna workflows from myExperiment
• Volatile Third-Party
• Missing Data
• Missing Execution Environments
• Poor descriptions
Belhajjame et. al. Why workflows break — Understanding
and combating decay in Taverna workflows e-Science 2012
(a) An overview of the decay causes. (b) Workﬂow decay due to third party resources.
Fig. 3. Summary of workﬂow decay causes.
Checklists and Validation
• Checklists widely used to support safety, quality and
• Common in experimental science
– Expressing minimum information
– Supporting “health” monitoring of
• Checklists can be defined in terms of
the RO model and its annotations
– Generic checklist service then
executes against that model and
the given annotations
– Provenance 23
Minim Data Model
pliant” or “ minimally compliant” with a checklist if it satisﬁes all of its MAY,
SHOULD or MUST items respectively.
Fig. 1. An overview of the Minim model schema.
toModel Notation key:
Explicit entity Implicit (super)class
max 1 1
Our Minim data model (see Figure 1) provides 4 core constructs to express
a quality requirement: 24
Zhao et. al. A Checklist-Based Approach for
Quality Assessment of Scientiﬁc Information
3rd In. Workshop on Linked Science, 2013
• A single, transferable object encapsulating the description and
resources of an RO
– Download, transfer, publish
• ZIP-based format (resources) plus a manifest describing
aggregation and annotations (description)
– Unpack with standard tooling
• JSON-LD as a representation for manifest
– Lightweight linked-data format
– Compatible with existing JSON tooling and services
– PROV-O and OAC for annotations
Bundling via git/Zenodo/figshare
• Scientist works with local folder structure.
– Version management via github.
– Local tooling produces metadata description
– Metadata about the aggregation (and its resources)
provided by “hidden folder”
• Zenodo/figshare pull snapshot from github
– Providing DOIs for the aggregrations
– Additional release cycles can prompt new DOIs
• Aggregation objects bundling together experimental resources
that are essential to a computational scientific study or
– Intended to support greater transparency and
• Annotations provide additional information
about the bundle and its contents
– Metadata is key here
• Use of existing standards, vocabularies and
• Nascent tooling to support creation,
management and publication
• All the members of the Wf4Ever team
– iSOCO: Intelligent Software Components S.A., Spain
– University of Manchester, School of Computer Science, Manchester, United
– University of Oxford, Department of Zoology, Oxford, UK
– Poznan Supercomputing and Networking Center. Poznan, Poland
– IAA: Instituto de Astrofísica de Andalucía, Granada, Spain
– Leiden University Medical Centre, Centre for Human and Clinical Genetics,
• Colleagues in Manchester’s Information Management Group
• RO Advisory Board Members