Reproducibility is our gold standard, but what happens when data are analysed in a safe haven? Restrictions, which can be a high bar, on those accessing data mean that reproducers also need to meet any requirements. The UK Statistics Authority Safe Researcher type model can help create a trusted network of people with the training and skills to access data to check and rerun code. Reproducibility certification, like cascad, can help provide a robust checking process for controlled data.

1. Reproducibility in Safe Havens
Louise Corti
Service Director Data Publishing and Access
UK Data Service
UKRN From data to metadata: Ensuring reproducibility in
biomedical research
Online
22 October 2020
Copyright © 2020 UK Data Service. Created by University of Essex

2. Safe haven data
Sensitive survey and administrative data
• Potential risk of disclosure
• Low levels of geography
Data access/ availability
• Restricted under the 5 SAFES framework
(ONS)
• Access only via Secure Lab (safe haven)
• High access bar
• Process for reproduction not set up by
journals

3. The UK Data Service
• Social science data service, funded by the UKRI ESRC
• Curates and provides access to data for research and
teaching
• Trusted Digital Repository (TDR), accredited to ISO27001 Info
Security Management standard, Digital Economy Act
Processor application in process
• Work closely with research funders and key data producers
/institutions: research centres; UK NSIs, govt departments,
British Library etc
• Around 8000 data collections, from open to secure

5. Accredited Researcher status
Social /economic sciences: Accredited Researcher concept
 Managed by the UK Statistics Authority
 Secure data owner requirement e.g. ONS, ESRC
 Based on experience/expertise and responsible attitude
 Understanding of disclosure risk and producing Safe Outputs
 Attend a one-day training course and pass a test
Medical research: focus on ‘bona fide researcher’ concept*
• the professional expertise and experience to conduct bona fide
research
• a formal relationship with a bona fide research organisation that
requires compliance with appropriate research governance and
management systems
* https://mrc.ukri.org/documents/pdf/data-sharing-from-population-and-patient-studies/

6. Reproducibility factors in safe havens
 FAIR data: Use of DOIs for all UKDS data
 FAIR data: Open data formats; high quality documentation
 Mandated use of DOIs in all checked outputs for publication
 Code can be taken out but is reviewed for risk
✘ No Lab rules for coding
✘ No mandated use of R Markdown /Jupyter Notebook
✘ No mandate from data owners to be research reproducible
✘ Weaker provenance chain for ‘research-ready’ admin data
(creation, cleaning and versioning)

7. cascad = Certification Agency for Scientific Code and Data
Certifying reproducibility (Safe havens)
cascad reproducibility certification attests that the numerical
results reported in a scientific article can be reproduced from a
set of numerical resources (code and data) provided by the
authors
• cascad.tech
• Non-profit academic initiative
• Founded by researchers
• CNRS, HEC Paris, U. Orléans
7Thanks to Christophe Perignon for cascad images

8. Step 1: ‘Submission’ of paper and digital resources (code + data)
Step 2: Compliance check by cascad staff:
• Gains access to secure environment
• Verifies whether the submitted resources comply with the guidelines
• Checks presentation/structure of code /data, aiming for
interpretable and reusable resources
• Aligns with guidance from AEA and Social Science Data Editors
• Non-conformity = no certificate
Certification process
8

9. Step 3: Review process
cascad assigns a reproducibility reviewer who:
• Gains access to the safe data environment
• Runs the code and compares the outputs from the code to
the numerical results of the paper
• Create an execution report: data description, code
description, replication steps, and findings
• Proposes a ‘reproducibility rating’
• RRR to DD (Perfect to Serious Discrepancies)
9
Certification process

10. Step 4: Final certification
• The reproducibility editor assigns the final rating
based on the results reported in the execution report
• The reproducibility certificate and the execution
report are sent to the author
• Cascad staff uploads the certificate on an open
access repository (e.g. Zenodo) with a DOI
• Full execution report available in safe haven
10
Certification process

11. UK Data Service Pilot
• 4 economists /sociologists Secure Lab users invited to
Be Reproduced to go onto try for the cascad certificate:
3 month deadline
• None would submit current code for rerunning against
outcomes until ’tidied up’
• 1 agreed to try to make deadline
• Started to rework code, but too much of a challenge, they could
not reproduce themselves
• Outcome = must start from scratch
• Approaches discussed at UKDS/ONS Workshop in Feb
2020 called #LoveYourCode

12. Longer-term goals
 Cascad –type process for safe havens
 Build reproducible processes into data production /analysis
• Encourage code tracking tools
• Replacement of older unreproducible models with new
documented code
• Mandate cleaning, value-added, final code available in safe
haven to enable reproduction and reuse
• Ensure checking process is established, costed (dedicated staff)
• Ensure data have DOIs and DOIs are used
 Training on creating ‘good code’ (e.g. Turing Way)
 Use of Reproducibility Certificates as incentives
 Early adopters – bare all! How I became reproducible

13. Questions
Louise Corti
UK Data Service
corti@essex.ac.uk