SlideShare a Scribd company logo

What is Reproducibility? The R* brouhaha (and how Research Objects can help)

presented at 1st First International Workshop on Reproducible Open Science @ TPDL, 9 Sept 2016, Hannover, Germany http://repscience2016.research-infrastructures.eu/

What is Reproducibility? The R* brouhaha (and how Research Objects can help)

1 of 34
Download to read offline
What is
Reproducibility?
The R* brouhaha
(and how Research Objects
can help)
Professor Carole Goble
The University of Manchester, UK
Software Sustainability Institute, UK
ELIXIR-UK, FAIRDOMAssociation e.V.
carole.goble@manchester.ac.uk
First International Workshop on Reproducible Open Science @ TPDL, 9 Sept 2016, Hannover, Germany
Acknowledgements
• Dagstuhl Seminar 16041 , January 2016
– http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041
• ATI Symposium Reproducibility, Sustainability and Preservation , April 2016
– https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/
– https://osf.io/bcef5/files/
• CTitus Brown
• Juliana Freire
• David De Roure
• Stian Soiland-Reyes
• Barend Mons
• Tim Clark
• Daniel Garijo
• Norman Morrison
“When I use a word," Humpty Dumpty
said in rather a scornful tone, "it means
just what I choose it to mean - neither
more nor less.”
Carroll, Through the Looking Glass
re-compute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle
redo
robustness
tolerance
verificationcompliancevalidation assurance
remix
Reproducibility of
Reproducibility Research
Computational Science
http://tpeterka.github.io/maui-project/
From:The Future of ScientificWorkflows, Report of DOEWorkshop 2015,
http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd
1. Observational,
experimental
2. Theoretical
3. Simulation
4. Data intensive
BioSTIF
Computational
Science
Ad

Recommended

CLINICAL STUDY REPORT - IN-TEXT TABLES, TABLES FIGURES AND GRAPHS, PATIENT AN...
CLINICAL STUDY REPORT - IN-TEXT TABLES, TABLES FIGURES AND GRAPHS, PATIENT AN...CLINICAL STUDY REPORT - IN-TEXT TABLES, TABLES FIGURES AND GRAPHS, PATIENT AN...
CLINICAL STUDY REPORT - IN-TEXT TABLES, TABLES FIGURES AND GRAPHS, PATIENT AN...Angelo Tinazzi
 
Data Management Plan_Katalyst HLS
Data Management Plan_Katalyst HLSData Management Plan_Katalyst HLS
Data Management Plan_Katalyst HLSKatalyst HLS
 
Data Archiving and Sharing
Data Archiving and SharingData Archiving and Sharing
Data Archiving and SharingC. Tobin Magle
 
Oracle Clinical Overview_Katalyst HLS
Oracle Clinical Overview_Katalyst HLSOracle Clinical Overview_Katalyst HLS
Oracle Clinical Overview_Katalyst HLSKatalyst HLS
 
Cdisc sdtm implementation_process _v1
Cdisc sdtm implementation_process _v1Cdisc sdtm implementation_process _v1
Cdisc sdtm implementation_process _v1ray4hz
 
SDTM - Adverse Events Vs. Clinical Events
SDTM - Adverse Events Vs. Clinical EventsSDTM - Adverse Events Vs. Clinical Events
SDTM - Adverse Events Vs. Clinical EventsVijayaraghava Karpurapu
 
Sap and its importance in clinical research
Sap and its importance in clinical researchSap and its importance in clinical research
Sap and its importance in clinical researchPooja Anothra
 

More Related Content

What's hot

Database Designing in Clinical Data Management
Database Designing in Clinical Data ManagementDatabase Designing in Clinical Data Management
Database Designing in Clinical Data ManagementClinosolIndia
 
Post-lock Data Flow: From CRF to FDA
Post-lock Data Flow: From CRF to FDAPost-lock Data Flow: From CRF to FDA
Post-lock Data Flow: From CRF to FDABrook White, PMP
 
A complex ADaM dataset - three different ways to create one
A complex ADaM dataset - three different ways to create oneA complex ADaM dataset - three different ways to create one
A complex ADaM dataset - three different ways to create oneKevin Lee
 
Oracle Study Setup_Katalyst HLS
Oracle Study Setup_Katalyst HLSOracle Study Setup_Katalyst HLS
Oracle Study Setup_Katalyst HLSKatalyst HLS
 
Introduction to SAS
Introduction to SASIntroduction to SAS
Introduction to SASizahn
 
Risk Based Monitoring in Clinical Trials.
Risk Based Monitoring in Clinical Trials.Risk Based Monitoring in Clinical Trials.
Risk Based Monitoring in Clinical Trials.ClinosolIndia
 
Discover Introduction to REDCap
Discover Introduction to REDCapDiscover Introduction to REDCap
Discover Introduction to REDCapSTARSurg
 
Oracle Clinical Overview_Katalyst HLS
Oracle Clinical Overview_Katalyst HLSOracle Clinical Overview_Katalyst HLS
Oracle Clinical Overview_Katalyst HLSKatalyst HLS
 
Discrepany Management_Katalyst HLS
Discrepany Management_Katalyst HLSDiscrepany Management_Katalyst HLS
Discrepany Management_Katalyst HLSKatalyst HLS
 
Introduction to clinical sas programming
Introduction to clinical sas programmingIntroduction to clinical sas programming
Introduction to clinical sas programmingray4hz
 
Safety_Data_Reconciliation_Katalyst HLS
Safety_Data_Reconciliation_Katalyst HLSSafety_Data_Reconciliation_Katalyst HLS
Safety_Data_Reconciliation_Katalyst HLSKatalyst HLS
 
Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...
Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...
Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...Winston Dodson
 
Elements of Data Documentation
Elements of Data DocumentationElements of Data Documentation
Elements of Data Documentationssri-duke
 
Presentation on CDISC- SDTM guidelines.
Presentation on CDISC- SDTM guidelines.Presentation on CDISC- SDTM guidelines.
Presentation on CDISC- SDTM guidelines.Khushbu Shah
 
Interim Analysis of Clinical Trial Data: Implementation and Practical Advice
Interim Analysis of Clinical Trial Data: Implementation and Practical AdviceInterim Analysis of Clinical Trial Data: Implementation and Practical Advice
Interim Analysis of Clinical Trial Data: Implementation and Practical AdviceNAMSA
 
Informatica PowerCenter
Informatica PowerCenterInformatica PowerCenter
Informatica PowerCenterRamy Mahrous
 

What's hot (20)

Cdm
CdmCdm
Cdm
 
Data Quality Control
Data Quality ControlData Quality Control
Data Quality Control
 
Database Designing in Clinical Data Management
Database Designing in Clinical Data ManagementDatabase Designing in Clinical Data Management
Database Designing in Clinical Data Management
 
Post-lock Data Flow: From CRF to FDA
Post-lock Data Flow: From CRF to FDAPost-lock Data Flow: From CRF to FDA
Post-lock Data Flow: From CRF to FDA
 
A complex ADaM dataset - three different ways to create one
A complex ADaM dataset - three different ways to create oneA complex ADaM dataset - three different ways to create one
A complex ADaM dataset - three different ways to create one
 
Oracle Study Setup_Katalyst HLS
Oracle Study Setup_Katalyst HLSOracle Study Setup_Katalyst HLS
Oracle Study Setup_Katalyst HLS
 
Introduction to SAS
Introduction to SASIntroduction to SAS
Introduction to SAS
 
Risk Based Monitoring in Clinical Trials.
Risk Based Monitoring in Clinical Trials.Risk Based Monitoring in Clinical Trials.
Risk Based Monitoring in Clinical Trials.
 
Discover Introduction to REDCap
Discover Introduction to REDCapDiscover Introduction to REDCap
Discover Introduction to REDCap
 
Oracle Clinical Overview_Katalyst HLS
Oracle Clinical Overview_Katalyst HLSOracle Clinical Overview_Katalyst HLS
Oracle Clinical Overview_Katalyst HLS
 
Discrepany Management_Katalyst HLS
Discrepany Management_Katalyst HLSDiscrepany Management_Katalyst HLS
Discrepany Management_Katalyst HLS
 
Introduction to clinical sas programming
Introduction to clinical sas programmingIntroduction to clinical sas programming
Introduction to clinical sas programming
 
Safety_Data_Reconciliation_Katalyst HLS
Safety_Data_Reconciliation_Katalyst HLSSafety_Data_Reconciliation_Katalyst HLS
Safety_Data_Reconciliation_Katalyst HLS
 
Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...
Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...
Artificial Intelligence - Implications for Business Strategy 2017-11-20. MIT-...
 
Elements of Data Documentation
Elements of Data DocumentationElements of Data Documentation
Elements of Data Documentation
 
Presentation on CDISC- SDTM guidelines.
Presentation on CDISC- SDTM guidelines.Presentation on CDISC- SDTM guidelines.
Presentation on CDISC- SDTM guidelines.
 
Introduction to meta analysis
Introduction to meta analysisIntroduction to meta analysis
Introduction to meta analysis
 
Introduction to SDTM
Introduction to SDTMIntroduction to SDTM
Introduction to SDTM
 
Interim Analysis of Clinical Trial Data: Implementation and Practical Advice
Interim Analysis of Clinical Trial Data: Implementation and Practical AdviceInterim Analysis of Clinical Trial Data: Implementation and Practical Advice
Interim Analysis of Clinical Trial Data: Implementation and Practical Advice
 
Informatica PowerCenter
Informatica PowerCenterInformatica PowerCenter
Informatica PowerCenter
 

Viewers also liked

Research Objects, SEEK and FAIRDOM
Research Objects, SEEK and FAIRDOMResearch Objects, SEEK and FAIRDOM
Research Objects, SEEK and FAIRDOMCarole Goble
 
Reproducibility, Research Objects and Reality, Leiden 2016
Reproducibility, Research Objects and Reality, Leiden 2016Reproducibility, Research Objects and Reality, Leiden 2016
Reproducibility, Research Objects and Reality, Leiden 2016Carole Goble
 
FAIR data and model management for systems biology.
FAIR data and model management for systems biology.FAIR data and model management for systems biology.
FAIR data and model management for systems biology.FAIRDOM
 
Reproducibility of model-based results: standards, infrastructure, and recogn...
Reproducibility of model-based results: standards, infrastructure, and recogn...Reproducibility of model-based results: standards, infrastructure, and recogn...
Reproducibility of model-based results: standards, infrastructure, and recogn...FAIRDOM
 
Capturing the context: one small(ish step for modellers, one giant leap for m...
Capturing the context: one small(ish step for modellers, one giant leap for m...Capturing the context: one small(ish step for modellers, one giant leap for m...
Capturing the context: one small(ish step for modellers, one giant leap for m...FAIRDOM
 
Publishing data and code openly
Publishing data and code openlyPublishing data and code openly
Publishing data and code openlyFAIRDOM
 
Report of the second FAIRDOM foundry
Report of the second FAIRDOM foundryReport of the second FAIRDOM foundry
Report of the second FAIRDOM foundryFAIRDOM
 
Licensing, Citation and Sustainability.
Licensing, Citation and Sustainability.Licensing, Citation and Sustainability.
Licensing, Citation and Sustainability.FAIRDOM
 
The FAIRDOM Commons for Systems Biology
The FAIRDOM Commons for Systems BiologyThe FAIRDOM Commons for Systems Biology
The FAIRDOM Commons for Systems BiologyFAIRDOM
 
Making your data good enough for sharing.
Making your data good enough for sharing.Making your data good enough for sharing.
Making your data good enough for sharing.FAIRDOM
 
FAIR data and model management for systems biology (and SOPs too!)
FAIR data and model management for systems biology (and SOPs too!)FAIR data and model management for systems biology (and SOPs too!)
FAIR data and model management for systems biology (and SOPs too!)FAIRDOM
 
Citing data in research articles: principles, implementation, challenges - an...
Citing data in research articles: principles, implementation, challenges - an...Citing data in research articles: principles, implementation, challenges - an...
Citing data in research articles: principles, implementation, challenges - an...FAIRDOM
 
Improving the management of computational models.
Improving the management of computational models.Improving the management of computational models.
Improving the management of computational models.FAIRDOM
 
Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.FAIRDOM
 
Advances in Scientific Workflow Environments
Advances in Scientific Workflow EnvironmentsAdvances in Scientific Workflow Environments
Advances in Scientific Workflow EnvironmentsCarole Goble
 
FAIR Data, Operations and Model management for Systems Biology and Systems Me...
FAIR Data, Operations and Model management for Systems Biology and Systems Me...FAIR Data, Operations and Model management for Systems Biology and Systems Me...
FAIR Data, Operations and Model management for Systems Biology and Systems Me...Carole Goble
 
Research Objects Tutorial (TPDL)
Research Objects Tutorial (TPDL)Research Objects Tutorial (TPDL)
Research Objects Tutorial (TPDL)dgarijo
 
Research Objects in Scientific Publications
Research Objects in Scientific PublicationsResearch Objects in Scientific Publications
Research Objects in Scientific Publicationsdgarijo
 
Research Shared: researchobject.org
Research Shared: researchobject.orgResearch Shared: researchobject.org
Research Shared: researchobject.orgNorman Morrison
 
ERA CoBioTech Data Management Webinar
ERA CoBioTech Data Management WebinarERA CoBioTech Data Management Webinar
ERA CoBioTech Data Management WebinarFAIRDOM
 

Viewers also liked (20)

Research Objects, SEEK and FAIRDOM
Research Objects, SEEK and FAIRDOMResearch Objects, SEEK and FAIRDOM
Research Objects, SEEK and FAIRDOM
 
Reproducibility, Research Objects and Reality, Leiden 2016
Reproducibility, Research Objects and Reality, Leiden 2016Reproducibility, Research Objects and Reality, Leiden 2016
Reproducibility, Research Objects and Reality, Leiden 2016
 
FAIR data and model management for systems biology.
FAIR data and model management for systems biology.FAIR data and model management for systems biology.
FAIR data and model management for systems biology.
 
Reproducibility of model-based results: standards, infrastructure, and recogn...
Reproducibility of model-based results: standards, infrastructure, and recogn...Reproducibility of model-based results: standards, infrastructure, and recogn...
Reproducibility of model-based results: standards, infrastructure, and recogn...
 
Capturing the context: one small(ish step for modellers, one giant leap for m...
Capturing the context: one small(ish step for modellers, one giant leap for m...Capturing the context: one small(ish step for modellers, one giant leap for m...
Capturing the context: one small(ish step for modellers, one giant leap for m...
 
Publishing data and code openly
Publishing data and code openlyPublishing data and code openly
Publishing data and code openly
 
Report of the second FAIRDOM foundry
Report of the second FAIRDOM foundryReport of the second FAIRDOM foundry
Report of the second FAIRDOM foundry
 
Licensing, Citation and Sustainability.
Licensing, Citation and Sustainability.Licensing, Citation and Sustainability.
Licensing, Citation and Sustainability.
 
The FAIRDOM Commons for Systems Biology
The FAIRDOM Commons for Systems BiologyThe FAIRDOM Commons for Systems Biology
The FAIRDOM Commons for Systems Biology
 
Making your data good enough for sharing.
Making your data good enough for sharing.Making your data good enough for sharing.
Making your data good enough for sharing.
 
FAIR data and model management for systems biology (and SOPs too!)
FAIR data and model management for systems biology (and SOPs too!)FAIR data and model management for systems biology (and SOPs too!)
FAIR data and model management for systems biology (and SOPs too!)
 
Citing data in research articles: principles, implementation, challenges - an...
Citing data in research articles: principles, implementation, challenges - an...Citing data in research articles: principles, implementation, challenges - an...
Citing data in research articles: principles, implementation, challenges - an...
 
Improving the management of computational models.
Improving the management of computational models.Improving the management of computational models.
Improving the management of computational models.
 
Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.
 
Advances in Scientific Workflow Environments
Advances in Scientific Workflow EnvironmentsAdvances in Scientific Workflow Environments
Advances in Scientific Workflow Environments
 
FAIR Data, Operations and Model management for Systems Biology and Systems Me...
FAIR Data, Operations and Model management for Systems Biology and Systems Me...FAIR Data, Operations and Model management for Systems Biology and Systems Me...
FAIR Data, Operations and Model management for Systems Biology and Systems Me...
 
Research Objects Tutorial (TPDL)
Research Objects Tutorial (TPDL)Research Objects Tutorial (TPDL)
Research Objects Tutorial (TPDL)
 
Research Objects in Scientific Publications
Research Objects in Scientific PublicationsResearch Objects in Scientific Publications
Research Objects in Scientific Publications
 
Research Shared: researchobject.org
Research Shared: researchobject.orgResearch Shared: researchobject.org
Research Shared: researchobject.org
 
ERA CoBioTech Data Management Webinar
ERA CoBioTech Data Management WebinarERA CoBioTech Data Management Webinar
ERA CoBioTech Data Management Webinar
 

Similar to What is Reproducibility? The R* brouhaha (and how Research Objects can help)

Results may vary: Collaborations Workshop, Oxford 2014
Results may vary: Collaborations Workshop, Oxford 2014Results may vary: Collaborations Workshop, Oxford 2014
Results may vary: Collaborations Workshop, Oxford 2014Carole Goble
 
Software Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceSoftware Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceCarole Goble
 
Research Objects for FAIRer Science
Research Objects for FAIRer Science Research Objects for FAIRer Science
Research Objects for FAIRer Science Carole Goble
 
Being Reproducible: SSBSS Summer School 2017
Being Reproducible: SSBSS Summer School 2017Being Reproducible: SSBSS Summer School 2017
Being Reproducible: SSBSS Summer School 2017Carole Goble
 
Docker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker, Inc.
 
Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...
Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...
Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...GigaScience, BGI Hong Kong
 
Sharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reportsSharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reportsGaignard Alban
 
Reproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trendsReproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trendsCarole Goble
 
Scientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & SociologyScientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & SociologyNeil Chue Hong
 
Data Sets as Facilitator for new Products and Services for Universities
Data Sets as Facilitator for new Products and Services for UniversitiesData Sets as Facilitator for new Products and Services for Universities
Data Sets as Facilitator for new Products and Services for UniversitiesHendrik Drachsler
 
Data Integration vs Transparency: Tackling the tension
Data Integration vs Transparency: Tackling the tensionData Integration vs Transparency: Tackling the tension
Data Integration vs Transparency: Tackling the tensionPaul Groth
 
Spark Summit Europe: Share and analyse genomic data at scale
Spark Summit Europe: Share and analyse genomic data at scaleSpark Summit Europe: Share and analyse genomic data at scale
Spark Summit Europe: Share and analyse genomic data at scaleAndy Petrella
 
Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)
Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)
Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)Blue BRIDGE
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Ola Spjuth
 
Recapitulation Workshop Cloud Reliability Resilience 2016
Recapitulation Workshop Cloud Reliability Resilience 2016Recapitulation Workshop Cloud Reliability Resilience 2016
Recapitulation Workshop Cloud Reliability Resilience 2016Jorge Cardoso
 
The BlueBRIDGE approach to collaborative research
The BlueBRIDGE approach to collaborative researchThe BlueBRIDGE approach to collaborative research
The BlueBRIDGE approach to collaborative researchBlue BRIDGE
 
Being FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data ScienceBeing FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data ScienceCarole Goble
 

Similar to What is Reproducibility? The R* brouhaha (and how Research Objects can help) (20)

Results may vary: Collaborations Workshop, Oxford 2014
Results may vary: Collaborations Workshop, Oxford 2014Results may vary: Collaborations Workshop, Oxford 2014
Results may vary: Collaborations Workshop, Oxford 2014
 
Reproducible Research and the Cloud
Reproducible Research and the CloudReproducible Research and the Cloud
Reproducible Research and the Cloud
 
Software Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceSoftware Sustainability: Better Software Better Science
Software Sustainability: Better Software Better Science
 
Research Objects for FAIRer Science
Research Objects for FAIRer Science Research Objects for FAIRer Science
Research Objects for FAIRer Science
 
Being Reproducible: SSBSS Summer School 2017
Being Reproducible: SSBSS Summer School 2017Being Reproducible: SSBSS Summer School 2017
Being Reproducible: SSBSS Summer School 2017
 
Docker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce Hoff
 
2015 genome-center
2015 genome-center2015 genome-center
2015 genome-center
 
Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...
Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...
Scott Edmunds: GigaScience - a journal or a database? Lessons learned from th...
 
Sharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reportsSharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reports
 
Reproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trendsReproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trends
 
Scientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & SociologyScientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & Sociology
 
Data Sets as Facilitator for new Products and Services for Universities
Data Sets as Facilitator for new Products and Services for UniversitiesData Sets as Facilitator for new Products and Services for Universities
Data Sets as Facilitator for new Products and Services for Universities
 
Data Integration vs Transparency: Tackling the tension
Data Integration vs Transparency: Tackling the tensionData Integration vs Transparency: Tackling the tension
Data Integration vs Transparency: Tackling the tension
 
Spark Summit Europe: Share and analyse genomic data at scale
Spark Summit Europe: Share and analyse genomic data at scaleSpark Summit Europe: Share and analyse genomic data at scale
Spark Summit Europe: Share and analyse genomic data at scale
 
Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)
Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)
Using e-infrastructures for biodiversity conservation - Gianpaolo Coro (CNR)
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...
 
Recapitulation Workshop Cloud Reliability Resilience 2016
Recapitulation Workshop Cloud Reliability Resilience 2016Recapitulation Workshop Cloud Reliability Resilience 2016
Recapitulation Workshop Cloud Reliability Resilience 2016
 
The Chemtools LaBLog
The Chemtools LaBLogThe Chemtools LaBLog
The Chemtools LaBLog
 
The BlueBRIDGE approach to collaborative research
The BlueBRIDGE approach to collaborative researchThe BlueBRIDGE approach to collaborative research
The BlueBRIDGE approach to collaborative research
 
Being FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data ScienceBeing FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data Science
 

More from Carole Goble

The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...Carole Goble
 
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...Carole Goble
 
Research Software Sustainability takes a Village
Research Software Sustainability takes a VillageResearch Software Sustainability takes a Village
Research Software Sustainability takes a VillageCarole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational WorkflowsCarole Goble
 
Open Research: Manchester leading and learning
Open Research: Manchester leading and learningOpen Research: Manchester leading and learning
Open Research: Manchester leading and learningCarole Goble
 
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational WorkflowsCarole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational WorkflowsCarole Goble
 
EOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryEOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryCarole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational WorkflowsCarole Goble
 
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows Carole Goble
 
FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout Carole Goble
 
FAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practiceFAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practiceCarole Goble
 
RO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research ObjectsRO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research ObjectsCarole Goble
 
The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects Carole Goble
 
How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)Carole Goble
 
What is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can helpWhat is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can helpCarole Goble
 
FAIR History and the Future
FAIR History and the FutureFAIR History and the Future
FAIR History and the FutureCarole Goble
 
ELIXIR UK Node presentation to the ELIXIR Board
ELIXIR UK Node presentation to the ELIXIR BoardELIXIR UK Node presentation to the ELIXIR Board
ELIXIR UK Node presentation to the ELIXIR BoardCarole Goble
 

More from Carole Goble (20)

The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
 
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
 
Research Software Sustainability takes a Village
Research Software Sustainability takes a VillageResearch Software Sustainability takes a Village
Research Software Sustainability takes a Village
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
Open Research: Manchester leading and learning
Open Research: Manchester leading and learningOpen Research: Manchester leading and learning
Open Research: Manchester leading and learning
 
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
EOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryEOSC-Life Workflow Collaboratory
EOSC-Life Workflow Collaboratory
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout
 
FAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practiceFAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practice
 
RO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research ObjectsRO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research Objects
 
The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects
 
How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)
 
What is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can helpWhat is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can help
 
FAIR History and the Future
FAIR History and the FutureFAIR History and the Future
FAIR History and the Future
 
ELIXIR UK Node presentation to the ELIXIR Board
ELIXIR UK Node presentation to the ELIXIR BoardELIXIR UK Node presentation to the ELIXIR Board
ELIXIR UK Node presentation to the ELIXIR Board
 

Recently uploaded

Physics Chapter Three - Electric Fields and Charges
Physics Chapter Three - Electric Fields and ChargesPhysics Chapter Three - Electric Fields and Charges
Physics Chapter Three - Electric Fields and Chargesalinford
 
RNA organic extraction.pdf
RNA organic extraction.pdfRNA organic extraction.pdf
RNA organic extraction.pdfNetHelix
 
Fair and just food systems enabling local midstream businesses? What does it ...
Fair and just food systems enabling local midstream businesses? What does it ...Fair and just food systems enabling local midstream businesses? What does it ...
Fair and just food systems enabling local midstream businesses? What does it ...SIANI
 
An Introduction to Quantum Programming Languages
An Introduction to Quantum Programming LanguagesAn Introduction to Quantum Programming Languages
An Introduction to Quantum Programming LanguagesDavid Yonge-Mallo
 
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptxExploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptxSamrat Tayade
 
ELK ELISA Kits Manufacturer in Singapore
ELK ELISA Kits Manufacturer in SingaporeELK ELISA Kits Manufacturer in Singapore
ELK ELISA Kits Manufacturer in SingaporeGaia Science Pte Ltd
 
dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...
dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...
dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...dkNET
 
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science EducationKavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Educationdsnow9802
 
Quality control test for containers and closure Pratik Ghive
Quality control test for containers and closure Pratik GhiveQuality control test for containers and closure Pratik Ghive
Quality control test for containers and closure Pratik Ghivepratikghive82
 
Duchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptxDuchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptxNavanidhan.M
 
From Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of CannabisFrom Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of CannabisMarkus Roggen
 
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdfCW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdfMollyWinterbottom
 
Geological evidence of extensive N-fixation by volcanic lightning during very...
Geological evidence of extensive N-fixation by volcanic lightning during very...Geological evidence of extensive N-fixation by volcanic lightning during very...
Geological evidence of extensive N-fixation by volcanic lightning during very...Sérgio Sacani
 
A review of volcanic electrification of the atmosphere and volcanic lightning
A review of volcanic electrification of the atmosphere and volcanic lightningA review of volcanic electrification of the atmosphere and volcanic lightning
A review of volcanic electrification of the atmosphere and volcanic lightningSérgio Sacani
 
Construction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdfConstruction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdfLossian Barbosa Bacelar Miranda
 
Carpal tunnel Syndrom Wesam Aljabali -1.pdf
Carpal tunnel Syndrom Wesam Aljabali -1.pdfCarpal tunnel Syndrom Wesam Aljabali -1.pdf
Carpal tunnel Syndrom Wesam Aljabali -1.pdfMsm_mo
 
6129093.ppt plasmodium parasite zoology.
6129093.ppt plasmodium parasite zoology.6129093.ppt plasmodium parasite zoology.
6129093.ppt plasmodium parasite zoology.ssuser4d911a
 
Advancing CAM Assay Image Analysis Using Deep Learning Software
Advancing CAM Assay Image Analysis Using Deep Learning SoftwareAdvancing CAM Assay Image Analysis Using Deep Learning Software
Advancing CAM Assay Image Analysis Using Deep Learning SoftwareKML Vision
 

Recently uploaded (20)

Physics Chapter Three - Electric Fields and Charges
Physics Chapter Three - Electric Fields and ChargesPhysics Chapter Three - Electric Fields and Charges
Physics Chapter Three - Electric Fields and Charges
 
RNA organic extraction.pdf
RNA organic extraction.pdfRNA organic extraction.pdf
RNA organic extraction.pdf
 
Fair and just food systems enabling local midstream businesses? What does it ...
Fair and just food systems enabling local midstream businesses? What does it ...Fair and just food systems enabling local midstream businesses? What does it ...
Fair and just food systems enabling local midstream businesses? What does it ...
 
An Introduction to Quantum Programming Languages
An Introduction to Quantum Programming LanguagesAn Introduction to Quantum Programming Languages
An Introduction to Quantum Programming Languages
 
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptxExploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
 
ELK ELISA Kits Manufacturer in Singapore
ELK ELISA Kits Manufacturer in SingaporeELK ELISA Kits Manufacturer in Singapore
ELK ELISA Kits Manufacturer in Singapore
 
dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...
dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...
dkNET Webinar: An Encyclopedia of the Adipose Tissue Secretome to Identify Me...
 
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science EducationKavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Education
 
Quality control test for containers and closure Pratik Ghive
Quality control test for containers and closure Pratik GhiveQuality control test for containers and closure Pratik Ghive
Quality control test for containers and closure Pratik Ghive
 
Duchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptxDuchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptx
 
VEM 023- LESSON 1.pdf
VEM 023- LESSON 1.pdfVEM 023- LESSON 1.pdf
VEM 023- LESSON 1.pdf
 
From Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of CannabisFrom Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
 
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdfCW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
 
Geological evidence of extensive N-fixation by volcanic lightning during very...
Geological evidence of extensive N-fixation by volcanic lightning during very...Geological evidence of extensive N-fixation by volcanic lightning during very...
Geological evidence of extensive N-fixation by volcanic lightning during very...
 
A review of volcanic electrification of the atmosphere and volcanic lightning
A review of volcanic electrification of the atmosphere and volcanic lightningA review of volcanic electrification of the atmosphere and volcanic lightning
A review of volcanic electrification of the atmosphere and volcanic lightning
 
Construction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdfConstruction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdf
 
Carpal tunnel Syndrom Wesam Aljabali -1.pdf
Carpal tunnel Syndrom Wesam Aljabali -1.pdfCarpal tunnel Syndrom Wesam Aljabali -1.pdf
Carpal tunnel Syndrom Wesam Aljabali -1.pdf
 
6129093.ppt plasmodium parasite zoology.
6129093.ppt plasmodium parasite zoology.6129093.ppt plasmodium parasite zoology.
6129093.ppt plasmodium parasite zoology.
 
ALL the evidence webinar: Appraising and using evidence about community conte...
ALL the evidence webinar: Appraising and using evidence about community conte...ALL the evidence webinar: Appraising and using evidence about community conte...
ALL the evidence webinar: Appraising and using evidence about community conte...
 
Advancing CAM Assay Image Analysis Using Deep Learning Software
Advancing CAM Assay Image Analysis Using Deep Learning SoftwareAdvancing CAM Assay Image Analysis Using Deep Learning Software
Advancing CAM Assay Image Analysis Using Deep Learning Software
 

What is Reproducibility? The R* brouhaha (and how Research Objects can help)

  • 1. What is Reproducibility? The R* brouhaha (and how Research Objects can help) Professor Carole Goble The University of Manchester, UK Software Sustainability Institute, UK ELIXIR-UK, FAIRDOMAssociation e.V. carole.goble@manchester.ac.uk First International Workshop on Reproducible Open Science @ TPDL, 9 Sept 2016, Hannover, Germany
  • 2. Acknowledgements • Dagstuhl Seminar 16041 , January 2016 – http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041 • ATI Symposium Reproducibility, Sustainability and Preservation , April 2016 – https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/ – https://osf.io/bcef5/files/ • CTitus Brown • Juliana Freire • David De Roure • Stian Soiland-Reyes • Barend Mons • Tim Clark • Daniel Garijo • Norman Morrison
  • 3. “When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.” Carroll, Through the Looking Glass re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle redo robustness tolerance verificationcompliancevalidation assurance remix
  • 5. Computational Science http://tpeterka.github.io/maui-project/ From:The Future of ScientificWorkflows, Report of DOEWorkshop 2015, http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd 1. Observational, experimental 2. Theoretical 3. Simulation 4. Data intensive
  • 7. Scientific publications goals: (i) announce a result (ii) convince readers its correct. Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension. Papers in computational science should describe the results and provide the complete software development environment, data and set of instructions which generated the figures. VirtualWitnessing* *Leviathan and theAir-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer. Jill Mesirov David Donoho
  • 8. Datasets, Data collections Standard operating procedures Software, algorithms Configurations, Tools and apps, services Codes, code libraries Workflows, scripts System software Infrastructure Compilers, hardware Systems of Systems Heterogeneous hybrid patchwork of tools and service evolving over time
  • 9. 10 “Simple” Rules for Reproducible Computational Research: RACE 1. For Every Result, Keep Track of How It Was Produced 2. Avoid Manual Data Manipulation Steps 3. Archive the Exact Versions of All External Programs Used 4. Version Control All Custom Scripts 5. Record All Intermediate Results, When Possible in Standardized Formats 6. For Analyses That Include Randomness, Note Underlying Random Seeds 7. Always Store Raw Data behind Plots 8. Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected 9. Connect Textual Statements to Underlying Results 10. Provide Public Access to Scripts, Runs, and Results Sandve GK, Nekrutenko A,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 Record Everything Automate Everything Contain Everything Expose Everything
  • 10. Preparation pain independent testing trials and tribulations [Norman Morrison] replication hostility no funding, time, recognition, place to publish resource intensive access to the complete environment
  • 11. Lab Analogy: Witnessing “Datascopes” Input Data Software Output Data Config Parameters Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, , ref resources Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
  • 13. Repeat, Replicate, Robust [CTitus Brown] https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html Why the differences? Reproduce,Trust
  • 14. “an experiment is reproducible until another laboratory tries to repeat it” Alexander Kohn Repeatability: “Sameness” Same result 1 Lab 1 experiment Reproducibility: “Similarity” Similar result > 1 Lab > 1 experiment Validate Verify
  • 15. Method Reproducibility the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated. Result Reproducibility (aka replicability) obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible What does research reproducibility mean? Steven N. Goodman, Daniele Fanelli, John P. A. Ioannidis ScienceTranslational Medicine 8 (341), 341ps12. [doi: 10.1126/scitranslmed.aaf5027] http://stm.sciencemag.org/content/scitransmed/8/341/341ps12.full.pdf
  • 17. reviewers want additional work statistician wants more runs analysis needs to be repeated post-doc leaves, student arrives new/revised datasets updated/new versions of algorithms/codes sample was contaminated better kit - longer simulations new partners, new projects Personal & Lab Productivity Public Good Reproducibility
  • 18. “Datascope” Lab Analogy Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
  • 19. “Datascope” Lab Analogy Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment Form Function
  • 20. “Datascope” Practicalities Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment Living Dependencies Science, methods, datasets questions stay, answers change breakage, labs decay, services and techniques come and go, new instruments, updated datasets, services, codes, hardware One offs, streams, stochastics, sensitivities, scale, non-portable data black boxes supercomputer access non-portable software licensing restrictions unreliable resources black boxes complexity
  • 21. T1 T2 evolving ref datasets, new simulation codes Environment Archived vs Active Contained vs Distributed Regimented vs Free-for-all Who owns the dependencies? Dependencies -> Manage Black boxes -> Expose Dynamics -> Fixity Reliability
  • 22. Replicate harder than Reproduce? Repeating the experiment or the set up? Container Conundrum Results willVary ReplicabilityWindow All experiments become less replicable over time Prepare to repair
  • 23. Levels of Computational Reproducibility Coverage: how much of an experiment is reproducible OriginalExperimentSimilarExperimentDifferentExperiment Portability Depth: how much of an experiment is available Binaries + Data Source Code / Workflow + Data Binaries + Data + Dependencies Source Code / Workflow + Data + Dependencies Virtual Machine Binaries + Data + Dependencies Virtual Machine Source Code / Workflow + Data + Dependencies Figures + Data [Freire, 2014] Minimum: data and source code available under terms that permit inspection and execution.
  • 24. Measuring Information Gain from Reproducibility Research goal Method/Alg. Platform/Exec Env Data Parameters Input data Actors Information Gain Implementation/Code No change Change Don’t care https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/ http://www.dagstuhl.de/16041
  • 25. How? Preserve by Reporting, Reproduce by Reading Archived Record Description Zoo standards, common metadata
  • 26. How? Preserve by Maintaining, Repairing, Containing Reproduce by Running, Emulating, Reconstructing Active Instrument Byte level Buildability Zoo
  • 27. provenance portability, preservation robustness, versioning access description standards common APIs licensing, identifiers standards, common metadata change variation sensitivity discrepancy handling packaging, containers FAIR RACE Reproducibility Dimensions dependencies steps
  • 28. Research Object Standards-based metadata framework for logically and physically bundling resources with context, http://researchobject.org Bigger on the inside than the outside external referencing
  • 29. Manifest Construction Aggregates link things together Annotations about things & their relationships Container Research Object Standards-based metadata framework for logically and physically bundling resources with context, http://researchobject.org Packaging content & links: Zip files, BagIt, Docker images Catalogues & Commons Platforms: FAIRDOM Manifest Description Dependencies what else is needed Versioning its evolution Checklists what should be there Provenance where it came from Identification locate things regardless where id
  • 30. Systems Biology Commons • Link data, models and SOPs • Standards • Span resources • Snapshot + DOIs • Bundle and export • Logical bundles
  • 31. Belhajjame et al (2015) Using a suite of ontologies for preserving workflow-centric research objects, JWeb Semantics doi:10.1016/j.websem.2015.01.003 application/vnd.wf4ever.robundle+zip Workflow Research Objects exchange, portability and maintenance *https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
  • 32. Asthma Research e-Lab Dataset building and releasing Standardised packing of Systems Biology models European Space Agency RO Library Large dataset management for life science workflows LHC ATLAS experiments Notre Dame U Rostock Encyclopedia of DNA Elements PeptideAtlas
  • 33. Words matter. Reproducibility is not a end. Its a means to an end. Beware reproducibility zealots. 50 Shades of Reproducibility. form vs function A conundrum: big co-operative data-driven science makes reproducibility desirable but also means dependency and change are to be expected. Lab analogy for computational science