SlideShare a Scribd company logo
RARE and FAIR Science:
Reproducibility and
Research Objects
Professor Carole Goble FREng FBCS
The University of Manchester, UK
The Software Sustainability Institute
carole.goble@manchester.ac.uk
Jisc Digital Festival, 9-10 March 2015, ICC Birmingham, UK
KnowledgeTurning, Flow
Barriers to Cure
» Access to scientific
resources
» Coordination and
Collaboration
» Flow of Information
http://fora.tv/2010/04/23/Sage_Commons_Josh_Sommer_Chordoma_Foundation
[Josh Sommer]
[Pettifer, Attwood]
http://getutopia.com
VirtualWitnessing*
Scientific publications:
» announce a result
» convince readers the result is correct
“papers in experimental [and computational
science] should describe the results and
provide a clear enough protocol [algorithm]
to allow successful repetition and extension”
Jill Mesirov, Broad Institute, 2010**
**Accessible Reproducible Research, Science 22January 2010,Vol. 327 no. 5964 pp. 415-416, DOI: 10.1126/science.1179653
*Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer.
Bramhall et al QUALITY OF METHODS REPORTING IN ANIMAL MODELS OF
COLITIS Inflammatory Bowel Diseases, , 2015,
“Only one of the 58 papers reported all essential
criteria on our checklist. Animal age, gender, housing
conditions and mortality/morbidity were all poorly
reported…..”
http://www.nature.com/news/male-researchers-stress-out-rodents-1.15106
“An article about computational science in a
scientific publication is not the scholarship
itself, it is merely advertising of the
scholarship.The actual scholarship is the
complete software development
environment, [the complete data] and the
complete set of instructions which generated
the figures.”
David Donoho, “Wavelab and Reproducible
Research,” 1995
datasets
data collections
standard operating
procedures
software
algorithms
configurations
tools and apps
codes
workflows, scripts
code libraries
services
system software
infrastructure
compilers, hardware
Morin et al Shining Light into Black Boxes
Science 13 April 2012: 336(6078) 159-160
Ince et alThe case for open computer programs, Nature 482,
2012
Of 50 papers randomly chosen from 378 manuscripts in 2011 that use
BurrowsWheeler Aligner for mapping Illumina reads
7studies listed necessary details
26no access to primary data sets, broken links to home websites
31no s/w version, parameters, exact version of genomic reference
sequence
Nekrutenko &Taylor, Next-generation sequencing data interpretation: enhancing, reproducibility and accessibility, NatureGenetics 13 (2012)
Broken software Broken science
» GeoffreyChang, Scripps Institute
» Homemade data-analysis program
inherited from another lab
» Flipped two columns of data,
inverting the electron-density map
used to derive protein structure
» Retract 3 Science papers and 2
papers in other journals
» One paper cited by 364
The structures of MsbA (purple) and
Sav1866 (green) overlap little (left)
until MsbA is inverted (right).
Miller A Scientist's Nightmare: Software Problem Leads to Five Retractions Science 22 December 2006: vol. 314 no. 5807 1856-1857
http://www.software.ac.uk/blog/2014-12-04-its-impossible-conduct-research-without-software-say-7-out-10-uk-researchers
Software making practices
“As a general rule,
researchers do not
test or document their
programs rigorously,
and they rarely
release their codes,
making it almost
impossible to
reproduce and verify
published results
generated by
scientific software”
2000 scientists. J.E. Hannay et al., “How Do Scientists Develop and Use Scientific Software?” Proc. ICSEWorkshop Software Eng. for
Computational Science and Eng., 2009, pp. 1–8.
Tools, Standards
Machine actionable,
Formats, Reporting,
Policies, Practices
Record and
Automate
Everything.
recomputation.org
sciencecodemanifesto.org
republic of science*
regulation of science
institution cores libraries
*Merton’s four norms of scientific behaviour (1942)
public services
Honest Error Science is messy
Inherent
Reinhart/Rogoff Austerity economics
Thomas Herndon
Nature Oct ’12
Zoë Corbyn
Fraud
“I can’t immediately reproduce the research in my own laboratory.
It took an estimated 280 hours for an average user to approximately
reproduce the paper.”
Prof Phil Bourne
Associate Director, NIH Big Data 2 Knowledge Program
When research goes “wrong”
»Tainted resources
»Black boxes
»Poor Reporting
»Unavailable resources /
results: data, software
»Bad maths
»Sins of omission
»Poor training, sloppiness
https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted)
Ioannidis, Why Most Published Research Findings Are False, August 2005
Joppa, et al,TroublingTrends inScientificSoftwareUseSCIENCE 340 May 2013
Scientific method
Social environment
» Impact factor mania
» Pressure to publish
» Broken peer review
» Research never reported
» Disorganisation
» Time pressures
» Prep & curate costs
When research goes “wrong”
https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted)
Nick D Kim, strange-matter.net
Norman Morrison
Do a Replication Study?
No thanks! Not FAIR.
Hard. Resource intensive.
Unrecognised. Trolled.
Just gathering the bits .
Cross-Institutional e-Laboratory
Scattered parts, Subject specific / General resources
Fragmented Landscape
101 Innovations in Scholarly Communication - the Changing ResearchWorkflow, Boseman and Kramer, 2015,
http://figshare.com/articles/101_Innovations_in_Scholarly_Communication_the_Changing_Research_Workflow/1286826
http://myexperiment.org
Research Objects
Compound Investigations, Research Products
Multi-various Products,
Platforms/Resources
Units of exchange, commons, contextual metadata
http://www.researchobject.org
http://www.researchobject.org
First class citizens - data, software, methods
- id, manage, credit, track, profile, focus
A Framework to Bundle and Relate (scattered) resources
Metadata Objects that carry Research Context
Research Objects
• closed <-> open
• local <-> alien
• embed <-> refer
• fixed <-> fluid
• nested
• multi –typed, stewarded,
sited, authored
• span research, researchers,
platforms, time
• cite? resolve? steward?
Goble, De Roure, Bechhofer, Accelerating KnowledgeTurns, I3CK, 2013
means
ends
driver
Research Object packages codes, study,
and metadata to exchange descriptions of
clinical study cohorts, statistical scripts,
data (CKAN for the Farr Commons).
STELAR Asthma e-Lab: StudyTeam for
Early Life Asthma Research
ClinicalCodes.org coded patient cohorts
exchanged with NHS FARSITE system
MRC funded multi-site collaboration to
support safe use of patient and research
data for medical research
STELAR e-Lab
Platform 1
Platform 2
Platform 3
Focus, Pivot and Profile
Profile around methods, workflows, scripts, software, data, figures….
Focus on the figure: F1000Research Living Figures,
versioned articles, in-article data manipulation
R Lawrence Force2015, Vision Award Runner Up http://f1000.com/posters/browse/summary/1097482
Simply data + code
Can change the definition of
a figure, and ultimately the
journal article
Colomb J and Brembs B.
Sub-strains of Drosophila Canton-S differ
markedly in their locomotor behavior [v1;
ref status: indexed, http://f1000r.es/3is]
F1000Research 2014, 3:176
Other labs can replicate the study, or
contribute their data to a meta-
analysis or disease model - figure
automatically updates.
Data updates time-stamped.
New conclusions added via versions.
Jennifer Schopf,Treating Data Like Software: A Case for Production Quality Data,JCDL 2012
Software-like Release paradigm
Not a static document paradigm
Reproduce looks backwards -> Release looks forwards
» Science, methods, data
change -> agile
evolution
» Comparisons , versions,
forks & merges,
dependencies
» Id & Citations
» Interlinked ROs
https://doi.org/10.15490/seek.1.investigation.56
[Snoep, 2015]
https://doi.org/10.15490/seek.1.investigation.56
Personal Data
Local Stores
External
Databases
Articles
Models
Standards
Aggregated Commons Infrastructure
ConsistentComparative Reporting
• Design, protocols, samples,
software, models….
• Just Enough Results Model
• Common and specific elements
http://www.seek4science.org http://www.fair-dom.org http://isatools.org
RO as Instrument, Materials, Method
RO as Instrument, Materials, Method
Input Data
Software
Output Data
Config
Parameters
Drummond, Replicability is not Reproducibility: Nor is it Good Science, online
Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
Public data sets
My algorithm
ROWorkflow as Instrument
BioSTIF
My data set
Public software
recompute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle
redo
What IS reproducibility?
Re: “do again”, “return to original state”
regenerate figure
“show A is true by doing B”
verify but not falsify
[Yong, Nature 485, 2012]
robustness tolerance
verificationcompliance
validation assurance
1. Science Changes. So does the Lab.
BioSTIF
“The questions don’t
change but the
answers do”
Dan Reed
The lab is not fixed
Updated resources
Uncertainty
Zhao, et al .Why workflows break - Understanding and combating decay in
Taverna workflows, 8th Intl Conf e-Science 2012
2. Instruments Break, Labs Decay
materials become unavailable, technicians leave
Reproducibility Window
» Bit rot, Black boxes
» Proprietary Licenses
» Clown services
» Partial replication
» Prepare to Repair
› form or function?
› preserve or sustain?
RO as Instrument, Materials, Method
Input Data
Software
Output Data
Config
Parameters
Methods
(techniques, algorithms,
spec. of the steps)
Materials
(datasets, parameters,
algorithm seeds)
Experiment
Instruments
(codes, services, scripts,
underlying libraries)
Laboratory
(sw and hw infrastructure,
systems software,
integrative platforms)
Setup
Drummond, Replicability is not Reproducibility: Nor is it Good Science, online
Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
Research Environment
submit article
and move on…
publish article
Publication
Environment
Research Environment
publish article
Publication
Environment
submit article
and move on…
[Adapted Freire, 2013]
transparency
dependencies
steps
provenance
portability
robustness
preservation
access
available
description
intelligible
standards
common APIs
licensing
standards
common
metadata
change management
versioning
packaging
Machine
actionable
Machine
actionable
Provenance – the link between doing and reporting
Reproduce by Reading
Archived Record, Retain the Process/Code
The IT Crowd, Series 3, Episode 4
The eLabVirtual Machine* (or Docker Image**)
* a black box though
**docker.com
Reproduce by Running:
Active Instrument
Retain the bits
The IT Crowd, Series 3, Episode 4
The Internet
service
Science as a Service
Integrative frameworks
Open Source
Workflows
Virtual Machines
Portable Packaging
Portability
Transparency
ReproZip
Workflows,makefiles
service
Science as a Service
Integrative frameworks
Open Source
Workflows
Virtual Machines
Portable Packaging
Metadata Objects
the secret is the manifest….
Workflow definition
Data (inputs, outputs)
Parameter configs
Provenance log
Hettne et al Structuring research methods and data with the research object model: genomics
workflows as a case study 2014 http://www.jbiomedsem.com/content/pdf/2041-1480-5-41.pdf
myRDM
http://www.cnri.reston.va.us/papers/OverviewDigitalObjectArchitecture.pdf
Depth and Coverage Profiles
NISO-JATS
NISO-JATS
Depth and Coverage Metadata Profiles
Zhao et. al. 2013
Method Matters
Make reproducible ->
Born
Be smart about
reproducibility
Think Commons not
Repository
Best Practices for ScientificComputing http://arxiv.org/abs/1210.0530
Stodden, Reproducible Research Standard, Intl J Comm Law & Policy, 13 2009
RARE & FAIR KnowledgeTurns with Research Objects
http://doctorwhosite1.weebly.com/sonic-screwdrivers.html
Researcher. Silver bullet tools.
Psychic paper.
Reality Check!
Jorge Cham, www.phdcomics.com
Stealthy not Sneaky
reduce the friction
instrumentation
span RARE and FAIR
OptimisingThe Neylon Equation
Auto-magical end-to-end Instrumentation
https://www.youtube.com/watch?v=QVQwSOX5S08?
ELNs and
Authoring Platforms
Sweave
Credit ≠ Authorship
Citing what?
Research Currencies
Training
56%
Of UK researchers develop their own
research software or scripts
73% Of UK researchers have had no formal
software engineering training
Survey of researchers from 15 RussellGroup universities conducted by SSI between August - October 2014.
406 respondents covering representative range of funders, discipline and seniority.
http://www.rse.ac.uk
BUT……
two years time when the paper is written
reviewers want additional work
statistician wants more runs
analysis may need to be repeated
post-doc leaves, student arrives
new data, revised data
updated versions of algorithms/codes
sample was contaminated
Inspired by Bob Harrison
• Incremental shift for infrastructure
providers.
• Moderate shift for policy makers and
stewards.
• Paradigm shift for researchers and their
institutions.
The Challenge
All the members of the Wf4Ever team
Colleagues in Manchester’s Information
Management Group
http://www.researchobject.org
http://www.wf4ever-project.org
http://www.fair-dom.org
http://seek4science.org
http://rightfield.org.uk
http://www.software.ac.uk
http://www.datafairport.org
http://myexperiment.org
http://www.biovel.euAlanWilliams
Norman Morrison
Stian Soiland-Reyes
Paul Groth
Tim Clark
Juliana Freire
Alejandra Gonzalez-Beltran
Philippe Rocca-Serra
Ian Cottam
Susanna Sansone
Kristian Garza
Barend Mons
Sean Bechhofer
Philip Bourne
Matthew Gamble
Raul Palma
Jun Zhao
Neil Chue Hong
Josh Sommer
Matthias Obst
Jacky Snoep
David Gavaghan
Rebecca Lawrence
Contact…
Professor Carole Goble CBE FREng FBCS
The University of Manchester, UK
carole.goble@manchester.ac.uk
https://sites.google.com/site/caro
legoble
http://www.mygrid.org.uk

More Related Content

What's hot

Reproducible research: First steps.
Reproducible research: First steps. Reproducible research: First steps.
Reproducible research: First steps.
Richard Layton
 
The Research Object Initiative: Frameworks and Use Cases
The Research Object Initiative:Frameworks and Use CasesThe Research Object Initiative:Frameworks and Use Cases
The Research Object Initiative: Frameworks and Use Cases
Carole Goble
 
Being FAIR: FAIR data and model management SSBSS 2017 Summer School
Being FAIR:  FAIR data and model management SSBSS 2017 Summer SchoolBeing FAIR:  FAIR data and model management SSBSS 2017 Summer School
Being FAIR: FAIR data and model management SSBSS 2017 Summer School
Carole Goble
 
Peer Review and Science2.0
Peer Review and Science2.0Peer Review and Science2.0
Peer Review and Science2.0
Jean-Claude Bradley
 
The Rhetoric of Research Objects
The Rhetoric of Research ObjectsThe Rhetoric of Research Objects
The Rhetoric of Research Objects
Carole Goble
 
What is Reproducibility? The R* brouhaha (and how Research Objects can help)
What is Reproducibility? The R* brouhaha (and how Research Objects can help)What is Reproducibility? The R* brouhaha (and how Research Objects can help)
What is Reproducibility? The R* brouhaha (and how Research Objects can help)
Carole Goble
 
Aspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceAspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth Science
Raul Palma
 
Research Objects for FAIRer Science
Research Objects for FAIRer Science Research Objects for FAIRer Science
Research Objects for FAIRer Science
Carole Goble
 
HKU Data Curation MLIM7350 Class 8
HKU Data Curation MLIM7350 Class 8HKU Data Curation MLIM7350 Class 8
HKU Data Curation MLIM7350 Class 8
Scott Edmunds
 
Capturing Context in Scientific Experiments: Towards Computer-Driven Science
Capturing Context in Scientific Experiments: Towards Computer-Driven ScienceCapturing Context in Scientific Experiments: Towards Computer-Driven Science
Capturing Context in Scientific Experiments: Towards Computer-Driven Science
dgarijo
 
Being FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data ScienceBeing FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data Science
Carole Goble
 
Beyond the PDF 2, 2013
Beyond the PDF 2, 2013Beyond the PDF 2, 2013
Beyond the PDF 2, 2013
Alejandra Gonzalez-Beltran
 
NETTAB 2013
NETTAB 2013NETTAB 2013
NETTAB 2012
NETTAB 2012NETTAB 2012
UKON 2014
UKON 2014UKON 2014
The beauty of workflows and models
The beauty of workflows and modelsThe beauty of workflows and models
The beauty of workflows and models
myGrid team
 
CSHALS 2013
CSHALS 2013CSHALS 2013
Reproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An OverviewReproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An Overview
dgarijo
 
ROHub
ROHubROHub
ROHub
Raul Palma
 
Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...
Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...
Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...
GigaScience, BGI Hong Kong
 

What's hot (20)

Reproducible research: First steps.
Reproducible research: First steps. Reproducible research: First steps.
Reproducible research: First steps.
 
The Research Object Initiative: Frameworks and Use Cases
The Research Object Initiative:Frameworks and Use CasesThe Research Object Initiative:Frameworks and Use Cases
The Research Object Initiative: Frameworks and Use Cases
 
Being FAIR: FAIR data and model management SSBSS 2017 Summer School
Being FAIR:  FAIR data and model management SSBSS 2017 Summer SchoolBeing FAIR:  FAIR data and model management SSBSS 2017 Summer School
Being FAIR: FAIR data and model management SSBSS 2017 Summer School
 
Peer Review and Science2.0
Peer Review and Science2.0Peer Review and Science2.0
Peer Review and Science2.0
 
The Rhetoric of Research Objects
The Rhetoric of Research ObjectsThe Rhetoric of Research Objects
The Rhetoric of Research Objects
 
What is Reproducibility? The R* brouhaha (and how Research Objects can help)
What is Reproducibility? The R* brouhaha (and how Research Objects can help)What is Reproducibility? The R* brouhaha (and how Research Objects can help)
What is Reproducibility? The R* brouhaha (and how Research Objects can help)
 
Aspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceAspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth Science
 
Research Objects for FAIRer Science
Research Objects for FAIRer Science Research Objects for FAIRer Science
Research Objects for FAIRer Science
 
HKU Data Curation MLIM7350 Class 8
HKU Data Curation MLIM7350 Class 8HKU Data Curation MLIM7350 Class 8
HKU Data Curation MLIM7350 Class 8
 
Capturing Context in Scientific Experiments: Towards Computer-Driven Science
Capturing Context in Scientific Experiments: Towards Computer-Driven ScienceCapturing Context in Scientific Experiments: Towards Computer-Driven Science
Capturing Context in Scientific Experiments: Towards Computer-Driven Science
 
Being FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data ScienceBeing FAIR: Enabling Reproducible Data Science
Being FAIR: Enabling Reproducible Data Science
 
Beyond the PDF 2, 2013
Beyond the PDF 2, 2013Beyond the PDF 2, 2013
Beyond the PDF 2, 2013
 
NETTAB 2013
NETTAB 2013NETTAB 2013
NETTAB 2013
 
NETTAB 2012
NETTAB 2012NETTAB 2012
NETTAB 2012
 
UKON 2014
UKON 2014UKON 2014
UKON 2014
 
The beauty of workflows and models
The beauty of workflows and modelsThe beauty of workflows and models
The beauty of workflows and models
 
CSHALS 2013
CSHALS 2013CSHALS 2013
CSHALS 2013
 
Reproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An OverviewReproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An Overview
 
ROHub
ROHubROHub
ROHub
 
Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...
Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...
Scott Edmunds talk at AIST: Overcoming the Reproducibility Crisis: and why I ...
 

Viewers also liked

EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...
EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...
EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...
EUROsociAL II
 
29311 m6
29311 m629311 m6
Que es administracion
Que  es administracionQue  es administracion
Que es administracion
yulyestefania1993
 
Computacion Afectiva, Aplicacion Educativa para TVDI - Sandra Baldassarri
Computacion Afectiva, Aplicacion Educativa para TVDI - Sandra BaldassarriComputacion Afectiva, Aplicacion Educativa para TVDI - Sandra Baldassarri
Computacion Afectiva, Aplicacion Educativa para TVDI - Sandra Baldassarri
Red Auti
 
Integrando Xtranormal en el quehacer educativo
Integrando Xtranormal en el quehacer educativoIntegrando Xtranormal en el quehacer educativo
Integrando Xtranormal en el quehacer educativo
rosemaryclass
 
Investor guide
Investor guideInvestor guide
Investor guide
peeyushgaur9785
 
Organizational Road maps for Institutional Online Learning production and del...
Organizational Road maps for Institutional Online Learning production and del...Organizational Road maps for Institutional Online Learning production and del...
Organizational Road maps for Institutional Online Learning production and del...
Dr. Patricio Montesinos
 
Record Label Contacts
Record Label ContactsRecord Label Contacts
Record Label Contacts
Alonzo Holmes
 
Bayside px gearheads_catalog
Bayside px gearheads_catalogBayside px gearheads_catalog
Bayside px gearheads_catalog
Electromate
 
Diseno arquitectonico (Ingenieria software)
Diseno arquitectonico (Ingenieria software)Diseno arquitectonico (Ingenieria software)
Diseno arquitectonico (Ingenieria software)
J.Luis Quinte.S.
 
Criart - Artes Gráficas e Lan House
Criart - Artes Gráficas e Lan HouseCriart - Artes Gráficas e Lan House
Criart - Artes Gráficas e Lan House
guestfb6ca6
 
Jesse Romero's Power Point for an Interview
Jesse Romero's Power Point for an InterviewJesse Romero's Power Point for an Interview
Jesse Romero's Power Point for an Interview
Jesse Romero
 
Catálogo nutrición 2013
Catálogo nutrición 2013Catálogo nutrición 2013
Catálogo nutrición 2013
Bgs Company
 
The OPEN Imperative: Kelly Mooney @ Global Ecommerce Summit
The OPEN Imperative: Kelly Mooney @ Global Ecommerce SummitThe OPEN Imperative: Kelly Mooney @ Global Ecommerce Summit
The OPEN Imperative: Kelly Mooney @ Global Ecommerce Summit
Resource/Ammirati
 
EL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACA
EL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACAEL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACA
EL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACA
Indioê Alan Autovicz
 
SeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link Building
SeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link BuildingSeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link Building
SeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link Building
Heyday ApS
 
Presentation 1 - Proposal for sale 16022016
Presentation 1 - Proposal for sale 16022016Presentation 1 - Proposal for sale 16022016
Presentation 1 - Proposal for sale 16022016
Chetan Nayak
 
Contabilidad
ContabilidadContabilidad
Diapositivas salud ocupacional
Diapositivas salud ocupacionalDiapositivas salud ocupacional
Diapositivas salud ocupacional
Ana Milena
 
Trends and innovation in Fintech
Trends and innovation in FintechTrends and innovation in Fintech
Trends and innovation in Fintech
SPIN Chennai
 

Viewers also liked (20)

EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...
EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...
EUROsociAL II Visita de Intercambio de experiencias Sur-Sur entre AATT sobre ...
 
29311 m6
29311 m629311 m6
29311 m6
 
Que es administracion
Que  es administracionQue  es administracion
Que es administracion
 
Computacion Afectiva, Aplicacion Educativa para TVDI - Sandra Baldassarri
Computacion Afectiva, Aplicacion Educativa para TVDI - Sandra BaldassarriComputacion Afectiva, Aplicacion Educativa para TVDI - Sandra Baldassarri
Computacion Afectiva, Aplicacion Educativa para TVDI - Sandra Baldassarri
 
Integrando Xtranormal en el quehacer educativo
Integrando Xtranormal en el quehacer educativoIntegrando Xtranormal en el quehacer educativo
Integrando Xtranormal en el quehacer educativo
 
Investor guide
Investor guideInvestor guide
Investor guide
 
Organizational Road maps for Institutional Online Learning production and del...
Organizational Road maps for Institutional Online Learning production and del...Organizational Road maps for Institutional Online Learning production and del...
Organizational Road maps for Institutional Online Learning production and del...
 
Record Label Contacts
Record Label ContactsRecord Label Contacts
Record Label Contacts
 
Bayside px gearheads_catalog
Bayside px gearheads_catalogBayside px gearheads_catalog
Bayside px gearheads_catalog
 
Diseno arquitectonico (Ingenieria software)
Diseno arquitectonico (Ingenieria software)Diseno arquitectonico (Ingenieria software)
Diseno arquitectonico (Ingenieria software)
 
Criart - Artes Gráficas e Lan House
Criart - Artes Gráficas e Lan HouseCriart - Artes Gráficas e Lan House
Criart - Artes Gráficas e Lan House
 
Jesse Romero's Power Point for an Interview
Jesse Romero's Power Point for an InterviewJesse Romero's Power Point for an Interview
Jesse Romero's Power Point for an Interview
 
Catálogo nutrición 2013
Catálogo nutrición 2013Catálogo nutrición 2013
Catálogo nutrición 2013
 
The OPEN Imperative: Kelly Mooney @ Global Ecommerce Summit
The OPEN Imperative: Kelly Mooney @ Global Ecommerce SummitThe OPEN Imperative: Kelly Mooney @ Global Ecommerce Summit
The OPEN Imperative: Kelly Mooney @ Global Ecommerce Summit
 
EL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACA
EL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACAEL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACA
EL GRAN BURLADOR DE AMÉRICA: ALVAR NÚÑEZ CABEZA DE VACA
 
SeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link Building
SeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link BuildingSeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link Building
SeoCustomer Hot Tricks & Tips 2011 - SEO, Social Media, SEM, Link Building
 
Presentation 1 - Proposal for sale 16022016
Presentation 1 - Proposal for sale 16022016Presentation 1 - Proposal for sale 16022016
Presentation 1 - Proposal for sale 16022016
 
Contabilidad
ContabilidadContabilidad
Contabilidad
 
Diapositivas salud ocupacional
Diapositivas salud ocupacionalDiapositivas salud ocupacional
Diapositivas salud ocupacional
 
Trends and innovation in Fintech
Trends and innovation in FintechTrends and innovation in Fintech
Trends and innovation in Fintech
 

Similar to RARE and FAIR Science: Reproducibility and Research Objects

Mtsr2015 goble-keynote
Mtsr2015 goble-keynoteMtsr2015 goble-keynote
Mtsr2015 goble-keynote
Carole Goble
 
Software Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceSoftware Sustainability: Better Software Better Science
Software Sustainability: Better Software Better Science
Carole Goble
 
RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015
William Gunn
 
Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.
FAIRDOM
 
Human Genome and Big Data Challenges
Human Genome and Big Data ChallengesHuman Genome and Big Data Challenges
Human Genome and Big Data Challenges
Philip Bourne
 
Minimal viable data reuse
Minimal viable data reuseMinimal viable data reuse
Minimal viable data reuse
voginip
 
Acting as Advocate? Seven steps for libraries in the data decade
Acting as Advocate? Seven steps for libraries in the data decadeActing as Advocate? Seven steps for libraries in the data decade
Acting as Advocate? Seven steps for libraries in the data decade
LizLyon
 
A Big Picture in Research Data Management
A Big Picture in Research Data ManagementA Big Picture in Research Data Management
A Big Picture in Research Data Management
Carole Goble
 
FAIRer Research
FAIRer ResearchFAIRer Research
FAIRer Research
Carole Goble
 
Upgrading the Scholarly Infrastructure
Upgrading the Scholarly InfrastructureUpgrading the Scholarly Infrastructure
Upgrading the Scholarly Infrastructure
Björn Brembs
 
How to Execute A Research Paper
How to Execute A Research PaperHow to Execute A Research Paper
How to Execute A Research Paper
Anita de Waard
 
The Future of Research (Science and Technology)
The Future of Research (Science and Technology)The Future of Research (Science and Technology)
The Future of Research (Science and Technology)
Duncan Hull
 
Talk at OHSU, September 25, 2013
Talk at OHSU, September 25, 2013Talk at OHSU, September 25, 2013
Talk at OHSU, September 25, 2013
Anita de Waard
 
Docker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce Hoff
Docker, Inc.
 
Open reproducible research
Open reproducible researchOpen reproducible research
Open reproducible research
SC CTSI at USC and CHLA
 
Digital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible researchDigital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible research
SC CTSI at USC and CHLA
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4Ever
Jose Enrique Ruiz
 
Reproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trendsReproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trends
Carole Goble
 
Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011
Rudy Potenzone
 
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data PublishingScott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
GigaScience, BGI Hong Kong
 

Similar to RARE and FAIR Science: Reproducibility and Research Objects (20)

Mtsr2015 goble-keynote
Mtsr2015 goble-keynoteMtsr2015 goble-keynote
Mtsr2015 goble-keynote
 
Software Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceSoftware Sustainability: Better Software Better Science
Software Sustainability: Better Software Better Science
 
RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015
 
Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.Reproducible and citable data and models: an introduction.
Reproducible and citable data and models: an introduction.
 
Human Genome and Big Data Challenges
Human Genome and Big Data ChallengesHuman Genome and Big Data Challenges
Human Genome and Big Data Challenges
 
Minimal viable data reuse
Minimal viable data reuseMinimal viable data reuse
Minimal viable data reuse
 
Acting as Advocate? Seven steps for libraries in the data decade
Acting as Advocate? Seven steps for libraries in the data decadeActing as Advocate? Seven steps for libraries in the data decade
Acting as Advocate? Seven steps for libraries in the data decade
 
A Big Picture in Research Data Management
A Big Picture in Research Data ManagementA Big Picture in Research Data Management
A Big Picture in Research Data Management
 
FAIRer Research
FAIRer ResearchFAIRer Research
FAIRer Research
 
Upgrading the Scholarly Infrastructure
Upgrading the Scholarly InfrastructureUpgrading the Scholarly Infrastructure
Upgrading the Scholarly Infrastructure
 
How to Execute A Research Paper
How to Execute A Research PaperHow to Execute A Research Paper
How to Execute A Research Paper
 
The Future of Research (Science and Technology)
The Future of Research (Science and Technology)The Future of Research (Science and Technology)
The Future of Research (Science and Technology)
 
Talk at OHSU, September 25, 2013
Talk at OHSU, September 25, 2013Talk at OHSU, September 25, 2013
Talk at OHSU, September 25, 2013
 
Docker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce Hoff
 
Open reproducible research
Open reproducible researchOpen reproducible research
Open reproducible research
 
Digital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible researchDigital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible research
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4Ever
 
Reproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trendsReproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trends
 
Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011Acs denver dirks potenzone 30 aug2011
Acs denver dirks potenzone 30 aug2011
 
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data PublishingScott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
 

More from Carole Goble

The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
Carole Goble
 
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Carole Goble
 
RO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital ObjectsRO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital Objects
Carole Goble
 
Research Software Sustainability takes a Village
Research Software Sustainability takes a VillageResearch Software Sustainability takes a Village
Research Software Sustainability takes a Village
Carole Goble
 
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
Open Research: Manchester leading and learning
Open Research: Manchester leading and learningOpen Research: Manchester leading and learning
Open Research: Manchester leading and learning
Carole Goble
 
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
EOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryEOSC-Life Workflow Collaboratory
EOSC-Life Workflow Collaboratory
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout
Carole Goble
 
FAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practiceFAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practice
Carole Goble
 
RO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research ObjectsRO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research Objects
Carole Goble
 
The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects
Carole Goble
 
How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)
Carole Goble
 
What is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can helpWhat is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can help
Carole Goble
 

More from Carole Goble (20)

The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
 
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
 
RO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital ObjectsRO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital Objects
 
Research Software Sustainability takes a Village
Research Software Sustainability takes a VillageResearch Software Sustainability takes a Village
Research Software Sustainability takes a Village
 
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
Open Research: Manchester leading and learning
Open Research: Manchester leading and learningOpen Research: Manchester leading and learning
Open Research: Manchester leading and learning
 
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
EOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryEOSC-Life Workflow Collaboratory
EOSC-Life Workflow Collaboratory
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout
 
FAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practiceFAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practice
 
RO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research ObjectsRO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research Objects
 
The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects
 
How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)
 
What is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can helpWhat is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can help
 

Recently uploaded

Chapter 12 - climate change and the energy crisis
Chapter 12 - climate change and the energy crisisChapter 12 - climate change and the energy crisis
Chapter 12 - climate change and the energy crisis
tonzsalvador2222
 
Deep Software Variability and Frictionless Reproducibility
Deep Software Variability and Frictionless ReproducibilityDeep Software Variability and Frictionless Reproducibility
Deep Software Variability and Frictionless Reproducibility
University of Rennes, INSA Rennes, Inria/IRISA, CNRS
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
University of Maribor
 
Equivariant neural networks and representation theory
Equivariant neural networks and representation theoryEquivariant neural networks and representation theory
Equivariant neural networks and representation theory
Daniel Tubbenhauer
 
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Ana Luísa Pinho
 
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
Sérgio Sacani
 
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...
THEMATIC  APPERCEPTION  TEST(TAT) cognitive abilities, creativity, and critic...THEMATIC  APPERCEPTION  TEST(TAT) cognitive abilities, creativity, and critic...
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...
Abdul Wali Khan University Mardan,kP,Pakistan
 
Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.
Aditi Bajpai
 
Oedema_types_causes_pathophysiology.pptx
Oedema_types_causes_pathophysiology.pptxOedema_types_causes_pathophysiology.pptx
Oedema_types_causes_pathophysiology.pptx
muralinath2
 
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
David Osipyan
 
Thornton ESPP slides UK WW Network 4_6_24.pdf
Thornton ESPP slides UK WW Network 4_6_24.pdfThornton ESPP slides UK WW Network 4_6_24.pdf
Thornton ESPP slides UK WW Network 4_6_24.pdf
European Sustainable Phosphorus Platform
 
如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样
如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样
如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样
yqqaatn0
 
Medical Orthopedic PowerPoint Templates.pptx
Medical Orthopedic PowerPoint Templates.pptxMedical Orthopedic PowerPoint Templates.pptx
Medical Orthopedic PowerPoint Templates.pptx
terusbelajar5
 
The debris of the ‘last major merger’ is dynamically young
The debris of the ‘last major merger’ is dynamically youngThe debris of the ‘last major merger’ is dynamically young
The debris of the ‘last major merger’ is dynamically young
Sérgio Sacani
 
What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.
moosaasad1975
 
ESR spectroscopy in liquid food and beverages.pptx
ESR spectroscopy in liquid food and beverages.pptxESR spectroscopy in liquid food and beverages.pptx
ESR spectroscopy in liquid food and beverages.pptx
PRIYANKA PATEL
 
Phenomics assisted breeding in crop improvement
Phenomics assisted breeding in crop improvementPhenomics assisted breeding in crop improvement
Phenomics assisted breeding in crop improvement
IshaGoswami9
 
Nucleic Acid-its structural and functional complexity.
Nucleic Acid-its structural and functional complexity.Nucleic Acid-its structural and functional complexity.
Nucleic Acid-its structural and functional complexity.
Nistarini College, Purulia (W.B) India
 
Shallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptxShallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptx
Gokturk Mehmet Dilci
 
The use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptx
The use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptxThe use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptx
The use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptx
MAGOTI ERNEST
 

Recently uploaded (20)

Chapter 12 - climate change and the energy crisis
Chapter 12 - climate change and the energy crisisChapter 12 - climate change and the energy crisis
Chapter 12 - climate change and the energy crisis
 
Deep Software Variability and Frictionless Reproducibility
Deep Software Variability and Frictionless ReproducibilityDeep Software Variability and Frictionless Reproducibility
Deep Software Variability and Frictionless Reproducibility
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
 
Equivariant neural networks and representation theory
Equivariant neural networks and representation theoryEquivariant neural networks and representation theory
Equivariant neural networks and representation theory
 
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
 
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
 
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...
THEMATIC  APPERCEPTION  TEST(TAT) cognitive abilities, creativity, and critic...THEMATIC  APPERCEPTION  TEST(TAT) cognitive abilities, creativity, and critic...
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...
 
Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.
 
Oedema_types_causes_pathophysiology.pptx
Oedema_types_causes_pathophysiology.pptxOedema_types_causes_pathophysiology.pptx
Oedema_types_causes_pathophysiology.pptx
 
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
 
Thornton ESPP slides UK WW Network 4_6_24.pdf
Thornton ESPP slides UK WW Network 4_6_24.pdfThornton ESPP slides UK WW Network 4_6_24.pdf
Thornton ESPP slides UK WW Network 4_6_24.pdf
 
如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样
如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样
如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样
 
Medical Orthopedic PowerPoint Templates.pptx
Medical Orthopedic PowerPoint Templates.pptxMedical Orthopedic PowerPoint Templates.pptx
Medical Orthopedic PowerPoint Templates.pptx
 
The debris of the ‘last major merger’ is dynamically young
The debris of the ‘last major merger’ is dynamically youngThe debris of the ‘last major merger’ is dynamically young
The debris of the ‘last major merger’ is dynamically young
 
What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.
 
ESR spectroscopy in liquid food and beverages.pptx
ESR spectroscopy in liquid food and beverages.pptxESR spectroscopy in liquid food and beverages.pptx
ESR spectroscopy in liquid food and beverages.pptx
 
Phenomics assisted breeding in crop improvement
Phenomics assisted breeding in crop improvementPhenomics assisted breeding in crop improvement
Phenomics assisted breeding in crop improvement
 
Nucleic Acid-its structural and functional complexity.
Nucleic Acid-its structural and functional complexity.Nucleic Acid-its structural and functional complexity.
Nucleic Acid-its structural and functional complexity.
 
Shallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptxShallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptx
 
The use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptx
The use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptxThe use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptx
The use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptx
 

RARE and FAIR Science: Reproducibility and Research Objects

  • 1. RARE and FAIR Science: Reproducibility and Research Objects Professor Carole Goble FREng FBCS The University of Manchester, UK The Software Sustainability Institute carole.goble@manchester.ac.uk Jisc Digital Festival, 9-10 March 2015, ICC Birmingham, UK
  • 2. KnowledgeTurning, Flow Barriers to Cure » Access to scientific resources » Coordination and Collaboration » Flow of Information http://fora.tv/2010/04/23/Sage_Commons_Josh_Sommer_Chordoma_Foundation [Josh Sommer]
  • 3.
  • 5. VirtualWitnessing* Scientific publications: » announce a result » convince readers the result is correct “papers in experimental [and computational science] should describe the results and provide a clear enough protocol [algorithm] to allow successful repetition and extension” Jill Mesirov, Broad Institute, 2010** **Accessible Reproducible Research, Science 22January 2010,Vol. 327 no. 5964 pp. 415-416, DOI: 10.1126/science.1179653 *Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer.
  • 6. Bramhall et al QUALITY OF METHODS REPORTING IN ANIMAL MODELS OF COLITIS Inflammatory Bowel Diseases, , 2015, “Only one of the 58 papers reported all essential criteria on our checklist. Animal age, gender, housing conditions and mortality/morbidity were all poorly reported…..” http://www.nature.com/news/male-researchers-stress-out-rodents-1.15106
  • 7. “An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship.The actual scholarship is the complete software development environment, [the complete data] and the complete set of instructions which generated the figures.” David Donoho, “Wavelab and Reproducible Research,” 1995 datasets data collections standard operating procedures software algorithms configurations tools and apps codes workflows, scripts code libraries services system software infrastructure compilers, hardware Morin et al Shining Light into Black Boxes Science 13 April 2012: 336(6078) 159-160 Ince et alThe case for open computer programs, Nature 482, 2012
  • 8. Of 50 papers randomly chosen from 378 manuscripts in 2011 that use BurrowsWheeler Aligner for mapping Illumina reads 7studies listed necessary details 26no access to primary data sets, broken links to home websites 31no s/w version, parameters, exact version of genomic reference sequence Nekrutenko &Taylor, Next-generation sequencing data interpretation: enhancing, reproducibility and accessibility, NatureGenetics 13 (2012)
  • 9. Broken software Broken science » GeoffreyChang, Scripps Institute » Homemade data-analysis program inherited from another lab » Flipped two columns of data, inverting the electron-density map used to derive protein structure » Retract 3 Science papers and 2 papers in other journals » One paper cited by 364 The structures of MsbA (purple) and Sav1866 (green) overlap little (left) until MsbA is inverted (right). Miller A Scientist's Nightmare: Software Problem Leads to Five Retractions Science 22 December 2006: vol. 314 no. 5807 1856-1857 http://www.software.ac.uk/blog/2014-12-04-its-impossible-conduct-research-without-software-say-7-out-10-uk-researchers
  • 10. Software making practices “As a general rule, researchers do not test or document their programs rigorously, and they rarely release their codes, making it almost impossible to reproduce and verify published results generated by scientific software” 2000 scientists. J.E. Hannay et al., “How Do Scientists Develop and Use Scientific Software?” Proc. ICSEWorkshop Software Eng. for Computational Science and Eng., 2009, pp. 1–8.
  • 11.
  • 12. Tools, Standards Machine actionable, Formats, Reporting, Policies, Practices
  • 14. republic of science* regulation of science institution cores libraries *Merton’s four norms of scientific behaviour (1942) public services
  • 15. Honest Error Science is messy Inherent Reinhart/Rogoff Austerity economics Thomas Herndon Nature Oct ’12 Zoë Corbyn Fraud
  • 16. “I can’t immediately reproduce the research in my own laboratory. It took an estimated 280 hours for an average user to approximately reproduce the paper.” Prof Phil Bourne Associate Director, NIH Big Data 2 Knowledge Program
  • 17. When research goes “wrong” »Tainted resources »Black boxes »Poor Reporting »Unavailable resources / results: data, software »Bad maths »Sins of omission »Poor training, sloppiness https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted) Ioannidis, Why Most Published Research Findings Are False, August 2005 Joppa, et al,TroublingTrends inScientificSoftwareUseSCIENCE 340 May 2013 Scientific method
  • 18.
  • 19. Social environment » Impact factor mania » Pressure to publish » Broken peer review » Research never reported » Disorganisation » Time pressures » Prep & curate costs When research goes “wrong” https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted) Nick D Kim, strange-matter.net Norman Morrison Do a Replication Study? No thanks! Not FAIR. Hard. Resource intensive. Unrecognised. Trolled. Just gathering the bits .
  • 20. Cross-Institutional e-Laboratory Scattered parts, Subject specific / General resources Fragmented Landscape 101 Innovations in Scholarly Communication - the Changing ResearchWorkflow, Boseman and Kramer, 2015, http://figshare.com/articles/101_Innovations_in_Scholarly_Communication_the_Changing_Research_Workflow/1286826
  • 21.
  • 23.
  • 24. Research Objects Compound Investigations, Research Products Multi-various Products, Platforms/Resources Units of exchange, commons, contextual metadata http://www.researchobject.org
  • 25. http://www.researchobject.org First class citizens - data, software, methods - id, manage, credit, track, profile, focus A Framework to Bundle and Relate (scattered) resources Metadata Objects that carry Research Context Research Objects
  • 26. • closed <-> open • local <-> alien • embed <-> refer • fixed <-> fluid • nested • multi –typed, stewarded, sited, authored • span research, researchers, platforms, time • cite? resolve? steward?
  • 27. Goble, De Roure, Bechhofer, Accelerating KnowledgeTurns, I3CK, 2013 means ends driver
  • 28. Research Object packages codes, study, and metadata to exchange descriptions of clinical study cohorts, statistical scripts, data (CKAN for the Farr Commons). STELAR Asthma e-Lab: StudyTeam for Early Life Asthma Research ClinicalCodes.org coded patient cohorts exchanged with NHS FARSITE system MRC funded multi-site collaboration to support safe use of patient and research data for medical research STELAR e-Lab Platform 1 Platform 2 Platform 3
  • 29. Focus, Pivot and Profile Profile around methods, workflows, scripts, software, data, figures….
  • 30. Focus on the figure: F1000Research Living Figures, versioned articles, in-article data manipulation R Lawrence Force2015, Vision Award Runner Up http://f1000.com/posters/browse/summary/1097482 Simply data + code Can change the definition of a figure, and ultimately the journal article Colomb J and Brembs B. Sub-strains of Drosophila Canton-S differ markedly in their locomotor behavior [v1; ref status: indexed, http://f1000r.es/3is] F1000Research 2014, 3:176 Other labs can replicate the study, or contribute their data to a meta- analysis or disease model - figure automatically updates. Data updates time-stamped. New conclusions added via versions.
  • 31. Jennifer Schopf,Treating Data Like Software: A Case for Production Quality Data,JCDL 2012 Software-like Release paradigm Not a static document paradigm Reproduce looks backwards -> Release looks forwards » Science, methods, data change -> agile evolution » Comparisons , versions, forks & merges, dependencies » Id & Citations » Interlinked ROs
  • 35. Aggregated Commons Infrastructure ConsistentComparative Reporting • Design, protocols, samples, software, models…. • Just Enough Results Model • Common and specific elements http://www.seek4science.org http://www.fair-dom.org http://isatools.org
  • 36.
  • 37. RO as Instrument, Materials, Method
  • 38. RO as Instrument, Materials, Method Input Data Software Output Data Config Parameters Drummond, Replicability is not Reproducibility: Nor is it Good Science, online Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
  • 39. Public data sets My algorithm ROWorkflow as Instrument BioSTIF My data set Public software
  • 40. recompute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle redo What IS reproducibility? Re: “do again”, “return to original state” regenerate figure “show A is true by doing B” verify but not falsify [Yong, Nature 485, 2012] robustness tolerance verificationcompliance validation assurance
  • 41. 1. Science Changes. So does the Lab. BioSTIF “The questions don’t change but the answers do” Dan Reed The lab is not fixed Updated resources Uncertainty
  • 42. Zhao, et al .Why workflows break - Understanding and combating decay in Taverna workflows, 8th Intl Conf e-Science 2012 2. Instruments Break, Labs Decay materials become unavailable, technicians leave Reproducibility Window » Bit rot, Black boxes » Proprietary Licenses » Clown services » Partial replication » Prepare to Repair › form or function? › preserve or sustain?
  • 43. RO as Instrument, Materials, Method Input Data Software Output Data Config Parameters Methods (techniques, algorithms, spec. of the steps) Materials (datasets, parameters, algorithm seeds) Experiment Instruments (codes, services, scripts, underlying libraries) Laboratory (sw and hw infrastructure, systems software, integrative platforms) Setup Drummond, Replicability is not Reproducibility: Nor is it Good Science, online Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
  • 44. Research Environment submit article and move on… publish article Publication Environment
  • 46. [Adapted Freire, 2013] transparency dependencies steps provenance portability robustness preservation access available description intelligible standards common APIs licensing standards common metadata change management versioning packaging Machine actionable Machine actionable
  • 47. Provenance – the link between doing and reporting
  • 48. Reproduce by Reading Archived Record, Retain the Process/Code
  • 49. The IT Crowd, Series 3, Episode 4 The eLabVirtual Machine* (or Docker Image**) * a black box though **docker.com Reproduce by Running: Active Instrument Retain the bits
  • 50. The IT Crowd, Series 3, Episode 4 The Internet
  • 51. service Science as a Service Integrative frameworks Open Source Workflows Virtual Machines Portable Packaging Portability Transparency
  • 52. ReproZip Workflows,makefiles service Science as a Service Integrative frameworks Open Source Workflows Virtual Machines Portable Packaging
  • 53. Metadata Objects the secret is the manifest….
  • 54. Workflow definition Data (inputs, outputs) Parameter configs Provenance log Hettne et al Structuring research methods and data with the research object model: genomics workflows as a case study 2014 http://www.jbiomedsem.com/content/pdf/2041-1480-5-41.pdf myRDM
  • 56. NISO-JATS Depth and Coverage Metadata Profiles Zhao et. al. 2013
  • 57. Method Matters Make reproducible -> Born Be smart about reproducibility Think Commons not Repository Best Practices for ScientificComputing http://arxiv.org/abs/1210.0530 Stodden, Reproducible Research Standard, Intl J Comm Law & Policy, 13 2009 RARE & FAIR KnowledgeTurns with Research Objects
  • 59. Reality Check! Jorge Cham, www.phdcomics.com
  • 60. Stealthy not Sneaky reduce the friction instrumentation span RARE and FAIR OptimisingThe Neylon Equation
  • 62. Credit ≠ Authorship Citing what? Research Currencies
  • 63. Training 56% Of UK researchers develop their own research software or scripts 73% Of UK researchers have had no formal software engineering training Survey of researchers from 15 RussellGroup universities conducted by SSI between August - October 2014. 406 respondents covering representative range of funders, discipline and seniority.
  • 65. BUT…… two years time when the paper is written reviewers want additional work statistician wants more runs analysis may need to be repeated post-doc leaves, student arrives new data, revised data updated versions of algorithms/codes sample was contaminated
  • 66. Inspired by Bob Harrison • Incremental shift for infrastructure providers. • Moderate shift for policy makers and stewards. • Paradigm shift for researchers and their institutions. The Challenge
  • 67. All the members of the Wf4Ever team Colleagues in Manchester’s Information Management Group http://www.researchobject.org http://www.wf4ever-project.org http://www.fair-dom.org http://seek4science.org http://rightfield.org.uk http://www.software.ac.uk http://www.datafairport.org http://myexperiment.org http://www.biovel.euAlanWilliams Norman Morrison Stian Soiland-Reyes Paul Groth Tim Clark Juliana Freire Alejandra Gonzalez-Beltran Philippe Rocca-Serra Ian Cottam Susanna Sansone Kristian Garza Barend Mons Sean Bechhofer Philip Bourne Matthew Gamble Raul Palma Jun Zhao Neil Chue Hong Josh Sommer Matthias Obst Jacky Snoep David Gavaghan Rebecca Lawrence
  • 68. Contact… Professor Carole Goble CBE FREng FBCS The University of Manchester, UK carole.goble@manchester.ac.uk https://sites.google.com/site/caro legoble http://www.mygrid.org.uk