SlideShare a Scribd company logo
Reproducibility
and Scientific
Research
Professor Carole Goble CBE FREng FBCS
The University of Manchester, UK
carole.goble@manchester.ac.uk
Open Data Manchester, 27th January 2015
icanhascheezburger.com
why, what, where, when, who, how
Scientific publications have at least
two goals:
(i) to announce a result and
(ii) to convince readers that the
result is correct
…..
papers in experimental [and
computational science] should
describe the results and provide a
clear enough protocol [or
algorithm] to allow successful
repetition and extension
Jill Mesirov
Accessible Reproducible Research
Science 22 January 2010:
Vol. 327 no. 5964 pp. 415-416
DOI: 10.1126/science.1179653
Virtual Witnessing / Minute Taking
[Pettifer, Attwood]
http://getutopia.com
Why smart parents often
tend to have smart kids
“an experiment is reproducible until
another laboratory tries to repeat it.”
Alexander Kohn
design
cherry picking data, random seed
reporting, non-independent bias, poor
positive and negative controls, dodgy
normalisation, arbitrary cut-offs,
premature data triage, un-validated
materials, improper statistical analysis,
poor statistical power, stop when “get to
the right answer”, software
misconfigurations misapplied black box
software
reporting
John P. A. Ioannidis Why Most Published Research FindingsAre False, August 30, 2005,
DOI: 10.1371/journal.pmed.0020124
incomplete reporting of software configurations, parameters & resource
versions, missed steps, missing data, vague methods, missing software
Joppa, et al,TroublingTrends in Scientific Software Use SCIENCE 340 May 2013
Empirical
Statistical
Computational
V. Stodden, IMS Bulletin (2013)
Transparency / Availability Gap
1. Ioannidis et al., 2009. Repeatability of published microarray gene expression analyses. Nature Genetics 41: 14
2. Science publishing: The trouble with retractions http://www.nature.com/news/2011/111005/full/478026a.html
3. Bjorn Brembs: Open Access and the looming crisis in science https://theconversation.com/open-access-and-the-looming-crisis-in-science-14950
Out of 18 microarray papers, results
from 10 could not be reproduced
Researcher survey, 1202 respondents
(PARSE.insight 2010)
Sustainability
WHERE?
[Hylke Koers]
Broken software, broken science
• GeoffreyChang, Scripps Institute
• Homemade data-analysis program
inherited from another lab
• Flipped two columns of data,
inverting the electron-density map
used to derive protein structure
• Retract 3 Science papers and 2 papers
in other journals
• One paper cited by 364The structures of MsbA (purple) and
Sav1866 (green) overlap little (left)
until MsbA is inverted (right).
Miller A Scientist's Nightmare: Software Problem Leads to Five Retractions Science 22 December 2006:
vol. 314 no. 5807 1856-1857
http://www.software.ac.uk/blog/2014-12-04-its-impossible-conduct-research-without-software-say-7-out-10-uk-researchers
“An article about computational science in a
scientific publication is not the scholarship
itself, it is merely advertising of the
scholarship.The actual scholarship is the
complete software development
environment, [the complete data] and the
complete set of instructions which generated
the figures.”
David Donoho, “Wavelab and Reproducible
Research,” 1995
Morin et al Shining Light into Black Boxes Science 13 April 2012: 336(6078) 159-160
Ince et alThe case for open computer programs, Nature 482, 2012
algorithms
configurations
tools and apps
codes
workflows
scripts
code libraries
third party services,
system software
infrastructure,
compilers
hardware
Self-contained codes??
WHY? 12+3 reasons
research goes “wrong”
1. Pressure to publish
2. Impact factor mania
3. Tainted resources
4. Bad maths
5. Sins of omission
6. Science is messy
7. Broken peer review
8. Some scientists don’t share
9. Research never reported
10. Poor training -> sloppiness
11. Honest error
12. Fraud
13. Disorganisation & time pressures
14. Cost to prepare and curate materials
15. Inherently “unreplicable ” (one-off data, specialist kit, stochastic)
https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted)
• replication hostility
• resource intensive
• no funding, time,
recognition, place to
publish
• the complete
environment?
Its HARD to
Prepare and Independently Test
[Norman Morrison]
Value People. Data. Method. Software.
re-compute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle
conceptual replication “show
A is true by doing B rather
than doing A again”
verify but not falsify
[Yong, Nature 485, 2012]
regenerate figure
redo
WHAT is reproducibility?
this is a heated topic of debate
robustness tolerance
verificationcompliance
validation assurance
Can I repeat
my method?
publish article
DEFEND
*Adapted from Mesirov, J. Accessible Reproducible Research Science 327(5964), 415-416 (2010)
WHEN? same experiment, set up, lab
submit article
(and move on…)
Can I replicate
your method?
CERTIFY
(a window before decay sets in … )
same experiment,
set up,
independent lab
Can I reproducemy
results using your method or
your results using my method?
COMPARE
variations on experiment, set up, lab
Can I reuseyour
results / method in my
research ?
TRANSFER
different experiment
WHO? scientific ego-system & access
trust, reciprocity, and competition
blame
scooping
no credit / credit drift
misinterpretation
scrutiny trolling
cost of preparation
support distraction
dependents on old news
loss of dowry
loss of special sauce
hugging
flirting
voyerism
cautionary creeping
Tenopir, et al. Data Sharing by Scientists: Practices and Perceptions. PLoS ONE 6(6) 2012
Borgman The conundrum of sharing research data, JASIST 2012
John P. A. Ioannidis How to Make More Published ResearchTrue, October 21, 2014 DOI: 10.1371/journal.pmed.1001747
Sandve GK, NekrutenkoA,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible Computational Research. PLoS
Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285
HOW?
[Adapted Freire, 2013]
transparency
dependencies
steps
provenance
portability
robustness
tolerance
preservation
packaging
versioning
access
available
standards
common APIs
licence
description
intelligible
standards
common metadata
HOW?
sustained sitesFindable
Accessible
Intelligible
Reproducible
http://software-carpentry.org/
http://datacarpentry.org/
http://www.nature.com/sdata/
ELNs
Automation
Checklists
eLabs
Gathering
scattered
research
components
Summary
• Replicable Science is
hard work and poorly
rewarded
• Reproducible Science
=> Transparent Science
but ideally needs to be
born that way
• Collective responsibility
• Barend Mons
• Sean Bechhofer
• Philip Bourne
• Matthew Gamble
• Raul Palma
• Jun Zhao
• Alan Williams
• Stian Soiland-Reyes
• Paul Groth
• Tim Clark
• Juliana Freire
• Alejandra Gonzalez-Beltran
• Philippe Rocca-Serra
• Ian Cottam
• Susanna Sansone
• Kristian Garza
• Hylke Koers
• Norman Morrison
• Ian Fore
• Jill Mesirov
• Robert Stevens
• Steve Pettifer
http://www.researchobject.org
http://www.wf4ever-project.org
http://www.fair-dom.org
http://www.software.ac.uk
Further Reading
• https://www.sciencenews.org/article/redoing-scientific-
research-best-way-find-truth
• Drummond C Replicability is not Reproducibility: Nor is it
Good Science, online
• Peng RD, Reproducible Research in Computational Science
Science 2 Dec 2011: 1226-1227.

More Related Content

What's hot

Big data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simpleBig data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simple
Hadas Jacoby
 
Academic Student Research Proposal Powerpoint Presentation Slides
Academic Student Research Proposal Powerpoint Presentation SlidesAcademic Student Research Proposal Powerpoint Presentation Slides
Academic Student Research Proposal Powerpoint Presentation Slides
SlideTeam
 
Research proposal
Research proposalResearch proposal
Research proposal
Dana Popescu
 
Research question
Research questionResearch question
Research question
Tarek Tawfik Amin
 
Overview of ORCID for researchers
Overview of ORCID for researchersOverview of ORCID for researchers
Overview of ORCID for researchers
ORCID, Inc
 
Research performance measurement
Research performance measurementResearch performance measurement
Research performance measurement
Elham Abied
 
Developing hypothesis and research questions
Developing hypothesis and research questionsDeveloping hypothesis and research questions
Developing hypothesis and research questions
Mahendra Kumar Ghadoliya
 
Materials and Methods
Materials and MethodsMaterials and Methods
Materials and Methods
Jayaramachandran S
 
Artificial Intelligence in Laboratory Services in Healthcare Sector
Artificial Intelligence in Laboratory Services in Healthcare SectorArtificial Intelligence in Laboratory Services in Healthcare Sector
Artificial Intelligence in Laboratory Services in Healthcare Sector
Dr. Shruti Aggarwal
 
Mixing Methods Tutorial
Mixing Methods Tutorial Mixing Methods Tutorial
Mixing Methods Tutorial
Maria Kapsali
 
Publishing Scientific Papers
Publishing Scientific Papers Publishing Scientific Papers
Publishing Scientific Papers
Pokhara University, Pokhara, Nepal
 
Quantitative Research Design-PS2
Quantitative Research Design-PS2Quantitative Research Design-PS2
Quantitative Research Design-PS2
prabin shakya
 
How to write a research proposal
How to write a research proposalHow to write a research proposal
How to write a research proposal
Promila Sheoran
 
Create your Personal Research Identifier/Profile
Create your Personal Research Identifier/ProfileCreate your Personal Research Identifier/Profile
Create your Personal Research Identifier/Profile
Md. Zahid Hossain Shoeb
 
RESEARCH TO PUBLICATION: A JOURNEY
RESEARCH TO PUBLICATION: A JOURNEYRESEARCH TO PUBLICATION: A JOURNEY
RESEARCH TO PUBLICATION: A JOURNEY
Khalid Hakeem
 
RES701 Research Methodology Lecture1
RES701 Research Methodology Lecture1RES701 Research Methodology Lecture1
RES701 Research Methodology Lecture1
VIT University (Chennai Campus)
 
Lecture 03 motivation in research and type of research
Lecture 03   motivation in research and type of researchLecture 03   motivation in research and type of research
Lecture 03 motivation in research and type of research
Deepak Sharma
 
scientific misconduct
scientific misconductscientific misconduct
scientific misconduct
kusuma latha
 
Challenges in clinical research
Challenges in clinical researchChallenges in clinical research
Challenges in clinical research
scanFOAM
 
How to Write a Research Proposal
How to Write a Research ProposalHow to Write a Research Proposal
How to Write a Research Proposal
adil raja
 

What's hot (20)

Big data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simpleBig data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simple
 
Academic Student Research Proposal Powerpoint Presentation Slides
Academic Student Research Proposal Powerpoint Presentation SlidesAcademic Student Research Proposal Powerpoint Presentation Slides
Academic Student Research Proposal Powerpoint Presentation Slides
 
Research proposal
Research proposalResearch proposal
Research proposal
 
Research question
Research questionResearch question
Research question
 
Overview of ORCID for researchers
Overview of ORCID for researchersOverview of ORCID for researchers
Overview of ORCID for researchers
 
Research performance measurement
Research performance measurementResearch performance measurement
Research performance measurement
 
Developing hypothesis and research questions
Developing hypothesis and research questionsDeveloping hypothesis and research questions
Developing hypothesis and research questions
 
Materials and Methods
Materials and MethodsMaterials and Methods
Materials and Methods
 
Artificial Intelligence in Laboratory Services in Healthcare Sector
Artificial Intelligence in Laboratory Services in Healthcare SectorArtificial Intelligence in Laboratory Services in Healthcare Sector
Artificial Intelligence in Laboratory Services in Healthcare Sector
 
Mixing Methods Tutorial
Mixing Methods Tutorial Mixing Methods Tutorial
Mixing Methods Tutorial
 
Publishing Scientific Papers
Publishing Scientific Papers Publishing Scientific Papers
Publishing Scientific Papers
 
Quantitative Research Design-PS2
Quantitative Research Design-PS2Quantitative Research Design-PS2
Quantitative Research Design-PS2
 
How to write a research proposal
How to write a research proposalHow to write a research proposal
How to write a research proposal
 
Create your Personal Research Identifier/Profile
Create your Personal Research Identifier/ProfileCreate your Personal Research Identifier/Profile
Create your Personal Research Identifier/Profile
 
RESEARCH TO PUBLICATION: A JOURNEY
RESEARCH TO PUBLICATION: A JOURNEYRESEARCH TO PUBLICATION: A JOURNEY
RESEARCH TO PUBLICATION: A JOURNEY
 
RES701 Research Methodology Lecture1
RES701 Research Methodology Lecture1RES701 Research Methodology Lecture1
RES701 Research Methodology Lecture1
 
Lecture 03 motivation in research and type of research
Lecture 03   motivation in research and type of researchLecture 03   motivation in research and type of research
Lecture 03 motivation in research and type of research
 
scientific misconduct
scientific misconductscientific misconduct
scientific misconduct
 
Challenges in clinical research
Challenges in clinical researchChallenges in clinical research
Challenges in clinical research
 
How to Write a Research Proposal
How to Write a Research ProposalHow to Write a Research Proposal
How to Write a Research Proposal
 

Viewers also liked

R reproducibility
R reproducibilityR reproducibility
R reproducibility
Revolution Analytics
 
Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks
Results Vary: The Pragmatics of Reproducibility and Research Object FrameworksResults Vary: The Pragmatics of Reproducibility and Research Object Frameworks
Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks
Carole Goble
 
FAIRer Research
FAIRer ResearchFAIRer Research
FAIRer Research
Carole Goble
 
The Research Object Initiative: Frameworks and Use Cases
The Research Object Initiative:Frameworks and Use CasesThe Research Object Initiative:Frameworks and Use Cases
The Research Object Initiative: Frameworks and Use Cases
Carole Goble
 
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
Carole Goble
 
RARE and FAIR Science: Reproducibility and Research Objects
RARE and FAIR Science: Reproducibility and Research ObjectsRARE and FAIR Science: Reproducibility and Research Objects
RARE and FAIR Science: Reproducibility and Research Objects
Carole Goble
 

Viewers also liked (6)

R reproducibility
R reproducibilityR reproducibility
R reproducibility
 
Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks
Results Vary: The Pragmatics of Reproducibility and Research Object FrameworksResults Vary: The Pragmatics of Reproducibility and Research Object Frameworks
Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks
 
FAIRer Research
FAIRer ResearchFAIRer Research
FAIRer Research
 
The Research Object Initiative: Frameworks and Use Cases
The Research Object Initiative:Frameworks and Use CasesThe Research Object Initiative:Frameworks and Use Cases
The Research Object Initiative: Frameworks and Use Cases
 
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
 
RARE and FAIR Science: Reproducibility and Research Objects
RARE and FAIR Science: Reproducibility and Research ObjectsRARE and FAIR Science: Reproducibility and Research Objects
RARE and FAIR Science: Reproducibility and Research Objects
 

Similar to Reproducibility and Scientific Research: why, what, where, when, who, how

Keynote speech - Carole Goble - Jisc Digital Festival 2015
Keynote speech - Carole Goble - Jisc Digital Festival 2015Keynote speech - Carole Goble - Jisc Digital Festival 2015
Keynote speech - Carole Goble - Jisc Digital Festival 2015
Jisc
 
How do we make science better?
How do we make science better?How do we make science better?
How do we make science better?
Christian Bokhove
 
Research Objects for FAIRer Science
Research Objects for FAIRer Science Research Objects for FAIRer Science
Research Objects for FAIRer Science
Carole Goble
 
Emerging challenges in data-intensive genomics
Emerging challenges in data-intensive genomicsEmerging challenges in data-intensive genomics
Emerging challenges in data-intensive genomics
mikaelhuss
 
Minimal viable data reuse
Minimal viable data reuseMinimal viable data reuse
Minimal viable data reuse
voginip
 
The State of Open Research Data - OpenCon 2014
The State of Open Research Data - OpenCon 2014The State of Open Research Data - OpenCon 2014
The State of Open Research Data - OpenCon 2014
Right to Research
 
The State of Open Research Data
The State of Open Research DataThe State of Open Research Data
The State of Open Research Data
Ross Mounce
 
Open Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill PandemicOpen Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill Pandemic
Dorothy Bishop
 
Evolution of e-Research
Evolution of e-ResearchEvolution of e-Research
Evolution of e-Research
David De Roure
 
Roche_open_science_NIOO_KNAW_workshop_NL
Roche_open_science_NIOO_KNAW_workshop_NLRoche_open_science_NIOO_KNAW_workshop_NL
Roche_open_science_NIOO_KNAW_workshop_NL
Dominique Roche
 
The Future of Research (Science and Technology)
The Future of Research (Science and Technology)The Future of Research (Science and Technology)
The Future of Research (Science and Technology)
Duncan Hull
 
Bias and the Data Lifecycle
Bias and the Data LifecycleBias and the Data Lifecycle
Bias and the Data Lifecycle
Richard Ferrers
 
On community-standards, data curation and scholarly communication" Stanford M...
On community-standards, data curation and scholarly communication" Stanford M...On community-standards, data curation and scholarly communication" Stanford M...
On community-standards, data curation and scholarly communication" Stanford M...
Susanna-Assunta Sansone
 
Why study Data Sharing? (+ why share your data)
Why study Data Sharing?  (+ why share your data)Why study Data Sharing?  (+ why share your data)
Why study Data Sharing? (+ why share your data)
Heather Piwowar
 
The Evolution of e-Research: Machines, Methods and Music
The Evolution of e-Research: Machines, Methods and MusicThe Evolution of e-Research: Machines, Methods and Music
The Evolution of e-Research: Machines, Methods and Music
David De Roure
 
Reproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trendsReproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trends
Carole Goble
 
Oxford_15-03-22.pptx
Oxford_15-03-22.pptxOxford_15-03-22.pptx
Oxford_15-03-22.pptx
SaigeRutherford
 
2021-01-27--biodiversity-informatics-gbif-(52slides)
2021-01-27--biodiversity-informatics-gbif-(52slides)2021-01-27--biodiversity-informatics-gbif-(52slides)
2021-01-27--biodiversity-informatics-gbif-(52slides)
Dag Endresen
 
RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015
William Gunn
 
تحليل البيانات وتفسير المعطيات
تحليل البيانات وتفسير المعطياتتحليل البيانات وتفسير المعطيات
تحليل البيانات وتفسير المعطيات
مركز البحوث الأقسام العلمية
 

Similar to Reproducibility and Scientific Research: why, what, where, when, who, how (20)

Keynote speech - Carole Goble - Jisc Digital Festival 2015
Keynote speech - Carole Goble - Jisc Digital Festival 2015Keynote speech - Carole Goble - Jisc Digital Festival 2015
Keynote speech - Carole Goble - Jisc Digital Festival 2015
 
How do we make science better?
How do we make science better?How do we make science better?
How do we make science better?
 
Research Objects for FAIRer Science
Research Objects for FAIRer Science Research Objects for FAIRer Science
Research Objects for FAIRer Science
 
Emerging challenges in data-intensive genomics
Emerging challenges in data-intensive genomicsEmerging challenges in data-intensive genomics
Emerging challenges in data-intensive genomics
 
Minimal viable data reuse
Minimal viable data reuseMinimal viable data reuse
Minimal viable data reuse
 
The State of Open Research Data - OpenCon 2014
The State of Open Research Data - OpenCon 2014The State of Open Research Data - OpenCon 2014
The State of Open Research Data - OpenCon 2014
 
The State of Open Research Data
The State of Open Research DataThe State of Open Research Data
The State of Open Research Data
 
Open Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill PandemicOpen Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill Pandemic
 
Evolution of e-Research
Evolution of e-ResearchEvolution of e-Research
Evolution of e-Research
 
Roche_open_science_NIOO_KNAW_workshop_NL
Roche_open_science_NIOO_KNAW_workshop_NLRoche_open_science_NIOO_KNAW_workshop_NL
Roche_open_science_NIOO_KNAW_workshop_NL
 
The Future of Research (Science and Technology)
The Future of Research (Science and Technology)The Future of Research (Science and Technology)
The Future of Research (Science and Technology)
 
Bias and the Data Lifecycle
Bias and the Data LifecycleBias and the Data Lifecycle
Bias and the Data Lifecycle
 
On community-standards, data curation and scholarly communication" Stanford M...
On community-standards, data curation and scholarly communication" Stanford M...On community-standards, data curation and scholarly communication" Stanford M...
On community-standards, data curation and scholarly communication" Stanford M...
 
Why study Data Sharing? (+ why share your data)
Why study Data Sharing?  (+ why share your data)Why study Data Sharing?  (+ why share your data)
Why study Data Sharing? (+ why share your data)
 
The Evolution of e-Research: Machines, Methods and Music
The Evolution of e-Research: Machines, Methods and MusicThe Evolution of e-Research: Machines, Methods and Music
The Evolution of e-Research: Machines, Methods and Music
 
Reproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trendsReproducibility (and the R*) of Science: motivations, challenges and trends
Reproducibility (and the R*) of Science: motivations, challenges and trends
 
Oxford_15-03-22.pptx
Oxford_15-03-22.pptxOxford_15-03-22.pptx
Oxford_15-03-22.pptx
 
2021-01-27--biodiversity-informatics-gbif-(52slides)
2021-01-27--biodiversity-informatics-gbif-(52slides)2021-01-27--biodiversity-informatics-gbif-(52slides)
2021-01-27--biodiversity-informatics-gbif-(52slides)
 
RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015RDA Scholarly Infrastructure 2015
RDA Scholarly Infrastructure 2015
 
تحليل البيانات وتفسير المعطيات
تحليل البيانات وتفسير المعطياتتحليل البيانات وتفسير المعطيات
تحليل البيانات وتفسير المعطيات
 

More from Carole Goble

The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
Carole Goble
 
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Carole Goble
 
RO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital ObjectsRO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital Objects
Carole Goble
 
Research Software Sustainability takes a Village
Research Software Sustainability takes a VillageResearch Software Sustainability takes a Village
Research Software Sustainability takes a Village
Carole Goble
 
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
Open Research: Manchester leading and learning
Open Research: Manchester leading and learningOpen Research: Manchester leading and learning
Open Research: Manchester leading and learning
Carole Goble
 
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
EOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryEOSC-Life Workflow Collaboratory
EOSC-Life Workflow Collaboratory
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
Carole Goble
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
Carole Goble
 
FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout
Carole Goble
 
FAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practiceFAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practice
Carole Goble
 
RO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research ObjectsRO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research Objects
Carole Goble
 
The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects
Carole Goble
 
How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)
Carole Goble
 
What is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can helpWhat is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can help
Carole Goble
 

More from Carole Goble (20)

The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
The ELIXIR FAIR Knowledge Ecosystem for practical know-how: RDMkit and FAIRCo...
 
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science,  a Digital Research...
Can’t Pay, Won’t Pay, Don’t Pay: Delivering open science, a Digital Research...
 
RO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital ObjectsRO-Crate: packaging metadata love notes into FAIR Digital Objects
RO-Crate: packaging metadata love notes into FAIR Digital Objects
 
Research Software Sustainability takes a Village
Research Software Sustainability takes a VillageResearch Software Sustainability takes a Village
Research Software Sustainability takes a Village
 
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
Title: Love, Money, Fame, Nudge: Enabling Data-intensive BioScience through D...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
Open Research: Manchester leading and learning
Open Research: Manchester leading and learningOpen Research: Manchester leading and learning
Open Research: Manchester leading and learning
 
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...RDMkit, a Research Data Management Toolkit.  Built by the Community for the ...
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
EOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryEOSC-Life Workflow Collaboratory
EOSC-Life Workflow Collaboratory
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...FAIR Data Bridging from researcher data management to ELIXIR archives in the...
FAIR Data Bridging from researcher data management to ELIXIR archives in the...
 
FAIR Computational Workflows
FAIR Computational WorkflowsFAIR Computational Workflows
FAIR Computational Workflows
 
FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout FAIR Workflows and Research Objects get a Workout
FAIR Workflows and Research Objects get a Workout
 
FAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practiceFAIRy stories: the FAIR Data principles in theory and in practice
FAIRy stories: the FAIR Data principles in theory and in practice
 
RO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research ObjectsRO-Crate: A framework for packaging research products into FAIR Research Objects
RO-Crate: A framework for packaging research products into FAIR Research Objects
 
The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects The swings and roundabouts of a decade of fun and games with Research Objects
The swings and roundabouts of a decade of fun and games with Research Objects
 
How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)How are we Faring with FAIR? (and what FAIR is not)
How are we Faring with FAIR? (and what FAIR is not)
 
What is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can helpWhat is Reproducibility? The R* brouhaha and how Research Objects can help
What is Reproducibility? The R* brouhaha and how Research Objects can help
 

Recently uploaded

Alternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart AgricultureAlternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart Agriculture
International Food Policy Research Institute- South Asia Office
 
Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...
Leonel Morgado
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
PirithiRaju
 
AJAY KUMAR NIET GreNo Guava Project File.pdf
AJAY KUMAR NIET GreNo Guava Project File.pdfAJAY KUMAR NIET GreNo Guava Project File.pdf
AJAY KUMAR NIET GreNo Guava Project File.pdf
AJAY KUMAR
 
Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...
Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...
Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...
Creative-Biolabs
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
Sérgio Sacani
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Sérgio Sacani
 
cathode ray oscilloscope and its applications
cathode ray oscilloscope and its applicationscathode ray oscilloscope and its applications
cathode ray oscilloscope and its applications
sandertein
 
Introduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptxIntroduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptx
QusayMaghayerh
 
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdfHolsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
frank0071
 
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Leonel Morgado
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
PsychoTech Services
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Sérgio Sacani
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
PirithiRaju
 
The binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defectsThe binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defects
Sérgio Sacani
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
hozt8xgk
 
Gadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdfGadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdf
PirithiRaju
 
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptxLEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
yourprojectpartner05
 
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
frank0071
 
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li..._Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
LucyHearn1
 

Recently uploaded (20)

Alternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart AgricultureAlternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart Agriculture
 
Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
 
AJAY KUMAR NIET GreNo Guava Project File.pdf
AJAY KUMAR NIET GreNo Guava Project File.pdfAJAY KUMAR NIET GreNo Guava Project File.pdf
AJAY KUMAR NIET GreNo Guava Project File.pdf
 
Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...
Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...
Mechanisms and Applications of Antiviral Neutralizing Antibodies - Creative B...
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
 
cathode ray oscilloscope and its applications
cathode ray oscilloscope and its applicationscathode ray oscilloscope and its applications
cathode ray oscilloscope and its applications
 
Introduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptxIntroduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptx
 
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdfHolsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
 
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
 
The binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defectsThe binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defects
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
 
Gadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdfGadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdf
 
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptxLEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
 
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
 
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li..._Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
 

Reproducibility and Scientific Research: why, what, where, when, who, how

  • 1. Reproducibility and Scientific Research Professor Carole Goble CBE FREng FBCS The University of Manchester, UK carole.goble@manchester.ac.uk Open Data Manchester, 27th January 2015 icanhascheezburger.com why, what, where, when, who, how
  • 2. Scientific publications have at least two goals: (i) to announce a result and (ii) to convince readers that the result is correct ….. papers in experimental [and computational science] should describe the results and provide a clear enough protocol [or algorithm] to allow successful repetition and extension Jill Mesirov Accessible Reproducible Research Science 22 January 2010: Vol. 327 no. 5964 pp. 415-416 DOI: 10.1126/science.1179653 Virtual Witnessing / Minute Taking
  • 4. Why smart parents often tend to have smart kids
  • 5.
  • 6.
  • 7. “an experiment is reproducible until another laboratory tries to repeat it.” Alexander Kohn
  • 8.
  • 9. design cherry picking data, random seed reporting, non-independent bias, poor positive and negative controls, dodgy normalisation, arbitrary cut-offs, premature data triage, un-validated materials, improper statistical analysis, poor statistical power, stop when “get to the right answer”, software misconfigurations misapplied black box software reporting John P. A. Ioannidis Why Most Published Research FindingsAre False, August 30, 2005, DOI: 10.1371/journal.pmed.0020124 incomplete reporting of software configurations, parameters & resource versions, missed steps, missing data, vague methods, missing software Joppa, et al,TroublingTrends in Scientific Software Use SCIENCE 340 May 2013 Empirical Statistical Computational V. Stodden, IMS Bulletin (2013)
  • 10.
  • 11. Transparency / Availability Gap 1. Ioannidis et al., 2009. Repeatability of published microarray gene expression analyses. Nature Genetics 41: 14 2. Science publishing: The trouble with retractions http://www.nature.com/news/2011/111005/full/478026a.html 3. Bjorn Brembs: Open Access and the looming crisis in science https://theconversation.com/open-access-and-the-looming-crisis-in-science-14950 Out of 18 microarray papers, results from 10 could not be reproduced
  • 12. Researcher survey, 1202 respondents (PARSE.insight 2010) Sustainability WHERE? [Hylke Koers]
  • 13. Broken software, broken science • GeoffreyChang, Scripps Institute • Homemade data-analysis program inherited from another lab • Flipped two columns of data, inverting the electron-density map used to derive protein structure • Retract 3 Science papers and 2 papers in other journals • One paper cited by 364The structures of MsbA (purple) and Sav1866 (green) overlap little (left) until MsbA is inverted (right). Miller A Scientist's Nightmare: Software Problem Leads to Five Retractions Science 22 December 2006: vol. 314 no. 5807 1856-1857 http://www.software.ac.uk/blog/2014-12-04-its-impossible-conduct-research-without-software-say-7-out-10-uk-researchers
  • 14. “An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship.The actual scholarship is the complete software development environment, [the complete data] and the complete set of instructions which generated the figures.” David Donoho, “Wavelab and Reproducible Research,” 1995 Morin et al Shining Light into Black Boxes Science 13 April 2012: 336(6078) 159-160 Ince et alThe case for open computer programs, Nature 482, 2012 algorithms configurations tools and apps codes workflows scripts code libraries third party services, system software infrastructure, compilers hardware Self-contained codes??
  • 15.
  • 16. WHY? 12+3 reasons research goes “wrong” 1. Pressure to publish 2. Impact factor mania 3. Tainted resources 4. Bad maths 5. Sins of omission 6. Science is messy 7. Broken peer review 8. Some scientists don’t share 9. Research never reported 10. Poor training -> sloppiness 11. Honest error 12. Fraud 13. Disorganisation & time pressures 14. Cost to prepare and curate materials 15. Inherently “unreplicable ” (one-off data, specialist kit, stochastic) https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted)
  • 17. • replication hostility • resource intensive • no funding, time, recognition, place to publish • the complete environment? Its HARD to Prepare and Independently Test [Norman Morrison]
  • 18. Value People. Data. Method. Software.
  • 19. re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle conceptual replication “show A is true by doing B rather than doing A again” verify but not falsify [Yong, Nature 485, 2012] regenerate figure redo WHAT is reproducibility? this is a heated topic of debate robustness tolerance verificationcompliance validation assurance
  • 20. Can I repeat my method? publish article DEFEND *Adapted from Mesirov, J. Accessible Reproducible Research Science 327(5964), 415-416 (2010) WHEN? same experiment, set up, lab submit article (and move on…) Can I replicate your method? CERTIFY (a window before decay sets in … ) same experiment, set up, independent lab Can I reproducemy results using your method or your results using my method? COMPARE variations on experiment, set up, lab Can I reuseyour results / method in my research ? TRANSFER different experiment
  • 21. WHO? scientific ego-system & access trust, reciprocity, and competition blame scooping no credit / credit drift misinterpretation scrutiny trolling cost of preparation support distraction dependents on old news loss of dowry loss of special sauce hugging flirting voyerism cautionary creeping Tenopir, et al. Data Sharing by Scientists: Practices and Perceptions. PLoS ONE 6(6) 2012 Borgman The conundrum of sharing research data, JASIST 2012
  • 22. John P. A. Ioannidis How to Make More Published ResearchTrue, October 21, 2014 DOI: 10.1371/journal.pmed.1001747 Sandve GK, NekrutenkoA,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 HOW?
  • 23. [Adapted Freire, 2013] transparency dependencies steps provenance portability robustness tolerance preservation packaging versioning access available standards common APIs licence description intelligible standards common metadata HOW? sustained sitesFindable Accessible Intelligible Reproducible
  • 27. Summary • Replicable Science is hard work and poorly rewarded • Reproducible Science => Transparent Science but ideally needs to be born that way • Collective responsibility
  • 28. • Barend Mons • Sean Bechhofer • Philip Bourne • Matthew Gamble • Raul Palma • Jun Zhao • Alan Williams • Stian Soiland-Reyes • Paul Groth • Tim Clark • Juliana Freire • Alejandra Gonzalez-Beltran • Philippe Rocca-Serra • Ian Cottam • Susanna Sansone • Kristian Garza • Hylke Koers • Norman Morrison • Ian Fore • Jill Mesirov • Robert Stevens • Steve Pettifer http://www.researchobject.org http://www.wf4ever-project.org http://www.fair-dom.org http://www.software.ac.uk
  • 29. Further Reading • https://www.sciencenews.org/article/redoing-scientific- research-best-way-find-truth • Drummond C Replicability is not Reproducibility: Nor is it Good Science, online • Peng RD, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.