Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
icanhascheezburger.com
Results
may vary
reproducibility.
science.
software.
Professor Carole Goble
The University of Manch...
“An article about computational
science in a scientific publication
is not the scholarship itself, it is
merely advertisin...
http://www.nature.com/nature/focus/reproducibility/index.html
Corbyn, Nature Oct 2012fraud
“I can’t immediately reproduce the
research in my own laboratory. It
took an estimated 280 ho...
Reporting (publishing)
availability
documentation
Replication Gap
1. Ioannidis et al., 2009. Repeatability of published microarray gene expression analyses. Nature Genetics...
Stodden V, Guo P, Ma Z (2013) Toward Reproducible Computational Research: An
Empirical Analysis of Data and Code Policy Ad...
10 Simple Rules for Reproducible
Computational Research
1. For Every Result, Keep Track of How It Was
Produced
2. Avoid Ma...
republic of science*
regulation of science
institution
core facilities
libraries
*Merton’s four norms of scientific behavi...
recomputation.org
sciencecodemanifesto.org
meta-manifesto
• all X should be available and assessable
forever and ever
• the copyright of X should be clear
• X should...
re-compute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle...
Scientific publications have at least
two goals:
(i) to announce a result and
(ii) to convince readers that the
result is ...
Computational Research Virtual Witnessing
Methods
(techniques, algorithms, spec. of the steps)
Instruments
(codes, service...
reusereproduce
repeat replicate
same experiment
same lab
same experiment
different lab
same experiment
different set up
di...
DesignDesign
ExecutionExecution
Result AnalysisResult Analysis
CollectionCollection
PublishPublish
Peer
Review
Peer
Review...
portability
variability sameness
availability
open
description
intelligibility
[Adapted Freire, 2013]
preservation
packagi...
BioSTIF
method
instruments and laboratory
Workflows:
capture the steps
standardised pipelines
repetition & comparison
reco...
Provenance
the link between computation and results
Record
static verifiable record
partially repeat/reproduce
Track
track...
http://nbviewer.ipython.org/urls/raw.githubusercontent.com/myGrid/DataHackLeiden/alan/Player_example.ipynb?create=1
Workfl...
Open, citable workflows
[Scott Edmunds]
Integrative Framework
galaxyproject.org/
portability
variability sameness
availability
open
description
intelligibility
[Adapted Freire, 2013]
preservation
packagi...
Reporting dimension
Authoring
Exec. Papers
Link docs to experiment
Sweave
Provenance
Track,Version
Replay
Workflows, makef...
Aggregated Assets Infrastructures
Sharing and interlinking multi-stewarded
Methods, Models, Data…
Data
Model
Article
Exter...
made reproducible
[Pettifer, Attwood]
http://getutopia.com
portability
variability sameness
availability
open
description
intelligibility
[Adapted Freire, 2013]
preservation
packagi...
Archiving & Porting Dimension
host
service
Open Store
Sci as a Service
Integrative fws
Preservation
Recompute, limited
ins...
specialist codes
libraries, platforms, tools
services
(cloud)
hosted
services
commodity
platforms
data collections
catalog...
“lets copy the box that the
internet is in”
Archive
Isolation
• Independent
• Self contained
• Single ownership
• Freehold...
Closed codes/services, method
obscurity, manual steps
Joppa et al SCIENCE 340 2013, Morin et al SCIENCE 336 2012
Mitigate
...
The Reproducibility Window
all experiments become less reproducible over time
• The how, why and what
• plan to preserve
•...
The Reproducibility Window
The explicit documentation of designed-in
and anticipated variation
Reproducibility = Hard Work
Data sets
Analyses
Linked to
Linked to
DOI
DOI
Open-Paper
Open-Review
DOI:10.1186/2047-217X-1-...
DesignDesign
ExecutionExecution
Result AnalysisResult Analysis
CollectionCollection
PublishPublish
Peer
Review
Peer
Review...
Software sustainability
Software practices
Software deposition
Long term access to
software
Credit for software
Software J...
The Neylon Equation
Process =
Interest
Friction
x
Number
people
reach
Cameron Neylon, BOSC 2013, http://cameronneylon.net/...
productivity
reproducibility
personal
side effect
public
side effect
From make reproducible to
born reproducible
ramps
Research is
like software.
Release
research.
Jennifer Schopf, Treating Data Like Software: A Case for Production Quality D...
Research Objects
• Bundles and relate multi-hosted digital resources of a scientific experiment or
investigation using sta...
Research Objects for…..
Preservation
Archiving
Exchange & Communication
Release-based Publishing
Credit
Recombination/Remi...
identification
aggregation
annotation
dependencies
provenance
checklists
versioning
RO Core Conventions
encoded using stan...
RO Extensions
code
workflows
data
experiments
biology astronomy
NGS
SysBio
Mass Spec
Discipline
Asset type
Howard Ratner, Chair STM Future Labs Committee, CEO EVP Nature Publishing
Group, Director of Development for CHORUS (Clear...
Victoria Stodden, AMP 2011 http://www.stodden.net/AMP2011/,
Special Issue Reproducible Research Computing in Science and E...
http://sciencecodemanifesto.org/http://matt.might.net/articles/crapl/
Technical stuff is the easy stuff
Social
Matters
Organisation
MetricsCulture
Process
[Adapted, Daron Green]
meta-manifesto
all X should be available and assessable forever
the copyright of X should be clear
X should have citable, ...
• myGrid
– http://www.mygrid.org.uk
• Taverna
– http://www.taverna.org.uk
• myExperiment
– http://www.myexperiment.org
• B...
Acknowledgements
• David De Roure
• Tim Clark
• Sean Bechhofer
• Robert Stevens
• Christine Borgman
• Victoria Stodden
• M...
Results may vary: Collaborations Workshop, Oxford 2014
Results may vary: Collaborations Workshop, Oxford 2014
Upcoming SlideShare
Loading in …5
×

Results may vary: Collaborations Workshop, Oxford 2014

1,418 views

Published on

Thoughts on computational science reproducibility with a focus on software. Given at the Software Sustainability Institute's 2014 Collaborations Workshop

Published in: Science
  • Be the first to comment

Results may vary: Collaborations Workshop, Oxford 2014

  1. 1. icanhascheezburger.com Results may vary reproducibility. science. software. Professor Carole Goble The University of Manchester, UK The Software Sustainability Institute carole.goble@manchester.ac.uk @caroleannegoble Collaborations Workshop, Oxford, 26 March 2014
  2. 2. “An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment, [the complete data] and the complete set of instructions which generated the figures.” David Donoho, “Wavelab and Reproducible Research,” 1995 datasets data collections algorithms configurations tools and apps codes workflows scripts code libraries services, system software infrastructure, compilers hardware Morin et al Shining Light into Black Boxes Science 13 April 2012: 336(6078) 159-160 Ince et al The case for open computer progra Nature 482, 2012
  3. 3. http://www.nature.com/nature/focus/reproducibility/index.html
  4. 4. Corbyn, Nature Oct 2012fraud “I can’t immediately reproduce the research in my own laboratory. It took an estimated 280 hours for an average user to approximately reproduce the paper. Data/software versions. Workflows are maturing and becoming helpful” disorganisation Phil Bourne Garijo et al. 2013 Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome PLOS ONE, DOI: 10.1371/journal.pone.0080278. inherent
  5. 5. Reporting (publishing) availability documentation
  6. 6. Replication Gap 1. Ioannidis et al., 2009. Repeatability of published microarray gene expression analyses. Nature Genetics 41: 14 2. Science publishing: The trouble with retractions http://www.nature.com/news/2011/111005/full/478026a.html 3. Bjorn Brembs: Open Access and the looming crisis in science https://theconversation.com/open-access-and-the-looming-crisis-in-science-14950 Out of 18 microarray papers, results from 10 could not be reproduced Out of 18 microarray papers, results from 10 could not be reproduced
  7. 7. Stodden V, Guo P, Ma Z (2013) Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals. PLoS ONE 8(6): e67111. doi:10.1371/journal.pone.0067111 Required as condition of publication Required but may not affect decisions Explicitly encouraged may be reviewed and/or hosted Implied No mention Required as condition of publication Required but may not affect decisions Explicitly encouraged may be reviewed and/or hostedImplied No mention 170 journals, 2011-2012
  8. 8. 10 Simple Rules for Reproducible Computational Research 1. For Every Result, Keep Track of How It Was Produced 2. Avoid Manual Data Manipulation Steps 3. Archive the Exact Versions of All External Programs Used 4. Version Control All Custom Scripts 5. Record All Intermediate Results, When Possible in Standardized Formats 6. For Analyses That Include Randomness, Note Underlying Random Seeds 7. Always Store Raw Data behind Plots 8. Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected 9. Connect Textual Statements to Underlying Results 10. Provide Public Access to Scripts, Runs, and Results Citation: Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 Record Everything Automate Everything
  9. 9. republic of science* regulation of science institution core facilities libraries *Merton’s four norms of scientific behaviour (1942) public services
  10. 10. recomputation.org sciencecodemanifesto.org
  11. 11. meta-manifesto • all X should be available and assessable forever and ever • the copyright of X should be clear • X should have citable, versioned identifiers • researchers using X should visibly credit X’s creators • credit should be assessable and count in all assessments • X should be curated, available, linked to all necessary materials, and intelligible
  12. 12. re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle conceptual replication “show A is true by doing B rather than doing A again” verify but not falsify [Yong, Nature 485, 2012] regenerate the figure redo
  13. 13. Scientific publications have at least two goals: (i) to announce a result and (ii) to convince readers that the result is correct ….. papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension Jill Mesirov Accessible Reproducible Research Science 22 Jan 2010: 327(5964): 415-416 DOI: 10.1126/science.1179653 Virtual Witnessing* *Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer.
  14. 14. Computational Research Virtual Witnessing Methods (techniques, algorithms, spec. of the steps) Instruments (codes, services, scripts, underlying libraries) Laboratory (sw and hw infrastructure, systems software, integrative platforms) Materials (datasets, parameters, algorithm seeds) Experiment Setup
  15. 15. reusereproduce repeat replicate same experiment same lab same experiment different lab same experiment different set up different experiment some of same test Drummond C Replicability is not Reproducibility: Nor is it Good Science, online Peng RD, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
  16. 16. DesignDesign ExecutionExecution Result AnalysisResult Analysis CollectionCollection PublishPublish Peer Review Peer Review Peer Reuse Peer Reuse PredictionPrediction Can I repeat & defend my method? Can I review / reproduce and compare my results / method with your results / method? Can I review / replicate and certify your method? Can I transfer your results into my research and reuse this method? * Adapted from Mesirov, J. Accessible Reproducible Research Science 327(5964), 415-416 (2010)
  17. 17. portability variability sameness availability open description intelligibility [Adapted Freire, 2013] preservation packaging gather dependencies capture steps track & keep results gather dependencies capture steps track & keep results A Reproducibility Framework Reporting dimension Archive dimension versioning
  18. 18. BioSTIF method instruments and laboratory Workflows: capture the steps standardised pipelines repetition & comparison record experiment & set-up provenance collection reporting embedded player variant reuse infrastructure shield localised / distributed in-house / external multi-code experiments materials http://www.taverna.org.uk
  19. 19. Provenance the link between computation and results Record static verifiable record partially repeat/reproduce Track track changes carry citation select data to keep/release Analytics repair calc data quality/trust compare diffs/discrepancies W3C PROV standard d1 S0 d2 S1 w S2 y S4 df d1' S0 d2 S1 z w S'2 y' S4 df' (i) Trace A (ii) Trace B PDIFF: comparing provenance traces to diagnose divergence across experimental results [Woodman et al, 2011]
  20. 20. http://nbviewer.ipython.org/urls/raw.githubusercontent.com/myGrid/DataHackLeiden/alan/Player_example.ipynb?create=1 Workflows: sharing and reporting
  21. 21. Open, citable workflows [Scott Edmunds]
  22. 22. Integrative Framework galaxyproject.org/
  23. 23. portability variability sameness availability open description intelligibility [Adapted Freire, 2013] preservation packaging gather dependencies capture steps track & keep results gather dependencies capture steps track & keep results A Reproducibility Framework Reporting dimension Archive dimension versioning
  24. 24. Reporting dimension Authoring Exec. Papers Link docs to experiment Sweave Provenance Track,Version Replay Workflows, makefiles service Sci as a Service Integrative fws Read & Run, Co-location No installation host Open Store Descriptive read, White Box Archived record
  25. 25. Aggregated Assets Infrastructures Sharing and interlinking multi-stewarded Methods, Models, Data… Data Model Article External Databases http://www.seek4science.org Metadata http://www.isatools.org
  26. 26. made reproducible [Pettifer, Attwood] http://getutopia.com
  27. 27. portability variability sameness availability open description intelligibility [Adapted Freire, 2013] preservation packaging gather dependencies capture steps track & keep results gather dependencies capture steps track & keep results A Reproducibility Framework Reporting dimension Archive dimension versioning
  28. 28. Archiving & Porting Dimension host service Open Store Sci as a Service Integrative fws Preservation Recompute, limited installation, Black Box Byte execution Descriptive read, White Box Archived record Read & Run, Co-location No installation ReproZipPackaging Porting White Box, Installation Archived record
  29. 29. specialist codes libraries, platforms, tools services (cloud) hosted services commodity platforms data collections catalogues software repositories my data my process my codes integrative frameworks gateways
  30. 30. “lets copy the box that the internet is in” Archive Isolation • Independent • Self contained • Single ownership • Freehold • Fixed • Self described Active Ecosystem • Dependent • Distributed • Multi-ownership • Tenancy • Changeable / variable • Multi-described
  31. 31. Closed codes/services, method obscurity, manual steps Joppa et al SCIENCE 340 2013, Morin et al SCIENCE 336 2012 Mitigate Detect Repair Zhao, Gomez-Perez, Belhajjame, Klyne, Garcia-Cuesta, Garrido, Hettne, Roos, De Roure and Goble. Why workflows break - Understanding and combating decay in Taverna workflows, 8th Intl Conf e-Science 2012
  32. 32. The Reproducibility Window all experiments become less reproducible over time • The how, why and what • plan to preserve • prepare to repair • description persists • common frameworks • partial replication • approximate reproduction • verification • benchmarks for codes Reproducibility by Invocation Run It Reproducibility by Inspection Read It
  33. 33. The Reproducibility Window The explicit documentation of designed-in and anticipated variation
  34. 34. Reproducibility = Hard Work Data sets Analyses Linked to Linked to DOI DOI Open-Paper Open-Review DOI:10.1186/2047-217X-1-18 >11000 accesses Open-Code 8 reviewers tested data in ftp server & named reports published DOI:10.5524/100044 Open-Pipelines Open-Workflows DOI:10.5524/100038 Open-Data 78GB CC0 data Code in sourceforge under GPLv3: http://soapdenovo2.sourceforge.net/>5000 downloads Enabled code to being picked apart by bloggers in wiki http://homolog.us/wiki/index.php?title=SOAPdenovo2 [Scott Edmunds]
  35. 35. DesignDesign ExecutionExecution Result AnalysisResult Analysis CollectionCollection PublishPublish Peer Review Peer Review Peer Reuse Peer Reuse PredictionPrediction * Adapted from Mesirov, J. Accessible Reproducible Research Science 327(5964), 415-416 (2010) Reproducible Research Environment Integrated infrastructure for producing and working with reproducible research. Reproducible Research Publication Environment Distributing and reviewing; credit; licensing etc. From make reproducible to born reproducible
  36. 36. Software sustainability Software practices Software deposition Long term access to software Credit for software Software Journals Licensing Open Source Software Best Practices for Scientific Computing http://arxiv.org/abs/1210.0530 Stodden, Reproducible Research Standard, Intl J Comm Law & Policy, 13 2009 From make reproducible to born reproducible From make reproducible to born reproducible
  37. 37. The Neylon Equation Process = Interest Friction x Number people reach Cameron Neylon, BOSC 2013, http://cameronneylon.net/ From make reproducible to born reproducible
  38. 38. productivity reproducibility personal side effect public side effect From make reproducible to born reproducible ramps
  39. 39. Research is like software. Release research. Jennifer Schopf, Treating Data Like Software: A Case for Production Quality Data, JCDL 2012 From make reproducible to born reproducible
  40. 40. Research Objects • Bundles and relate multi-hosted digital resources of a scientific experiment or investigation using standard mechanisms • Exchange, Releasing paradigm for publishing http://www.researchobject.org/
  41. 41. Research Objects for….. Preservation Archiving Exchange & Communication Release-based Publishing Credit Recombination/Remix Reproducibility, Computation Training
  42. 42. identification aggregation annotation dependencies provenance checklists versioning RO Core Conventions encoded using standards Minim Information Model Ontology W3C PROV PAV, VoID Git OAI-ORE W3C OAM DOI, ORCID, PURL
  43. 43. RO Extensions code workflows data experiments biology astronomy NGS SysBio Mass Spec Discipline Asset type
  44. 44. Howard Ratner, Chair STM Future Labs Committee, CEO EVP Nature Publishing Group, Director of Development for CHORUS (Clearinghouse for the Open Research of US) STM Innovations Seminar 2012 http://www.youtube.com/watch?v=p-W4iLjLTrQ&list=PLC44A300051D052E5
  45. 45. Victoria Stodden, AMP 2011 http://www.stodden.net/AMP2011/, Special Issue Reproducible Research Computing in Science and Engineering July/August 2012, 14(4) Howison and Herbsleb (2013) "Incentives and Integration In Scientific Software Production" CSCW 2013.
  46. 46. http://sciencecodemanifesto.org/http://matt.might.net/articles/crapl/
  47. 47. Technical stuff is the easy stuff Social Matters Organisation MetricsCulture Process [Adapted, Daron Green]
  48. 48. meta-manifesto all X should be available and assessable forever the copyright of X should be clear X should have citable, versioned identifiers researchers using X should visibly credit X’s creators credit should be assessable and count in all assessments X should be curated, available, linked to all necessary materials, and intelligible • reproducibility spectrum • descriptive reproducibility • papers -> research objects • make reproducible -> born reproducible • ramp up tools -> working practice • adapt and train -> researchers • cost & responsibility -> transparent, accountable and collective • dominants -> society, culture and policy • take action, be imperfect
  49. 49. • myGrid – http://www.mygrid.org.uk • Taverna – http://www.taverna.org.uk • myExperiment – http://www.myexperiment.org • BioCatalogue – http://www.biocatalogue.org • Biodiversity Catalogue – http://www.biodiversitycatalogue.org • Seek – http://www.seek4science.org • Rightfield – http://www.rightfield.org.uk • Open PHACTS – http://www.openphacts.org • Wf4ever – http://www.wf4ever-project.org • Software Sustainability Institute – http://www.software.ac.uk • BioVeL – http://www.biovel.eu • Force11 – http://www.force11.org
  50. 50. Acknowledgements • David De Roure • Tim Clark • Sean Bechhofer • Robert Stevens • Christine Borgman • Victoria Stodden • Marco Roos • Jose Enrique Ruiz del Mazo • Oscar Corcho • Ian Cottam • Steve Pettifer • Magnus Rattray • Chris Evelo • Katy Wolstencroft • Robin Williams • Pinar Alper • C. Titus Brown • Greg Wilson • Kristian Garza • Wf4ever, SysMO, BioVel, UTOPIA and myGrid teams • Juliana Freire • Jill Mesirov • Simon Cockell • Paolo Missier • Paul Watson • Gerhard Klimeck • Matthias Obst • Jun Zhao • Pinar Alper • Daniel Garijo • Yolanda Gil • James Taylor • Alex Pico • Sean Eddy • Cameron Neylon • Barend Mons • Kristina Hettne • Stian Soiland-Reyes • Rebecca Lawrence

×