Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Being Reproducible: SSBSS Summer School 2017


Published on

Lecture 2:
Being Reproducible: Models, Research Objects and R* Brouhaha
Reproducibility is a R* minefield, depending on whether you are testing for robustness (rerun), defence (repeat), certification (replicate), comparison (reproduce) or transferring between researchers (reuse). Different forms of "R" make different demands on the completeness, depth and portability of research. Sharing is another minefield raising concerns of credit and protection from sharp practices.
In practice the exchange, reuse and reproduction of scientific experiments is dependent on bundling and exchanging the experimental methods, computational codes, data, algorithms, workflows and so on along with the narrative. These "Research Objects" are not fixed, just as research is not “finished”: the codes fork, data is updated, algorithms are revised, workflows break, service updates are released. is an effort to systematically support more portable and reproducible research exchange.
In this talk I will explore these issues in more depth using the FAIRDOM Platform and its support for reproducible modelling. The talk will cover initiatives and technical issues, and raise social and cultural challenges.

Published in: Science
  • Login to see the comments

  • Be the first to like this

Being Reproducible: SSBSS Summer School 2017

  1. 1. Being Reproducible: Models, Research Objects and R* Brouhaha Professor Carole Goble, The University of Manchester, UK The FAIRDOM Association Coordinator ELIXIR-UK Head of Node Co-lead ELIXIR Interoperability Platform SSBSS 2017, July 17 2017, Cambridge, UK 4th International Synthetic & Systems Biology Summer School
  2. 2. Reproducibility Rampancy
  3. 3. 47/53 “landmark” publications could not be replicated [Begley, Ellis Nature, 483, 2012]
  4. 4. Retraction Misconduct is the main cause of life-sciences retractions Zoë Corbyn 01 October 2012
  5. 5. Vahan Simonyan, Center for Biologics Evaluation and Research Food and Drug Administration USA
  6. 6. NIH Rigor and Reproducibility training/rigor-reproducibility projects/reproducibility-and-reliability-of- biomedical-research/
  7. 7. John P. A. Ioannidis How to Make More Published ResearchTrue, October 21, 2014 DOI: 10.1371/journal.pmed.1001747
  8. 8. Reproducibility of biological experiments is hard for in vivo/vitro and for in silico analysis • OS version • Revision of scripts • Data analysis software versions • Version of data files • Command line parameters written on a napkin • “Black magic” only a grad student knows Fix with latest technologies, best practices and willingness [Keiichiro Ono, Scripps Institute] The first step is to be FAIR See the whole of the previous talk…
  9. 9. Record All Automate All Contain All Access All Findable (Citable) Accessible (Trackable) Interoperable (Intelligible) Reusable (Reproducible)
  10. 10. design cherry picking data, random seed reporting, non-independent bias, poor positive and negative controls, dodgy normalisation, arbitrary cut-offs, premature data triage, un-validated materials, improper statistical analysis, poor statistical power, stop when “get to the right answer”, software misconfigurations misapplied black box software reporting incomplete reporting of software configurations, parameters & resource versions, missed steps, missing data, vague methods, missing software Empirical Statistical Computational V. Stodden, IMS Bulletin (2013) Reproducibility and reliability of biomedical research: improving research practice
  11. 11. “When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.” Carroll, Through the Looking Glass re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle redo robustness tolerance verificationcompliancevalidation assurance remix
  12. 12. Scientific publications goals: (i) announce a result (ii) convince readers its correct. Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension. Papers in computational science should describe the results and provide the complete software development environment, data and set of instructions which generated the figures. VirtualWitnessing* *Leviathan and theAir-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer. Jill Mesirov David Donoho
  13. 13. “Micro” Reproducibility “Macro” Reproducibility Fixivity Validate Verify Trust
  14. 14. Repeatability: “Sameness” Same result 1 Lab 1 experiment Reproducibility: “Similarity” Similar result > 1 Lab > 1 experiment why the differences? https://2016-oslo- Validate Verify
  15. 15. Method Reproducibility the provision of enough detail about study procedures and data so, in theory or in actuality, the same procedures could be exactly repeated. Result Reproducibility (aka replicability) obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible Goodman, et al ScienceTranslational Medicine 8 (341) 2016 Validate Verify
  16. 16. What are you reproducing? Algorithm vs its script conflation Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
  17. 17. Productivity Track differences Validate Verify
  18. 18. Validate Verify Recompute By Degrees Fixivity - Liveness • New/updated/deprecated methods, datasets, services, codes, h/w • Snapshots Dependency – Containment • Streams, non-portable data/software, • 3rd party services, supercomputer access, licensing restrictions…. • Locally contained and maintained • External dependencies Transparency • Blackboxes, proprietary software, manual steps Robustness • Bounds of use • Stochastics, non-deterministics, contexts
  19. 19. Components and Dependencies Software are typically compound works. Libraries. Plug-ins. Code fragments. We are encouraged to reuse and not reinvent Combining licenses. License compatibilities
  20. 20. Black boxes • closed codes • closed external or cloud services • method obscurity • manual steps [Thanks to Jason Scott]
  21. 21. The ReproducibilityWindow all experiments become less reproducible over time…. • Can’t contain everything – Pesky Internet in a Box • Can’t automate everything – Pesky people intervening • Can’t fix and fossils everything – Pesky science keeps changing Results may vary
  22. 22. Bonus slide At SSBSS Theodor Gescher came up with REALSCI Robust -many runs Environment -describe the equipment/OS Another -done by not your lab Limits -parameters Standards -well understood/comprehensible methods Complete -not cherry picking Immortal -community supported commodity systems
  23. 23. Mixed Central and Distributed stores: Containment and Dependencies. Upload vs Referencing In House Stores External Databases Publishing services Model Resources
  24. 24. Mixed Central and Distributed stores: Containment and Dependencies. Upload vs Referencing In House Stores External Databases Publishing services Model Resources Migrations into FAIRDOMHub For long term reproducibility
  25. 25. Shades of Reproducibility Running an active instrument Reading an archived record Are you using hard-wired localhost ids? Workflows SOPs Containers, cloud services, common services Markup languages, reporting guidelines and checklists, ontologies, catalogues Sounds hard…. what can I do? Catalogue
  26. 26. Protocol specs and sharing… A language for specifying experimental protocols for biological research in way that is precise, unambiguous, and understandable by both humans and computers.
  27. 27. Validation Data
  28. 28. Standard Operating Procedures Quality Control
  29. 29. in situ reproducible models in FAIRDOM metadata annotation against standards validation, comparison and simulation SBML Model simulation Model comparison Model versioning Reproducing simulations [Jacky Snoep, Dagmar Waltemath, Martin Peters, Martin Scharm] JWS Online
  30. 30. Tracking versi0ns
  31. 31. Tracking model versions smartly Scharm, M., Wolkenhauer, O., & Waltemath, D. (2015). An algorithm to detect and communicate the differences in computational models describing biological systems. Bioinformatics, btv484
  32. 32. Model simulation in FAIRDOMHub using JWS Online
  33. 33. A simulation database allows a one-click, live figure reproduction in a FAIRDOM-SEEK JWS model Excel data file Dagmar Waltemath, Uni Rostock Jacky Snoep, Uni Stellenbosch Simulation Experiment Description Markup Language: XML-based format for encoding simulation setups, to ensure exchangeability and reproducibility of simulation experiments • which models to use in an experiment, • modifications to apply on the models before using them, • which simulation procedures to run on each model, • what analysis results to output, • and how the results should be presented.
  34. 34. FAIRDOMHub Journal Programme Molecular Systems Biology
  35. 35. ModelTechnical curation forJournals [Jacky Snoep (Stellenbosch), DagmarWaltemath, Martin Peters, Martin Scharm (Rostock)] * store DOI citable supplementary files on FAIRDOMHub ** model and data curation *** reproducible clickable figures in papers using SED-ML
  36. 36. Cataloguing Packaging Penkler, G., du Toit, F., Adams, W., Rautenbach, M., Palm, D. C., van Niekerk, D. D. and Snoep, J. L. (2015), Construction and validation of a detailed kinetic model of glycolysis in Plasmodium falciparum. FEBS J, 282: 1481–1511. doi:10.1111/febs.13237 DOI: 10.15490/seek.1.investigation.56 Snapshot preservation active
  37. 37. 18/07/2017 39 An “evolving manuscript” would begin with a pre- publication, pre-peer review “beta 0.9” version of an article, followed by the approved published article itself, [ … ] “version 1.0”. Subsequently, scientists would update this paper with details of further work as the area of research develops. Versions 2.0 and 3.0 might allow for the “accretion of confirmation [and] reputation”. Ottoline Leyser […] assessment criteria in science revolve around the individual. “People have stopped thinking about the scientific enterprise”.
  38. 38. Packaging: CombineArchive Scharm M,Wendland F, Peters M,Wolfien M,TheileT,Waltemath D SEMS, University of Rostock zip-like file with a manifest & metadata - Bundling files - Keeping provenance - Exchanging data - Shipping results Bergmann, F.T.,Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., ... & Olivier, B. G. (2014). COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project. BMC bioinformatics,15(1), 1.
  39. 39. Standards-based metadata framework for bundling (scattered) resources with context and citation Packaging: Research Objects
  40. 40. Packaging: Research Objects Publishing Archive Institutional Archive 1.Export 2.Exchange
  41. 41. Manifest Construction Container Manifest Description Packaging Platforms: Zip files, BagIt, Docker, Conda, Singularity Repositories FAIRDOMHub Packaging: Research Objects in a nutshell Different manifest description profiles for different kinds of objects
  42. 42. FromVirtual Machines to Executable Containers for portable execution • Containers everything required to make a piece of software run is packaged into isolated containers. • UnlikeVMs, containers do not bundle a full operating system - only libraries and settings required to make the software work. • Efficient, lightweight, self-contained systems • Guarantees that software will always run the same, regardless of where it’s deployed. Biocontainers
  43. 43. Use commodity and community systems Sustained platforms Communities to drive them Tooling and training Spreadsheets are the Cockroaches of Science
  44. 44. EU FAIR Data Expert Group Consultation EG/consultation/issues
  45. 45. What to know more? Go on a Software or Data Carpentry Course
  46. 46. Make software open and reusable
  47. 47. Software Sustainability Institute , Goble, Better Software Better Research IEEE Internet Computing 18(5), (2014 ) DOI: 10.1109/MIC.2014.88 Jiménez RC, Kuzak M, Alhamdoosh M et al. Four simple recommendations to encourage best practices in research software [version 1; referees: 3 approved]. F1000Research 2017, 6:876 (doi: 10.12688/f1000research.11407.1)
  48. 48. Use Common Platforms Get the licencing right… MATLAB Mathematica…. Proprietary software Cloud Centralised Service insitu reproducibility…. Galaxy FAIRDOMHub + JWS Online Blackbox vs Whitebox
  49. 49. sWebTeam/ebi-metagenomics- cwl/tree/fa86fce/workflows/rna-selector.cwl Use and document workflows preferrably a workflow management system, Living Research Objects! Workflow repository
  50. 50. Use a workflow – the vision! preferrably a workflow management system preferrably described using CommonWorkflow Language Experimental workflows Event BUS Business Process Management Taverna Knime Galaxy Workflow BPM layer Workflow Computation Application layer Computing resources Databases Effector layer Front-end Web interface / Monitoring interface Pipeline Pilot FAIRDOM SEEK Workflow repository Workflow portal repository launch, results FAIRDOM [Jean Loup Fallon, Carole Goble]
  51. 51. Reproducible Pipelines for Robust Regulation BioCompute Objects Emphasis on fixing the pipeline so it can be replicated, and on reporting the parameter space
  52. 52. Use an Electronic Lab Notebook
  53. 53. What can you do? • Follow the 10 RACA Principles • Take action, be imperfect • Demand reproducibility in reviews. • Educate your PIs and supervisors.
  54. 54. [Norman Morrison] Technological Debt: Appropriate Effort Retrospective Reusability 
  55. 55. What are the incentives? [Garza] [Malone] [Resnik]
  56. 56. Acknowledgements • David De Roure • Tim Clark • Sean Bechhofer • Robert Stevens • Christine Borgman • Victoria Stodden • Marco Roos • Jose Enrique Ruiz del Mazo • Oscar Corcho • Ian Cottam • Steve Pettifer • Magnus Rattray • Chris Evelo • Katy Wolstencroft • Robin Williams • Pinar Alper • C. Titus Brown • Greg Wilson • Kristian Garza • Juliana Freire • Jill Mesirov • Simon Cockell • Paolo Missier • Paul Watson • Gerhard Klimeck • Matthias Obst • Jun Zhao • Pinar Alper • Daniel Garijo • Yolanda Gil • James Taylor • Alex Pico • Sean Eddy • Cameron Neylon • Barend Mons • Kristina Hettne • Stian Soiland-Reyes • Rebecca Lawrence • Michael Crusoe
  57. 57. Jon OlavVik, Norwegian University of Life Science Maksim Zakhartsev University Hohenheim, Stuttgart, Germany Alexey Kolodkin Siberian Branch Russian Academy of Sciences Tomasz Zieliński, SynthSys Centre University Edinburgh, UK Martin Peters, Martin Scharm Systems Biology Bioinformatics University of Rostock, Germany
  58. 58. Web sites • Force11 • TeSS • FAIRDOM • FAIRDOMHub • Software Carpentry • Data Carpentry • Software Sustainability Institute • Rightfield • FAIRSharing • CommonWorkflow Language
  59. 59. Reading List (refs also throughout) • John P. A. Ioannidis How to Make More Published ResearchTrue, October 21, 2014 DOI: 10.1371/journal.pmed.1001747 • Ioannidis JPA (2005) Why Most Published Research FindingsAre False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 • Steven N. Goodman*, Daniele Fanelli and John P. A. Ioannidis,What does research reproducibility mean? Science Translational Medicine 01 Jun 2016:Vol. 8, Issue 341, pp. 341ps12 DOI: 10.1126/scitranslmed.aaf5027 • Sandve GK, Nekrutenko A,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 • Massimiliano Assante, Leonardo Candela, DonatellaCastelli, Paolo Manghi and Pasquale Pagano, Science 2.0 Repositories:Time for a Change in Scholarly Communication, D-Lib Magazine January/February 2015,Volume 21, Number 1/2 , DOI: 10.1045/january2015-assante • Waltemath, D., Henkel, R., Hälke, R., Scharm, M., &Wolkenhauer, O. (2013). Improving the reuse of computational models through version control.Bioinformatics, 29(6), 742-748. • Bergmann, F.T., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., ... & Olivier, B. G. (2014). COMBINE archive andOMEX format: one file to share all information to reproduce a modeling project. BMC bioinformatics,15(1), 1. • Scharm, M.,Wolkenhauer, O., &Waltemath, D. (2015). An algorithm to detect and communicate the differences in computational models describing biological systems. Bioinformatics, btv484 • •