Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks


Published on

Keynote presentation at the iConference 2015, Newport Beach, Los Angeles, 26 March 2015.
Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks
BEWARE: presentation includes hidden slides AND in situ build animations - best viewed by downloading.

Published in: Science
  • Very Easy To Follow. This plans are great. I was able to bring my laptop batteries and several other types of batteries back to life with your methods. Your instructions are very easy to follow. I have a few more batteries I'm going to recondition today also. ●●●
    Are you sure you want to  Yes  No
    Your message goes here
  • 8 Ways to Communicate with Your Guardian Angels... ■■■
    Are you sure you want to  Yes  No
    Your message goes here
  • How to Love Yourself: 15 Tips for Developing Self Love ♥♥♥
    Are you sure you want to  Yes  No
    Your message goes here
  • How to Manifest Anything You Want in 24 hours ■■■
    Are you sure you want to  Yes  No
    Your message goes here

Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks

  1. 1. ResultsVary:The Pragmatics of Reproducibility and Research Object Frameworks Professor Carole Goble CBE FREng FBCS The University of Manchester, UK The Software Sustainability Institute iConference, 26 March 2015, Newport Beach, Los Angeles, USA
  2. 2. What do I do? CyberInfrastructure EcoSystems. e-Lab Collabs. & Shared Asset Repositories Knowledge, Metadata, Linked Data, Ontologies Software Engineering for Scientists Computational Workflow Systems Scholarly Comms Reproducibility Micro Publications Open Science Research Objects Linked Data for Science
  3. 3. Scientific EgoSystems Biodiversity Systems Biology Synthetic Biology Astronomy HelioPhysics Genomics Health Epidemiology Digital Preservation Social Science Pharmacology
  4. 4. KnowledgeTurning, Flow Barriers to Cure » Access to scientific resources » Coordination and Collaboration » Flow of Information
  5. 5. [Pettifer, Attwood]
  6. 6. VirtualWitnessing* Scientific publications: » announce a result » convince readers the result is correct “papers in experimental [and computational science] should describe the results and provide a clear enough protocol [algorithm] to allow successful repetition and extension” Jill Mesirov, Broad Institute, 2010** **Accessible Reproducible Research, Science 22January 2010,Vol. 327 no. 5964 pp. 415-416, DOI: 10.1126/science.1179653 *Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer.
  7. 7. Bramhall et al QUALITY OF METHODS REPORTING IN ANIMAL MODELS OF COLITIS Inflammatory Bowel Diseases, , 2015, “Only one of the 58 papers reported all essential criteria on our checklist. Animal age, gender, housing conditions and mortality/morbidity were all poorly reported…..”
  8. 8. “An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship.The actual scholarship is the complete software development environment, [the complete data] and the complete set of instructions which generated the figures.” David Donoho, “Wavelab and Reproducible Research,” 1995 Datasets, Data collections Standard operating procedures Software, algorithms Configurations, Tools and apps, services Codes, code libraries Workflows, scripts System software Infrastructure Compilers, hardware Morin et al Shining Light into Black Boxes Science 2012: 336(6078) 159-160 , Ince et alThe case for open computer programs, Nature 482, 2012 50papers randomly chosen from 378 manuscripts in 2011 that use BurrowsWheeler Aligner for mapping Illumina reads 31no s/w version, parameters, exact version of genomic reference sequence 26no access to primary data sets Nekrutenko &Taylor, Next-generation sequencing data interpretation: enhancing, reproducibility and accessibility, Nature Genetics 13 (2012)
  9. 9. Broken software Broken science » GeoffreyChang, Scripps Institute » Homemade data-analysis program inherited from another lab » Flipped two columns of data, inverting the electron-density map used to derive protein structure » Retract 3 Science papers and 2 papers in other journals » One paper cited by 364 The structures of MsbA (purple) and Sav1866 (green) overlap little (left) until MsbA is inverted (right). Miller A Scientist's Nightmare: Software Problem Leads to Five Retractions Science 22 December 2006: vol. 314 no. 5807 1856-1857
  10. 10. Software making practices “As a general rule, researchers do not test or document their programs rigorously, and they rarely release their codes, making it almost impossible to reproduce and verify published results generated by scientific software” 2000 scientists. J.E. Hannay et al., “How Do Scientists Develop and Use Scientific Software?” Proc. ICSEWorkshop Software Eng. for Computational Science and Eng., 2009, pp. 1–8.
  11. 11. republic of science* regulation of science institution cores libraries *Merton’s four norms of scientific behaviour (1942) public services
  12. 12. Tools, Standards Machine actionable, Formats, Reporting, Policies, Practices
  13. 13. Record and Automate Everything. PotentialTrace Heaven Folks!
  14. 14. Honest Error Science is messy Inherent Reinhart/Rogoff Austerity economics Thomas Herndon Nature Oct ’12 Zoë Corbyn Fraud
  15. 15. “I can’t immediately reproduce the research in my own laboratory. It took an estimated 280 hours for an average user to approximately reproduce the paper.” Prof Phil Bourne Associate Director, NIH Big Data 2 Knowledge Program
  16. 16. When research goes “wrong” »Tainted resources »Black boxes »Poor Reporting »Unavailable resources / results: data, software »Bad maths »Sins of omission »Poor training, sloppiness (adapted) Ioannidis, Why Most Published Research Findings Are False, August 2005 Joppa, et al,TroublingTrends inScientificSoftwareUseSCIENCE 340 May 2013 Scientific method
  17. 17. Social environment » Impact factor mania » Pressure to publish » Broken peer review » Research never reported » Disorganisation » Time pressures » Prep & curate costs When research goes “wrong” (adapted) Morrison Do a Replication Study? No thanks! Not FAIR. Hard. Resource intensive. Unrecognised. Trolled. Just gathering the bits together .
  18. 18. Cross-Institutional e-Laboratory Fragmentation Scattered parts, Subject specific / General resources 101 Innovations in Scholarly Communication - the Changing ResearchWorkflow, Boseman and Kramer, 2015,
  19. 19. Process at Scale More on Models
  20. 20. [Snoep, 2015]
  21. 21.
  22. 22. Personal Data Local Stores External Databases Articles Models Standards SOPs
  23. 23. Aggregated Commons Infrastructure Consistent,Comparative Reporting Design, protocols, samples, software, models….
  24. 24. Pop-Up Start Ups Little Science within Big Science
  25. 25. How do Scientists Collaborate & Cooperatively Exchange? Cautiously. Its all aboutTheTrust. Extrinsic Driver
  26. 26. How do you get Scientists and Developers to work together? Socially. Its all aboutTheTrust. Jam today, Jam tomorrow, Jam for all, Just enough Jam Just inTime not Just in Case.
  27. 27. Research Objects Compound Interconnected Investigations, Research Products Multi-various Products, Platforms/Resources Units of exchange, commons, contextual metadata
  28. 28. First class citizens - data, software, methods - id, manage, credit, track, profile, focus A Framework to Bundle and Link (scattered) resources, related experiments. Metadata Objects that carry Research Context Research Objects
  29. 29. Bigger on the inside than the outside Content • closed <-> open • local <-> alien • embed <-> refer • fixed <-> fluid • nested • cite? resolve? steward? Contributions • multi –typed, stewarded, sited, authored • span research, researchers, platforms, time • cite? resolve? steward?
  30. 30. Identity + Minimal Provenance RO Resolution and Citation: › Defend it (snapshot) › Locate it (most recent) › Reuse it (a version, a component) › Credit it (contributory authorship) › Cross link it (connections) Biological Study Records (e.g. PRIDE): stable Biological Knowledge (e.g. UNIPROT): evolving
  31. 31. Goble, De Roure, Bechhofer, Accelerating KnowledgeTurns, I3CK, 2013 means ends driver
  32. 32. Research Object packages codes, study, and metadata to exchange descriptions of clinical study cohorts, statistical scripts, data. Farr ResearchObject Commons STELARAsthma e-Lab: StudyTeam for Early Life Asthma Research Platform exchange: coded patient cohorts exchange with NHS FARSITE system STELAR e-Lab Platform 1 Platform 2 Platform 3 A multi-site collaboration to support safe use of patient and research data for medical research Research Object Currency Cohort Studies
  33. 33. Focus on methods, models, workflows, scripts, software, data, figures…. Research Object Pivots and Profiles
  34. 34. Focus on the figure: F1000Research Living Figures, versioned articles, in-article data manipulation R Lawrence Force2015, Vision Award Runner Up Simply data + code Can change the definition of a figure, and ultimately the journal article Colomb J and Brembs B. Sub-strains of Drosophila Canton-S differ markedly in their locomotor behavior [v1; ref status: indexed,] F1000Research 2014, 3:176 Other labs can replicate the study, or contribute their data to a meta- analysis or disease model - figure automatically updates. Data updates time-stamped. New conclusions added via versions.
  35. 35. Jennifer Schopf,Treating Data Like Software: A Case for Production Quality Data,JCDL 2012 Software-like Release paradigm Not a static document paradigm Reproduce looks backwards -> Release looks forwards » Science, methods, data change -> agile evolution » Comparisons , versions, forks & merges, dependencies » Id & Citations » Interlinked ROs
  36. 36. [McEntyre] Retrospective Release Research Object
  37. 37. The ROs Meme
  38. 38. recompute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle redo What IS reproducibility? Re: “do again”, “return to original state” “show A is true by doing B” verify but not falsify [Yong, Nature 485, 2012] robustness tolerance verificationcompliance validation assurance
  39. 39. RO as Instrument, Materials, Method Input Data Software Output Data Config Parameters Drummond, Replicability is not Reproducibility: Nor is it Good Science, online Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
  40. 40. 1. Science Changes. So does the Lab. “The questions don’t change but the answers do” Dan Reed The lab is not fixed Updated resources UncertaintyBioSTIF
  41. 41. Zhao, et al .Why workflows break - Understanding and combating decay in Taverna workflows, 8th Intl Conf e-Science 2012 2. Instruments Break, Labs Decay materials become unavailable, technicians leave Reproducibility Window » Bit rot, Black boxes » Proprietary Licenses » Clown services* » Partial replication » Prepare to Repair › form or function? › preserve or sustain? *Jason Scott
  42. 42. RO as Instrument, Materials, Method Input Data Software Output Data Config Parameters Methods (techniques, algorithms, spec. of the steps) Materials (datasets, parameters, algorithm seeds) Experiment Instruments (codes, services, scripts, underlying libraries) Laboratory (sw and hw infrastructure, systems software, integrative platforms) Setup Drummond, Replicability is not Reproducibility: Nor is it Good Science, online Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.
  43. 43. Research Environment submit article and move on… publish article Publication Environment
  44. 44. Research Environment publish article Publication Environment submit article and move on…
  45. 45. [Adapted Freire, 2013] transparency dependencies steps, features provenance trace portability robustness preservation access available description intelligible standards common APIs licensing standards common metadata change management versioning packaging Machine actionable Machine actionable Reproducibility Framework
  46. 46. submit article and move on… Reporting Documentation Provenance – ThickTrace Data to Distilled Reporting Distillation and Summarisation Alper P , et al LabelFlow: Exploiting Workflow Provenance to Surface Scientific Data Provenance. IPAW 2014: 84-96;
  47. 47. Reproduce by Reading Archived Record, Retain the Process/Code
  48. 48. The IT Crowd, Series 3, Episode 4 The eLabVirtual Machine* (or Docker Image**) * a black box though ** Reproduce by Running: Active Instrument Retain the bits
  49. 49. service Science as a Service Integrative frameworks Open Source Workflows/Scripts Virtual Machines Portable Packaging Portability Transparency
  50. 50. ReproZip Workflows,makefiles service Science as a Service Integrative frameworks Open Source Workflows/Scripts Virtual Machines Portable Packaging
  51. 51. Fifty Shades of Research Object Workflow Instrument Example data and config. Components. Plug-ins,Versions Workflow System Instrument Software package Workflow Runs Data and configs Provenance logs Study Shared Repository Personal Notebook Community Registry Publishing Resource
  52. 52. Fifty Shades of Research Object Workflow Instrument Example data and config. Components. Plug-ins,Versions Workflow System Instrument Software package Workflow Runs Data and configs Provenance logs Study
  53. 53. standards Adobe UCF ORE PROVODF formats api Instrument
  54. 54. Instrument
  55. 55. NISO-JATS Instrument J Zhao,G Klyne, M Gamble,CA Goble -A Checklist-Based Approach for QualityAssessment of Scientific Information Proceedings of theThird Linked Science Workshop 2013
  56. 56. Platform profiles NISO-JATS Instrument
  57. 57. Container Manifest OMEX archive Bergman et al COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project, BMC Bioinformatics 2014, 15:369 Retro-Fitted ROs using off the shelf platforms
  58. 58. Method Matters Reproducibility Smarts Commons not Repository ResearchTardis Retro-fit ROs Do As Little As Possible Make -> Born Native RO platforms RARE & FAIR KnowledgeTurns Means Research Objects
  59. 59. Researchers. Silver bullet tools. Psychic paper. PI Team RARE Research Reality Check!
  60. 60. RARE Research Reality Check!
  61. 61. Tribal Behaviour » Gangs share, but not with the public » Tribal behaviours › Modellers share more than Experimentalists › Experimentalists reuse models more than Modellers » Trading behaviours › Collaboration – complementarity correlations » Structured consortia less likely to publicly share than individuals » Post-hoc rationalised Data/Model Cycles [Garza, 2014]
  62. 62. » Fluid, transient collaborations > “my gang” management » Shameless exploitation of head teacher (PI) competitiveness & vanity » Class captains (prefects) » Get the cool kids on board. » Head teacher leadership [Garza, 2014] Playground Rules
  63. 63. Trace Data 27/03/2015 74
  64. 64. me ME my team close colleagues peers The Research Release Creep Spiral » Data Hugging & Flirting. » Reciprocity norms. » HansW request. » Dowry phenomenon. » Private installations. » Private spaces on shared installations. » Safe havens.
  65. 65. Too ugly to show anyone else. Readers who have access will want user support. No-one else would be interested/find it useful/be able to use it. The code is too sophisticated for most readers/referees. I didn't work out all the details. I didn't actually write the code -- my student did. My competitors would be unfair to me. Its valuable intellectual property. It would make papers much longer. Referees would never agree to check the code. My code invokes other code with unpublished (proprietary) code. Randall J. LeVeque ,TopTen ReasonsTo Not ShareYour Code (and why you should anyway) April 2013 SIAM News Victoria Stodden,AMP 2011,
  66. 66. Drivers love money fame duty fear time/ effort shame duty [Apologies to Resnick and Malone]
  67. 67. Stealthy not Sneaky reduce the friction instrumentation span RARE and FAIR OptimisingThe Neylon Equation
  68. 68. Interface Framing » Limited scheduled sharing choices › Never say never » “Citable” not “Shared” » Feedback › Guilt tripping › Outlier finger pointing [Garzia]
  69. 69. Auto-magical end-to-end Instrumentation ELNs and Authoring Platforms Sweave
  70. 70. Credit ≠ Authorship Research Currencies “ResearchBitCoin” Citation Semantics
  71. 71. Training 56% Of UK researchers develop their own research software or scripts 73% Of UK researchers have had no formal software engineering training Survey of researchers from 15 RussellGroup universities conducted by SSI between August - October 2014. 406 respondents covering representative range of funders, discipline and seniority.
  72. 72. Instrument Artisans [Shapin 84]
  73. 73. Make SoftwareVisible [1960s Boeing 747-100 Software Configuration] * Howison and Bullard 2014The visibility of software in the scientific literature: how do scientists mention software and how effective are those mentions? J Assoc fo Info Science andTechnology In review 87% software findable 78% credit 37% formal citation 5% actual version 90 Bio articles 24% journals had citation policy
  74. 74. BUT…… two years time when the paper is written reviewers want additional work statistician wants more runs analysis may need to be repeated post-doc leaves, student arrives new data, revised data updated versions of algorithms/codes sample was contaminated
  75. 75. Inspired by Bob Harrison • Incremental shift for infrastructure providers. • Moderate shift for policy makers and stewards. • Paradigm shift for researchers and their institutions. The RO & Reproducibility Challenge
  76. 76. All the members of the Wf4Ever team Colleagues in Manchester’s Information Management Group http://www.datafairport.orgAlanWilliams Jo McEntyre Norman Morrison Stian Soiland-Reyes Paul Groth Tim Clark Juliana Freire Alejandra Gonzalez-Beltran Philippe Rocca-Serra Ian Cottam Susanna Sansone Kristian Garza Barend Mons Sean Bechhofer Philip Bourne Matthew Gamble Raul Palma Jun Zhao Neil Chue Hong Josh Sommer Matthias Obst Jacky Snoep David Gavaghan Rebecca Lawrence
  77. 77. Contact… Professor Carole Goble The University of Manchester, UK @CaroleAnneGoble