Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

What can be done to improve reproducibility?

Riffyn co-Founder Matthew Cockerill discusses the need for upstream changes to scientific methodology, moving it on from the artesan era, in order to make published results more reliable. (Talk presented at The Future of Scientific Scholarly Communication, a Royal Society symposium celebrating 350 years of the scientific journal, on 5th May 2015.)

  • Login to see the comments

  • Be the first to like this

What can be done to improve reproducibility?

  1. 1. Matthew Cockerill, Co-Founder, Riffyn Inc www.riffyn.com What can be done to improve reproducibility? The future of scientific scholarly communication, Royal Society, 5th May 2015
  2. 2. Reproducibility issues begin upstream of publication • Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability • Essential methodological knowledge is implicit and undocumented • To make things work “spend time in the relevant lab” • Technology transfer too often fails • If we want published results to be more reliable, we need to to change the approach to methodology
  3. 3. Reproducibility issues begin upstream of publication • Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability • Essential methodological knowledge is implicit and undocumented • To make things work “spend time in the relevant lab” • Technology transfer too often fails • If we want published results to be more reliable, we need to to change the approach to methodology
  4. 4. In industry, the costs of haziness are clear Certainly it is a problem that we don’t write down our processes - people change what they know, not what necessarily matters. - Head of Process Development of a biopharma company When we first transferred our manufacturing process to full-scale operations, our yields dropped 90%. - CEO of a diagnostics company We lost four months tracking-down a problem at the demo scale. Root cause was a parameter change that a new employee thought did not matter. - VP Process Development of an algae company Below: Process design sent to manufacturing
  5. 5. Industrial R&D seeks to improve processes Improve Design Analyse Measure
  6. 6. Industrial R&D seeks to improve processes Improve Design Analyse Measure Noise and irreproducibility gets in the way
  7. 7. Insights from manufacturing Quality-oriented approach Design • Unambiguous CAD files • Manage complexity via modularity Standardize & automate • Reduce variability in execution Continuous improvement • Use analytics to identify and resolve remaining causes of noise Change of mindset • Make metrics transparent to entire organization is focused on quality
  8. 8. Insights from manufacturing How can this benefit R&D? • It means you can build reliably on your own results, and those of others • But needs to be more flexible – in R&D processes change all the time! Quality-oriented approach Design • Unambiguous CAD files • Manage complexity via modularity Standardize & automate • Reduce variability in execution Continuous improvement • Use analytics to identify and resolve remaining causes of noise Change of mindset • Make metrics transparent so entire organization is focused on quality
  9. 9. Examples of application in real world R&D
  10. 10. Reducing noise allows more rapid progress Reproduced from Gardner TS, Trends in Biotechnology, March 2013, Vol. 31, No. 3 Data from product development at leading biotech firm Quality improvement cuts error by 6X. R&D productivity doubles overnight. Fermentationprocessyield Date trend lines 30% relative error 5% relative error
  11. 11. Analytics to track down root cause of noise Howitworks Define process inputs, outputs and critical variables1 Measure inputs and outputs2 StrainHTscreeningscore Tray # Analyse and improve3
  12. 12. How do we generalize to basic research?
  13. 13. New generation of lab software tools Make experiments more reproducible via: • Cloud-based software • CAD approach to experimental design • Integrated data-stream capture and analysis • Sharing and versioning of methodology via new publication outlets with github-like approaches
  14. 14. Parallel from cloud computing • Traditional ad hoc approach to configuring and managing IT systems unreliable and unscalable • Configuration management systems now use automated ‘recipes’ which reproducibly deliver a fully-configured virtual server which behaves the same every time
  15. 15. Reusable workflows are well established in computational science We need to move this to the lab bench Taverna Kepler Galaxy
  16. 16. New approaches are starting to bring similar automation to the lab bench
  17. 17. So what does the future look like?
  18. 18. Standards for structured, interoperable data Software and tools for lab automation/analytics Hubs supporting sharing of data and methodology An open ecosystem for reproducible science
  19. 19. How will we make it happen?
  20. 20. Incentives to drive adoption of new approach In industry • Measurable improvements to R&D productivity create powerful financial incentives In academia • Tools will gain adoption if they make researchers life easier and more productive • Endorsement and encouragement from funders • Requirement from participating journals for sharing of experimental process descriptions • Metrics that reward the sharing of reusable protocols
  21. 21. Cultural shift won’t be easy, but it is needed Change in the lab requires corresponding change in publishing

×