1. Matthew Cockerill, Co-Founder, Riffyn Inc
www.riffyn.com
What can be done to improve reproducibility?
The future of scientific scholarly communication, Royal Society, 5th May 2015

2. Reproducibility issues begin upstream of publication
• Biomedical research still has more in common with a
“blacksmith shop” than a reliable modern process
engineered for reliability
• Essential methodological knowledge is implicit and
undocumented
• To make things work “spend time in the relevant lab”
• Technology transfer too often fails
• If we want published results to be more reliable, we
need to to change the approach to methodology

3. Reproducibility issues begin upstream of publication
• Biomedical research still has more in common with a
“blacksmith shop” than a reliable modern process
engineered for reliability
• Essential methodological knowledge is implicit and
undocumented
• To make things work “spend time in the relevant lab”
• Technology transfer too often fails
• If we want published results to be more reliable, we
need to to change the approach to methodology

4. In industry, the costs of haziness are clear
Certainly it is a problem that we don’t
write down our processes - people
change what they know, not what
necessarily matters.
- Head of Process Development of a
biopharma company
When we first transferred our
manufacturing process to full-scale
operations, our yields dropped 90%.
- CEO of a diagnostics company
We lost four months tracking-down a
problem at the demo scale. Root cause
was a parameter change that a new
employee thought did not matter.
- VP Process Development
of an algae company
Below: Process design sent to manufacturing

5. Industrial R&D seeks to improve processes
Improve
Design
Analyse
Measure

6. Industrial R&D seeks to improve processes
Improve
Design
Analyse
Measure
Noise and irreproducibility gets in the way

7. Insights from manufacturing
Quality-oriented approach
Design
• Unambiguous CAD files
• Manage complexity via modularity
Standardize & automate
• Reduce variability in execution
Continuous improvement
• Use analytics to identify and resolve
remaining causes of noise
Change of mindset
• Make metrics transparent to entire
organization is focused on quality

8. Insights from manufacturing
How can this benefit R&D?
• It means you can build reliably on your own results, and those of others
• But needs to be more flexible – in R&D processes change all the time!
Quality-oriented approach
Design
• Unambiguous CAD files
• Manage complexity via modularity
Standardize & automate
• Reduce variability in execution
Continuous improvement
• Use analytics to identify and resolve
remaining causes of noise
Change of mindset
• Make metrics transparent so entire
organization is focused on quality

9. Examples of application in real world R&D

10. Reducing noise allows more rapid progress
Reproduced from Gardner TS, Trends in Biotechnology, March 2013, Vol. 31, No. 3
Data from product development at leading biotech firm
Quality improvement cuts error by 6X. R&D productivity doubles overnight.
Fermentationprocessyield
Date
trend lines
30% relative error 5% relative error

11. Analytics to track down root cause of noise
Howitworks
Define process inputs, outputs and critical variables1
Measure inputs and outputs2
StrainHTscreeningscore
Tray #
Analyse and improve3

12. How do we generalize to basic research?

13. New generation of lab software tools
Make experiments more reproducible via:
• Cloud-based software
• CAD approach to experimental design
• Integrated data-stream capture and analysis
• Sharing and versioning of methodology via new
publication outlets with github-like approaches

14. Parallel from cloud computing
• Traditional ad hoc approach to configuring and managing IT
systems unreliable and unscalable
• Configuration management systems now use automated
‘recipes’ which reproducibly deliver a fully-configured virtual
server which behaves the same every time

15. Reusable workflows are well established in computational science
We need to move this to the lab bench
Taverna Kepler
Galaxy

16. New approaches are starting to bring similar
automation to the lab bench

17. So what does the future look like?

18. Standards for structured, interoperable data
Software and tools for lab automation/analytics
Hubs supporting sharing of data and methodology
An open ecosystem for reproducible science

19. How will we make it happen?

20. Incentives to drive adoption of new approach
In industry
• Measurable improvements to R&D productivity create
powerful financial incentives
In academia
• Tools will gain adoption if they make researchers life easier
and more productive
• Endorsement and encouragement from funders
• Requirement from participating journals for sharing of
experimental process descriptions
• Metrics that reward the sharing of reusable protocols

21. Cultural shift won’t be easy, but it is needed
Change in the lab requires corresponding change in publishing