The Reproducibility Crisesin Biomedical Researchand its Impact onPharmaceutical R&DProductivityÜlo Palm, MD, PhD,Senior Vi...
Moore’s Law:    Transistor Count Doubling Every Two                   YearsSource: Wikipedia                     2
Eroom’s Law:Number Of New Drug Approvals Per Billion US Dollars Halved       Every Nine Years                             ...
“ War On Cancer” Going On For 40 Years                                             • About $200 billion spent on cancer re...
Some Causes Of Declining      Productivity Of Biomedical              Research• Biomedical R&D increasingly  complex*• Hig...
Is There Another Major, Root Cause Of TheDeclining Productivity Of Biomedical Research?                                   ...
Lack Of Reproducibility In Biomedical  Research – Literature And Press:“Five of the seven largest published studiesaddress...
Lack Of Reproducibility In Biomedical  Research – Literature And Press:                                    8
Lack Of Reproducibility In Biomedical  Research – Literature And Press:          November 2010                            ...
Lack Of Reproducibility In Biomedical  Research – Literature And Press:December 2011 “This is one of medicines dirty secre...
Lack Of Reproducibility In Biomedical  Research – Literature And Press:                         December 2011             ...
Lack Of Reproducibility In Biomedical  Research – Literature And Press:    2012                 “Forensic bioinformatics i...
13
Lancet 2002 • A new mass spectrometry method a highly   sensitive and specific diagnostic tests for ovarian   cancer • NCI...
Nature Medicine 2006• Microarray-based signatures of drug sensitivity  derived from cell lines predict patient response to...
A few “What If Questions”                            16
Impact On Industry                 ProductivityAmgen*•   Published literature described that inhibition of STK 33    destr...
What Are The Reasons ForThe Lack Of Reproducibility  In Modern Biomedical        Research?                              18
Modern Science Is Based On      Collective Intelligence• Collective intelligence requires:   – a shared body of knowledge,...
No Well             Defined Quality Standard Exists             In Biomedical Research• “Genes were mislabeled due to an o...
WHO: Development Of A Common Quality Standard   For Biomedical Research A Pressing Global Need                            ...
American Society For Quality (ASQ) : Best Quality  Practices For Biomedical Research In Drug  Development                 ...
A Vision For The Future Standardized and validated internal research                      End-to-end electronic           ...
Summary Of Issues• Declining productivity of Drug  Development• Poor reproducibility of Biomedical  Research• Lack of a co...
SolutionEstablish a common quality standard forbiomedical research in drug development•WHO and ASQ published quality guide...
Thank You!             26
Backup slides                27
The Flow Of Research Activities       From Planning To PublishingSource: Handbook: Quality Practices in Basic Biomedical R...
“ Best Quality Practices For Biomedical ResearchAnd Drug Development”- Technical Report Content -• Management system      ...
Upcoming SlideShare
Loading in …5
×

The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity

10,897 views

Published on

There has been a significant decline of productivity in pharmaceutical R&D
• Many different reasons have been discussed, but little attention has been paid to the role of poor reproducibility of biomedical research in general
• There are often fundamental doubts about the credibility of research data leading to costly and often futile repetition of biomedical research studies
• One of the root causes for the reproducibility problem is the lack of a common quality standard for biomedical research
• There have been recent initiatives by the WHO and the FD&C Division of the American Society for Quality (ASQ) to address this issue
• The pharmaceutical industry needs to embrace these initiatives in order to reverse the negative productivity trend

Published in: Health & Medicine
13 Comments
2 Likes
Statistics
Notes
  • Ulo and Cathy. In the language of my birthplace, we're singing from the same page in the hymnal. Regardless of the concern we have as scientists - and it is to be hoped that the concern becomes more widespread and more public - the drivers of today's problems are those who must also step forward: those who control funding. This will be particularly uncomfortable for the NIH that has emphasized 'translational research' only now to discover that the use of accepted procedures need revisiting. To add my voice to yours: reproducing data is only a start.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Ulo and Ray, I agree that reproducing data alone is not enough but it is a good start. It is also essential that all scientists and those reproducing data adhere to the type of quidelines that WHO and ASQ have suggested. The challenge as reflected in this discussion is to get scientists to act on these. We need those with responsibility for funding and publishing to actively 'encourage' this.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Ray, I agree, this is an interesting initiative. It shows how widespread the concern about the lack of reproducibility in biomedical research is. However, trying to validate research by replicating it is not sustainable. We need to address the root problem which is that individual research labs do not adhere to the same scientific and quality standards. There is a lack of a common language resulting in the inability of scientists to guarantee authenticity, integrity, and completeness of their data and results. That is why we need a broadly accepted quality standard for biomedical research. The WHO made the first step in the right direction with their 'Quality Practices In Basic Biomedical Research', the ASQ followed by publishing the 'Best Quality Practices For Biomedical Research in Drug Development'(I had the honor to be a member of the committee writing this document).
    To Ron Waife's point below, a quality guideline alone is not the solution. What counts is how it is implemented and adhered to. This is where scientists need to come together and demonstrate the discipline to adhere to common rules how to plan and conduct scientific experiments as well as how to capture and report scientific data and results in a consistent manner. Only then will we create the full transparency and traceability needed to build credibility and trust in biomedical research results. Unfortunately, many scientists in biomedical research believe that sticking to standards and rules stifles creativity. Nothing is further from the truth in my believe. This is as if a writer would not use the common English language out of fear that adhering to rules of grammar would limit his or her artistic creativity. The opposite is true, using a common language with defined rules is the only way for a writer to share his or her creative thoughts with others. For scientist this means that conducting research following a defined, common standard is the only way to share the results with others in a credible way.
    I view the WHO and ASQ quality guidelines as a first attempt to write a 'grammar book' for basic biomedical research. How the 'grammar' is then used is up to the creativity of the individual scientist.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi Everyone. I happened across this most interesting initiative. Apparently the problem of reproducibility is gaining a larger audience, among scientists if not among funding agencies.

    http://blog.scienceexchange.com/2013/03/reproducibility-initiative-updates-optins-and-validations/

    A quote or two from the site:

    In August 2012, we launched the Reproducibility Initiative as a partnership between PLOS, figshare, Mendeley, and Science Exchange- looking to identify and reward high quality reproducible research.

    Our outreach over the past couple months has indicated that thousands of researchers wished to participate in the Reproducibility Initiative, but did not have the funding to pay for the validations.

    Already, 1,892 researchers have opted in to have their studies reproduced. We believe that opting in itself provides a valuable signal in identifying reproducible findings, and we are now approaching public, charitable, and commercial funding bodies

    I find this promising inasmuch as the problem appears increasingly recognized and potential solutions are being offered.

    Ray
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • @CathyTralauStewart

    Right on Target (pun intended). Money talks but sometimes it says, 'Duh.'
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
10,897
On SlideShare
0
From Embeds
0
Number of Embeds
145
Actions
Shares
0
Downloads
184
Comments
13
Likes
2
Embeds 0
No embeds

No notes for slide
  • “ General studies of microarray gene-expression profiling have been undertaken to predict cancer outcome. Knowledge of this gene-expression profile or molecular signature should improve treatment of patients byallowing treatment to be tailored to the severity of the disease. Five
  • The stronger the shared praxis, the common language, the better collective intelligence works to solve the many scientific challenges of today
  • The stronger the shared praxis, the common language, the better collective intelligence works to solve the many scientific challenges of today The Human genome map is the collective output of many scientists and institution around the world* Modern particle physics is possible only through the power of collective intelligence*
  • The stronger the shared praxis, the common language, the better collective intelligence works to solve the many scientific challenges of today for example, getting labels transposed, disconnecting labels from data, and confounding the design. No malice of any type is required; the first two mistakes in particular are easy to make in Excel
  • The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity

    1. 1. The Reproducibility Crisesin Biomedical Researchand its Impact onPharmaceutical R&DProductivityÜlo Palm, MD, PhD,Senior Vice President Clinical Operations & BiometricsForest Research Institute February 11, 2013 1
    2. 2. Moore’s Law: Transistor Count Doubling Every Two YearsSource: Wikipedia 2
    3. 3. Eroom’s Law:Number Of New Drug Approvals Per Billion US Dollars Halved Every Nine Years 3Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March2012, page 191 - 200
    4. 4. “ War On Cancer” Going On For 40 Years • About $200 billion spent on cancer research in the US since 1971* • More than 1.5 million papers published* But despite some progress in prevention and treatment • Nearly 1 in 2 men and more than 1 in 3 women diagnosed with cancer during their lifetime** • Almost 1.5 million cancer cases and 560 000 cancer deaths in the United States in 2009** • Cancer now the second leading cause of death in the US** 4Source: * Fortune Magazine March 22, 2004, ** JAMA, March 17, 2010 - Vol 303,
    5. 5. Some Causes Of Declining Productivity Of Biomedical Research• Biomedical R&D increasingly complex*• Higher regulatory hurdles *• Pharmaceutical companies too big to innovate 5 *Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March 2012, page 191 - 200
    6. 6. Is There Another Major, Root Cause Of TheDeclining Productivity Of Biomedical Research? 6
    7. 7. Lack Of Reproducibility In Biomedical Research – Literature And Press:“Five of the seven largest published studiesaddressing cancer prognosis did not classifypatients better than chance” 7
    8. 8. Lack Of Reproducibility In Biomedical Research – Literature And Press: 8
    9. 9. Lack Of Reproducibility In Biomedical Research – Literature And Press: November 2010 9
    10. 10. Lack Of Reproducibility In Biomedical Research – Literature And Press:December 2011 “This is one of medicines dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, cant be reproduced” 10
    11. 11. Lack Of Reproducibility In Biomedical Research – Literature And Press: December 2011 “An empirical assessment of 18 published papers of microarray studies showed that independent analysts could perfectly reproduce the results of only two of the studies…” 11
    12. 12. Lack Of Reproducibility In Biomedical Research – Literature And Press: 2012 “Forensic bioinformatics is the art of using raw data and reported results to infer what must have happened to get from one to the other” 12
    13. 13. 13
    14. 14. Lancet 2002 • A new mass spectrometry method a highly sensitive and specific diagnostic tests for ovarian cancer • NCI launched clinical Proteomics initiative • Companies formed to take assays based on this method to the clinic • Turned out to be an experimental artifact due to running of all of the controls before all of the cancersSource: Keith A . Baggerly and Kevin R . Coombes Petricoin EF, Ardekani AM, Hitt BA, Levine PJ,Handbook of Statistics in Clinical Oncology, Third Fusaro VA, Steinberg SM, Mills GB et al. Use ofEdition proteomic patterns in serum to identify ovarianAntje Crowley and John Hoering 14 cancer. Lancet 2002; 359(9306):572–577Chapman and Hall/CRC 2012 Pages 605–618
    15. 15. Nature Medicine 2006• Microarray-based signatures of drug sensitivity derived from cell lines predict patient response to specific chemotherapeutics• Discover magazine designated this paper as one of the top 100 breakthroughs of 2006• Large clinical trials started using this methodology• In 2009, independent analysis demonstrated that the data were wrong due to mislabeling and indexing errorsSource:Keith A . Baggerly and Kevin R . Coombes Potti A, Dressman HK, Bild A, Riedel RF, ChanHandbook of Statistics in Clinical Oncology, Third G, Sayer R, Cragun J et al. Genomic signaturesEdition to guide the use of chemotherapeutics. Nat MedAntje Crowley and John Hoering 2006; 12:1294–1300. Retracted January 7,Chapman and Hall/CRC 2012 Pages 605–618 2011. 15
    16. 16. A few “What If Questions” 16
    17. 17. Impact On Industry ProductivityAmgen*• Published literature described that inhibition of STK 33 destroyed cancer cells• Amgen could not replicate the results despite massive research effortsBayer*• Bayer halted in 2011 nearly two-thirds of its early drug target projects because in-house experiments failed to match claims made in the literature.Pfizer***• Pfizer wrote off $750 million in 2012 after results of a study with Dimebon for Alzheimer, published originally in the journal Lancet*, could not be reproducedSource: *WSJ, December 2, 2011; ** Fierce Biotech, January 17, 2012 17
    18. 18. What Are The Reasons ForThe Lack Of Reproducibility In Modern Biomedical Research? 18
    19. 19. Modern Science Is Based On Collective Intelligence• Collective intelligence requires: – a shared body of knowledge, methods, and techniques, a shared praxis* – a shared and agreed upon quality standard how to plan, conduct, and report scientific work*Source: Reinventing Discovery – The new era of networked science by Michael Nielsen, 192012
    20. 20. No Well Defined Quality Standard Exists In Biomedical Research• “Genes were mislabeled due to an off-by-one indexing error”• “The method didn’t actually work at all; it only appeared to work due to poor bookkeeping”• “A disconnect between the numbers and the sample names rendered the predictions invalid”• “Poor documentation allowed errors to go unnoticed until after things had proceeded to clinical trials “Source: Keith A . Baggerly and Kevin R . Coombes, Handbook of Statistics in ClinicalOncology, Third Edition, Antje Crowley and John HoeringChapman and Hall/CRC 2012 Pages 605–618 20
    21. 21. WHO: Development Of A Common Quality Standard For Biomedical Research A Pressing Global Need • “The world’s population is facing serious health challenges… • .. there is increased demand for new drugs and new principles for treatment … • ..it is essential that basic scientific (biomedical) research , …is conducted in a proper fashion… using processes that minimize waste of resources • ….to reduce the need for costly repetition of work already performed”Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006 21
    22. 22. American Society For Quality (ASQ) : Best Quality Practices For Biomedical Research In Drug Development • The objective is to increase the overall productivity of biomedical research by promoting a common quality standard: • “A common quality guideline for biomedical research in drug development will ensure the validity and credibility of scientific data from different research institutions and facilitate the mutual acceptanceSource: http://asq.org/quality-press/display-item/index.html? of research results”item=T919E 22
    23. 23. A Vision For The Future Standardized and validated internal research End-to-end electronic •Integration Electronic •Analysis Submission •Modeling •Simulation Standardized and validated external research 23
    24. 24. Summary Of Issues• Declining productivity of Drug Development• Poor reproducibility of Biomedical Research• Lack of a common quality standard in Biomedical Research 24
    25. 25. SolutionEstablish a common quality standard forbiomedical research in drug development•WHO and ASQ published quality guidelines forBiomedical Research•Pharmaceutical Industry should promote theseguidelines to improve R&D productivity 25
    26. 26. Thank You! 26
    27. 27. Backup slides 27
    28. 28. The Flow Of Research Activities From Planning To PublishingSource: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006
    29. 29. “ Best Quality Practices For Biomedical ResearchAnd Drug Development”- Technical Report Content -• Management system • Document storage• Organization • Technical Requirements• Project Management • Test Equipment• Quality Management • Test methods / Method System Validation• Documentation • Sampling and Chain of Custody• Document control / • Materials Document approval and • Legal and Ethical issue: Considerations• Document changes • Vendor Selection and Qualification

    ×