Your SlideShare is downloading. ×
0
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend - Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C)

1,138

Published on

Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C) - Speaker at the marcus evans Evolution Summit 2012, held in Wheeling, IL, April 30 - May 2, 2012, delivered his …

Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C) - Speaker at the marcus evans Evolution Summit 2012, held in Wheeling, IL, April 30 - May 2, 2012, delivered his presentation on Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend

Published in: Business, Health & Medicine
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,138
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
26
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Lack Of Reproducibility In p yBiomedical Research –How A Common Quality Q yStandard For Non- Non-regulated Biomedical gResearch Could ChangeThe TrendÜlo Palm, MD, PhD, MBA, CMQ/OESenior Member of the American Society for Quality(ASQ)Food, Drugs & Cosmetics Division (FD&C) Marcus EvansSenior Vice President Clinical Operations & Biometrics p Evolution Summit,Forest Research Institute April 30, 2012
  • 2. Moore’s Law: Transistor Count Doubling Every Two YearsSource: Wikipedia
  • 3. Eroom’s Law: Number Of New Drug ApprovalsPer Billion US Dollars Halved Every Nine YearsSource: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March2012, page 191 - 200
  • 4. “War On Cancer” Going On For 40 Years War Cancer • About $200 billion spent on cancer research in the  US since 1971* US i 1971* • More than 1.5 million papers published* But despite some progress in prevention and  treatment • Almost 1.5 million cancer cases and 560 000 cancer  deaths in the United States in 2009** • Cancer now the second leading cause of death in  the US** • Nearly 1 in 2 men and more than 1 in 3 women will  y be diagnosed with cancer during their lifetime**  • Avastin increases survival in mCRC patients by 4.7  months*** at an annual treatment cost of   $58.000****Source: * Fortune Magazine March 22, 2004, ** JAMA, March 17, 2010 - Vol 303, *** Avastin label, **** Reuters, Thu Jun 30, 2011
  • 5. Some Causes Of DecliningProductivity Of Biomedical Research • Biomedical R&D has become more  complex* • Regulatory hurdles have gone up* Regulatory hurdles have gone up • Industrialization of drug discovery  (focus on single targets) has put  (focus on single targets) has put R&D on an overall less effective  path path* • Pharmaceutical companies have  become too big to innovate b t bi t i t *Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March 2012, page 191 - 200
  • 6. Could there be another, major, more fundamental root causeof the declining productivity of biomedical research?
  • 7. Lack Of Reproducibility In Biomedical Research Bi di l R h • “New Tools and technologies, massive amounts of  data, long‐term studies, interdisciplinary  d l di i di i li approaches, and the complexity of questions being  asked are complicating replication efforts…” • “An empirical assessment of 18 published papers of  “A ii l f 18 bli h d f microarray studies showed that independent  analysts could perfectly reproduce the results of  only two of the studies… only two of the studies ” • “This is one of medicines dirty  y secrets: Most results, including  those that appear in top‐flight  peer‐reviewed journals, cant be  reproduced”
  • 8. Lack Of Reproducibility: Industry ExperienceAmgen*• Published literature described that inhibition of  theserine/threonine kinase 33 (STK33) destroyed cancer cells• Amgen launched massive research effort but could not  replicate the results pBayer*• Bayer reported in September 2011 that it had halted nearly  two thirds of its early drug target projects because in house  two‐thirds of its early drug target projects because in‐house experiments failed to match claims made in the literature.Pfizer***• Pfizer announced January 2012 that it had to write off $750 Pfizer announced January 2012 that it had to write off $750  million after results of a study with Dimebon for Alzheimer,  published originally in the journal Lancet*,  could not be  reproduced  reproducedSource: *WSJ, December 2, 2011; ** Fierce Biotech, January 17, 2012
  • 9. Lack Of Reproducibility:Case St d 1 fC Study from A d Academia i • In 2002 paper published in Lancet by authors from  the FDA and NCI claimed that a mass spectrometry  method could provide highly sensitive and specific  diagnostic tests for ovarian cancer g • NCI announced a Clinical Proteomics initiative and  companies were formed to take assays based on this  method to the clinic method to the clinic • Independent analysis by MD Anderson researcher  demonstrated that the results were due to  experimental artifacts (running of all of the controls  before all of the cancers)Source: K i h A . BS Keith Baggerly and Kevin R . Coombes l dK i C bHandbook of Statistics in Clinical Oncology, Third EditionAntje Crowley and John HoeringChapman and Hall/CRC 2012 Pages 605–618
  • 10. Lack Of Reproducibility:Case St d 2 fC Study from A d Academia i• In 2006 a paper published in Nature Medicine by Duke  University professor Anil Potti claimed that microarray‐ based signatures of drug sensitivity derived from cell  lines could predict patient response to specific  chemotherapeutics• Discover magazine designated this paper one of the top  100 breakthroughs of 2006 g• Large clinical trials were started using this methodology• In 2009, independent analysis demonstrated that the  data were wrong due to mislabeling and indexing errors d t d t i l b li di d iSource: Keith A . Baggerly and Kevin R . Coombes Baggerly KA, Coombes KR.Handbook of Statistics in Clinical Oncology, Third Deriving chemosensiti it Deri ing chemosensitivity from cellEdition lines: Forensic bioinformatics andAntje Crowley and John Hoering reproducible research in high-Chapman and Hall/CRC 2012 Pages 605–618 throughput biology. Ann Appl Stat 2009; 3(4):1309–1334.
  • 11. Non-reproducible Research Is Noise And Not Knowledge• What if the recent revelations indicate that most of  published biomedical research is noise? – (“This is one of medicines dirty secrets: Most  results, including those that appear in top‐flight  peer‐reviewed journals, can t be reproduced peer‐reviewed journals cant be reproduced”  WSJ, December 2, 2011 )• What is the noise/knowledge ratio of the 22 million  / g records in Medline? 30%?, 50%?, ….??• How can biomedical research worldwide be  productive while trying to reproduce “noise”?  d ti hil t i t d “ i ”?• What if the amount of “noise” in modern  g g y biomedical research is beginning to suffocate any  productive research? 
  • 12. What are the reasons for the lack f l k of reproducibility in d ibilit imodern biomedical research?
  • 13. Modern Science Is Based On Collective I t lli C ll ti Intelligence• Collective intelligence requires:  – a shared body of knowledge, methods, and  y g techniques, a shared praxis*  – a shared and agreed upon quality standard  how to plan, conduct, and report scientific  work*Source: Reinventing Discovery – The new era of networked science by Michael Nielsen, 2012
  • 14. Modern Biomedical Research Does not Have A Well Defined Quality Standard• “Genes were mislabeled due to an off‐by‐one indexing  “G i l b l dd t ff b i d i error”• “We concluded that the method didn’t actually work at  all; it only appeared to work due to poor bookkeeping all; it only appeared to work due to poor bookkeeping”• “A disconnect between the numbers and the sample  names rendered the predictions invalid”• “poor documentation allowed errors to go unnoticed  p g until after things had proceeded to clinical trials “• “the most common mistakes people make are simple  ones” • “If the analyses are opaque, then the simple errors may  go unnoticed, and simple mistakes are still important”Source: Keith A . Baggerly and Kevin R . Coombes, Handbook of Statistics in ClinicalOncology, Third Edition, Antje Crowley and John HoeringChapman and Hall/CRC 2012 Pages 605–618
  • 15. In 2000, The WHO Identified The Development Of A Common Quality Standard For Biomedical Research As A Q y Pressing Global Need • “The world’s population is facing serious  health challenges….. there is increased  demand for new drugs and new principles for  treatment …..it is essential that basic scientific  (biomedical) research as a whole, ……. be  (b d l) h h l b conducted in a proper fashion using processes  that minimize waste of resources and reduce  the need for costly confirmation and  the need for costly confirmation and repetition of work already performed” • “It is hoped that wide application of the  “I i h d h id li i f h quality practices proposed in this handbook  will lead to cost‐effective, accelerated  discovery research and will ultimately benefit  discovery research and will ultimately benefit human  health”Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006
  • 16. The WHO Rationale For A Quality St d d I Biomedical Q lit Standard In Bi di l Research• To minimize waste of resources and  reduce the need for costly confirmation  and repetition of work already  d titi f k l d performed • To generate reliable data to ensure a To generate reliable data to ensure a  solid basis for deciding whether to invest  in further development of a strategy or  in further development of a strategy or product• Quality means better science Q y
  • 17. The Flow Of Research Activities From Planning To PublishingSource: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006
  • 18. In 2009, a committee of the FD&CDivision of the American Society forQuality (ASQ) was charged withdeveloping a Quality Standard forBiomedical Research and DrugDevelopment in the US“Best quality practices for biomedical research and  drug development” 1. ASQ Technical Report 2. ASQ Standard 3. ISO Standard 3 ISO St d d
  • 19. “Best Quality Practices For Biomedical ResearchAnd Drug Development Development”- Technical Report Content -• Management system g y • Document storageg• Organization • Technical Requirements• Project Management • Test Equipment• Quality Management  • Test methods / Method  System Validation• Documentation • Sampling and Chain of Custody Sampling and Chain of Custody• Document control /  • Materials pp Document approval and  • Legal and Ethical  Legal and Ethical issue: Considerations• Document changes • Vendor Selection and  Qualification
  • 20. The New Quality Standard for BiomedicalResearch and Drug Development Will Be Based esea c a d ug e e op e t e asedOn Current State-Of-The-Art Quality Practices State-Of-The-• World Health Organization Handbook: Quality Practices World Health Organization  Handbook: Quality Practices  for Biomedical Research• ISO 17025• BARQA Guidelines for Quality in Non‐Regulated Scientific  Research• ICH Q2 Validation of Analytical Procedures: Text and  Methodology• 21 CFR Part 58• ICH Q9 Quality Risk Management• ISO 900 ISO 900x
  • 21. CoCommittee Members ttee e be s• George Bernstein, MAI  g , • Juli Motika, Regeneron , g Consulting • Ülo Palm, Forest Research • Rick Calabrese, Sartorius‐ Institute Stedim St di • Michele Pruett, Innovative • Keith Conerly, Sodexho Consultants GXP• Li‐Chung Huang Eli Lilly Li Chung Huang, Eli Lilly • Sandra R Storli Abbott Labs Sandra R. Storli, Abbott Labs• Alice Krumenaker,  • John Surack, Clemson  CorePharma, LLC University• Richard Lombardi, Forest  • A. Mark Trotter, Trotter Biotech  Research Institute Solutions• J June Morita, Consultant M it C lt t
  • 22. Summary • Declining productivity of Drug Development Declining productivity of Drug Development • Poor reproducibility of Biomedical Research • Lack of a common language, a common quality Lack of a common language, a common quality  standard in Biomedical Research • Impacting medical progress and human health • WHO developed Handbook on Quality Practices  in Basic Biomedical Research • ASQ‐FD&C Division developing a new Quality  & d l l Standard for Biomedical Research & Drug  Development in the US Development in the US

×