EXL Clinical Quality Oversight Forum

1,239 views

Published on

Managing Risk and Ensuring Effective Oversight in Outsourced Clinical Trials

Published in: Health & Medicine, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,239
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

EXL Clinical Quality Oversight Forum

  1. 1. Managing Risk and Ensuring Effective Oversight in Outsourced Clinical Trials Clinical Quality Oversight Forum
  2. 2. 2 Panel Members Mitch Katz Executive Director Medical Research Operations, PURDUE PHARMA LP  David Marks Executive Vice President, Quality Management, RESEARCH PHARMACEUTICAL SERVICES, INC. (RPS)  David Nickerson Senior Director, Clinical Quality Management, PFIZER Clinical Quality Oversight Forum
  3. 3. 3 Today’s Topics •  Background: Challenges in quality management •  Part One: Proactive quality management and effective oversight •  Part Two: Risk sharing  •  Part Three: Risk assessment and risk management  Clinical Quality Oversight Forum
  4. 4. Background and Overview
  5. 5. 5 Mission: To accelerate the development of best practice approaches and industry standards for the proactive quality management of outsourced trials Lead Sponsors and CROs in optimizing their approaches to proactive quality management with an emphasis on bringing them into greater alignment. Avoca Quality Consortium
  6. 6. 6 Avoca Quality Consortium Pharma/Biotech Industry Participation to Date **Roche is also Consortium Members
  7. 7. 7 Avoca Quality Consortium CRO Industry Participation
  8. 8. 8 Are there quality issues with outsourced clinical trials? Challenges in Quality Management
  9. 9. 9 Challenges in Quality Management Disconnect between sponsors and CROs regarding perceptions of quality delivered by CROs Sponsor: Overall, how satisfied are you with the quality delivered by your Clinical Service Providers (including but not limited to CROs)? 2% 45% 30% 20% 3% 17% 70% 10% 3% CRO: Overall, how satisfied are you with the quality that your company delivers for its sponsors? N=245* N=200* 5% 13% 8% 8% 15% 15% 40% 10% 3% 11% 27% 95% 100% 67% 77% 83% 64% 58% 20% 60% 71% 57% 66% 60% 20% 15% 15% 20% 20% 20% 14% 23% 16% 10% 8% 6% 7% 20% 10% 14% 17% 7% 2% 0% 20% 40% 60% 80% 100% A B C D E F G H I J K -All- Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied *2013 Avoca Quality Consortium Assessment
  10. 10. 10 Overview: Challenges in Quality Management “Reactive” vs “proactive” approaches to quality management: a look at whether quality issues exist Areas with relatively high satisfaction: ●  Compliance with SOPs and other written procedures ●  Data quality and integrity ●  Audit plans and execution Areas with relatively high dissatisfaction: ●  Oversight of third party vendors ●  Governance of quality ●  Communications surrounding quality ●  Availability of quality personnel for projects ●  Efficiency/timeliness in achieving clean data ●  Adherence to monitoring plan 3.3 – 3.5 (on scale of 1 to 5) 2.9 – 3.2 (on scale of 1 to 5)
  11. 11. 11 Quality by Task: Comparison of Sponsor and CRO Results (2012 Avoca Quality Consortium Assessment) Sponsors: On average, how satisfied are you with your clinical service providers’… CROs: On average, how would you rate your company's… Compliance with SOPs and other written procedures 3.5 3.7 Data quality and integrity 3.4 3.8 Audit plans and execution 3.3 3.6 Adequacy of Monitoring Plan 3.2 3.5 Document control 3.2 3.3 Adherence to Monitoring Plan 3.2 3.4 Efficiency/timeliness in achieving clean data 3.2 3.5 Monitoring of protocol compliance 3.1 3.7 Availability of quality personnel for my projects 3.1 3.2 Management of protocol compliance 3.1 3.6 Staff training 3.0 3.3 Site training 3.0 3.4 Governance of quality (e.g. accountability, management system, leadership support) 2.9 3.4 Oversight of third party vendors (e.g. labs, IVRS vendors, etc.) 2.9 3.3 Communications surrounding quality 2.9 3.4
  12. 12. 12  Research Finding: Strong correlation between taking proactive approaches to quality management and satisfaction with the quality of deliverables. Sponsor Proactivity Drives Satisfaction 12 0   1   2   3   4   5   0   1   2   3   4   5   Sa*sfac*on  with  the  "quality"   delivered  by  clinical  service  providers   The  (sponsor)  teams  with  which  I  work  iden*fy  all     processes/deliverables  for  which  quality  expecta*ons  of  clinical  service   providers  need  to  be  set   0 1 2 3 4 5 0 1 2 3 4 5 Satisfactionwiththe"quality" deliveredbyclinicalservice providers The (sponsor) teams with which I work effectively communicate their expectations regarding quality to their clinical service providers
  13. 13. 13 Clarity of expectations: scores range between 2.4 on the low end to 4.8 on the high end (based on a scale of 1 to 5) Avoca Relationship Assessments Clarity of Roles and Expectations vs. Sponsor’s Overall Satisfaction: Across Strategic Partnerships* *p<.05 Statistically significant correlation across strategic partnerships: Clarity of expectations, roles, and responsibilities, and overall satisfaction with the work performed.
  14. 14. 14 Data suggests room for improvement in proactive approaches Many sponsors companies have reported difficulty in operationalizing approaches for identifying and communicating expectations. The (sponsor) teams with which I work effectively communicate their expectations regarding quality to their clinical service providers.
  15. 15. 15 Operationalizing Proactive Communication
  16. 16. 16 Operationalizing Proactive Communication
  17. 17. 17 Operationalizing Proactive Communication Construction Industry ü  Has operationalized proactive, systematic and integrated communication strategies Although buildings are more complex and sophisticated, building failure is exceedingly rare: (.00002 percent/year) “The biggest cause of serious failure in this business is a failure of communication.” Finbarr O’Sullivan, Project Executive, Moriarty & Associates, Project Executive, Russia Wharf office building on Boston Waterfront
  18. 18. 18 Leuchten’s Art Studio/Garage Subset of problems to date: ü  Foundation footings poured based on old set of drawings ü  Scheduling snafu’s between builder and roofer ü  Failed inspection due to issues with electrical and plumbing Root causes of issues: Problems with proactive communication, collaboration and clarity of expectations
  19. 19. 19 Themes to explore today •  Proactive communication •  Collaboration •  Clarity of expectations ...and impact on quality of outsourced clinical trials Clinical Quality Oversight Forum
  20. 20. Part One: Proactive quality management and effective oversight
  21. 21. 21 •  Research Topics –  Micromanagement –  Documentation around roles and responsibilities –  Overseeing work at a strategic level rather than tactical –  Definition of roles to minimize duplication of effort –  Efficiency in use of resources –  Use of technology to promote efficient oversight practices Proactive Quality Management and Effective Oversight `
  22. 22. 22 What do you think? Polling question for audience: Sponsors: My company is efficient in the use of resources applied to outsourced projects. CROs: Sponsors that I work with are efficient in the use of resources applied to outsourced projects. A.  Strongly agree B.  Agree C.  Neither agree nor disagree D.  Disagree E.  Strongly disagree
  23. 23. 23 21% 13% 5% 6% 6% 2% 30% 42% 34% 34% 28% 18% 34% 21% 34% 34% 29% 31% 11% 20% 18% 18% 32% 34% 4% 5% 8% 8% 6% 15% The operational teams involved in overseeing our CROs consistently micromanage to ensure that the quality of deliverables meets expectations. My company clearly documents CRO oversight practices, roles, and responsibilities. I personally have received high-quality deliverables from CROs, even when overseeing their work at a "strategic" rather than detailed level. My company clearly defines the roles of internal and CRO staff so as to minimize duplication of effort. My company is efficient in the use of resources applied to the oversight of outsourced trials. The technology systems used by my company and its CRO partners promote efficient oversight practices. Sponsor Perceptions of Oversight Practices Mean N 3.5 219 3.4 219 3.1 203 3.1 218 2.9 218 2.6 215 Perceptions of Oversight Practices Consortium Sponsor Findings 5 Strongly Agree 4 3 Neither Agree nor Disagree 2 1 Strongly Disagree
  24. 24. 24 8% 5% 3% 4% 3% 4% 47% 34% 30% 14% 23% 22% 32% 34% 33% 44% 35% 29% 11% 23% 25% 32% 31% 29% 3% 5% 10% 5% 8% 16% The sponsors with which I work consistently micromanage to ensure that the quality of deliverables meets expectations. The sponsors with which I work clearly document CRO oversight practices, roles, and responsibilities. The technology systems used by my company and its sponsor partners promote efficient oversight practices. The sponsors with which I work are efficient in the use of resources applied to the oversight of outsourced trials. The sponsors with which I work clearly define the roles of internal and CRO staff so as to minimize duplication of effort. There is a direct positive relationship between the intensity of sponsor oversight and the quality of CRO deliverables. CRO Perceptions of Sponsor Oversight Practices Mean N 3.5 157 3.1 153 2.9 159 2.8 152 2.8 149 2.7 158 Perceptions of Oversight Practices CRO Findings 5 Strongly Agree 4 3 Neither Agree nor Disagree 2 1 Strongly Disagree
  25. 25. 25 Statements Evaluated by Sponsors Sponsor Average CRO Average Statements Evaluated by CROs The operational teams involved in overseeing our CROs consistently micromanage to ensure that the quality of deliverables meets expectations.  3.5  3.5 The sponsors with which I work consistently micromanage to ensure that the quality of deliverables meets expectations. My company clearly documents CRO oversight practices, roles, and responsibilities.  3.4  3.1 The sponsors with which I work clearly document CRO oversight practices, roles, and responsibilities. My company clearly defines the roles of internal and CRO staff so as to minimize duplication of effort.  3.1  2.8 The sponsors with which I work clearly define the roles of internal and CRO staff to minimize duplication of effort. I personally have received high-quality deliverables from CROs, even when overseeing their work at a "strategic" rather than detailed level.  3.1  2.7 There is a direct positive relationship between the intensity of sponsor oversight and the quality of CRO deliverables. My company is efficient in the use of resources applied to the oversight of outsourced trials.  2.9 2.8  The sponsors with which I work are efficient in the use of resources applied to the oversight of outsourced trials. The technology systems used by my company and its CRO partners promote efficient oversight practices.  2.6 2.9 The technology systems used by my company and its sponsor partners promote efficient oversight practices. N≥203 N≥149 Perceptions of Oversight Practices 5 = Strongly Agree, 1 = Strongly Disagree Perceptions of Oversight Practices Sponsor and CRO Findings
  26. 26. 26 Best Practices in Quality Management Operationalizing approaches to setting and communicating expectations drives consistency and increases quality
  27. 27. 27 Operationalizing Proactive Approaches to Quality Management and Oversight Governance  /   Organiza*onal   Construct   Technical  /   Project   Oversight   Processes   Communica*on   Oversight   Leadership   Requirements   Metrics  /   Analy*cs  /   Technology   Roles  /   Responsibili*es   Proac*ve  Risk   Management  /   Con*ngencies   Effective Oversight Define   Plan   Lead   Decide   Measure   Deliver   Proactive Quality Management
  28. 28. 28 Verbatim Comments “[Sponsors don’t give us] ownership of decisions that we should have. This delays our deliverables which Sponsors will hold us accountable for in the end.” [CRO] “CRO's don't like to problem solve and offer suggestions. They'd rather just do as they are told.” [Sponsor] “One Sponsor that I work with is new to fully outsource trials, so at the beginning of our working relationship, they micromanaged us. This has slowly relaxed as we showed that we were consistently meeting project deliverables. “ [CRO]
  29. 29. Panel Discussion
  30. 30. Part Two: Risk-sharing
  31. 31. 31 Risk-sharing between Sponsors and Providers Types of Risk-sharing ●  Provider bonuses for achieving milestones and targets ●  Provider penalties for missing milestones and targets ●  Guarantees of continued work/revenue stream in exchange for provider commitments ●  Provider stake in outcome of study (e.g., company stock)
  32. 32. 32 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Provider penalties for failure to achieve milestones/targets Provider bonuses for achievement of milestones/targets Guarantee of continued work/revenue stream in exchange for provider commitments None of the above Strategic partnerships/alliances (N=205) Preferred provider relationships (N=194) Transactional relationships (N=217) Risk-Sharing Models Used by Type of Relationship Risk-Sharing Model Usage Sponsor Findings Risk-sharing models are used more commonly in strategic partnerships or preferred provider agreements than in transactional relationships.
  33. 33. 33 What is your experience? Polling question for audience: With which risk-sharing method have you had the most positive experience? A.  Provider bonuses for achieving milestones and targets B.  Provider penalties for missing milestones and targets C.  Guarantees of continued work/revenue stream in exchange for provider commitments D.  Provider stake in outcome of study (e.g., company stock)
  34. 34. 34 41% 30% 25% 50% 62% 21% 48% 61% 52% 47% 34% 55% 10% 9% 23% 3% 4% 24% Provider bonuses for achievement of milestones/targets Guarantee of continued work/revenue stream in exchange for provider commitments Provider penalties for failure to achieve milestones/targets Provider bonuses for achievement of milestones/targets Guarantee of continued work/revenue stream in exchange for provider commitments Provider penalties for failure to achieve milestones/targets Experience/Satisfaction with Risk-Sharing Models Used Primarily Positive A mix of Positive and Negative Primarily Negative Risk-Sharing Satisfaction Sponsor and CRO Findings N 87 94 104 92 74 95 SponsorsCROs Consor*um  sponsors  report  more  posi*ve  experiences  with  the  use  of  provider  bonuses   compared  to  provider  penal*es.  
  35. 35. 35 41% 30% 25% 50% 62% 21% 48% 61% 52% 47% 34% 55% 10% 9% 23% 3% 4% 24% Provider bonuses for achievement of milestones/targets Guarantee of continued work/revenue stream in exchange for provider commitments Provider penalties for failure to achieve milestones/targets Provider bonuses for achievement of milestones/targets Guarantee of continued work/revenue stream in exchange for provider commitments Provider penalties for failure to achieve milestones/targets Experience/Satisfaction with Risk-Sharing Models Used Primarily Positive A mix of Positive and Negative Primarily Negative Risk-Sharing Satisfaction Sponsor and CRO Findings N 87 94 104 92 74 95 SponsorsCROs Similar results are reported by sponsors and CROs in regard to use of provider penalties for failure to achieve milestones/targets.
  36. 36. Panel Discussion
  37. 37. Part Three: Risk Assessment and Risk Management
  38. 38. 38 37% 18% 20% 13% 12% Usage Frequency of Systematic Risk Assessment Processes Systematic Risk Assessment Frequency Sponsor and CRO Findings 51-75% of trials >75% of trials 25-50% of trials 1-24% of trials Never 52% 21% 13% 13% 1% Sponsors CROs N=174 N=135
  39. 39. 39 Use of Systematic Risk Assessment “Please briefly describe the conditions under which a systematic risk assessment process is used.” – Verbatim Responses ●  “Not consistent” ●  “Don’t know” ●  “Often done ‘in silos’” ●  Only… ╸ During RFP process ╸ At governance level ╸ Ad hoc ╸ At program level, not study level ╸ For audit strategy ╸ Upon sponsor request ╸ For key clients ╸ When resource limitations are a big concern ●  Done, but not necessarily “systematic and rigorous” ╸ Too high level and “standard” ╸ copy and paste from previous studies ╸ no time to think about study-specific nuances
  40. 40. 40 Risk Assessment and Risk Management “We do not consistently approach the work from a risk assessment perspective and we continually put ourselves in the position of being surprised by sponsor questions, demands or comments.” [CRO]
  41. 41. 41 Insights from Avoca Data 5 = Very Satisfied, 1 = Very Dissatisfied CONSORTIUM GROUP AVERAGES Performance Aspect In-house Teams CRO Partners CRO Self Assessment Proactive identification of potential risks 3.5 2.8 3.5 Proactive risk analysis and evaluation 3.3 2.7 3.4 Compilation of risk-related trial information during a trial (observations, trends, etc.) 3.2 2.6 3.2 Frequency of review of risk-related trial information 3.3 2.7 3.3 Rigor of review of risk-related trial information 3.1 2.5 3.1 Communications regarding risk-related trial information 3.3 2.8 3.2 Appropriateness of measures suggested or taken in reaction to risk-related information 3.4 3.0 3.3 Overall performance on risk assessment and management related activities 3.2 2.7 3.3 Low to middling scores suggest knowledge gaps in key areas related to risk identification, analysis and management Performance Satisfaction on Select Risk Assessment Variables
  42. 42. 42 Systematic Risk Assessment Results Sponsor and CRO Findings More efficient use of resources for your company and/or your partner? Increased quality? Q: In general, have your risk assessment and management approaches resulted in… N 204 159 203 159 9% 16% 20% 28% 39% 40% 36% 43% 17% 16% 8% 6% 35% 28% 36% 23% Sponsors CROs Sponsors CROs Yes Sometimes No Don’t Know / Too Soon to Tell
  43. 43. 43 Systematic Risk Assessment Results Sponsor and CRO Findings More efficient use of resources for your company and/or your partner? Increased quality? Q: In general, have your risk assessment and management approaches resulted in… N 204 159 203 159 9% 16% 20% 28% 39% 40% 36% 43% 17% 16% 8% 6% 35% 28% 36% 23% Sponsors CROs Sponsors CROs Yes Sometimes No Don’t Know / Too Soon to Tell
  44. 44. 44 What is your experience? Polling question for audience: How would you rate your understanding of best practices in risk assessment and management in clinical trials? A.  Very strong understanding B.  Good understanding C.  Fair understanding D.  Poor understanding
  45. 45. 45 Understanding of Best Practices in Risk Assessment/ Management 8% 39% 42% 11% Very strong understanding Good understanding Fair understanding Poor understanding How would you rate your understanding of best practices in risk assessment and management in clinical trials? 9% 49% 35% 7% Very strong understanding Good understanding Fair understanding Poor understanding N=207* N=159* Sponsor: CRO: *2013 Avoca Quality Consortium Assessment
  46. 46. 46 “Team members inability to understand the big picture of what their contribution means to the broader study and what the broader study means to the overall strategy for the product is a big contributor to failed risk management.” [Sponsor] “We don't proactively manage risk effectively. Instead we micromanage our CROs in the name of sponsor oversight, which is not a good use of anyone's time or resources.” [Sponsor] Verbatim Comments
  47. 47. Panel Discussion
  48. 48. Thank you! Patricia Leuchten patty.leuchten@theavocagroup.com Mitch Katz Mitchell.Katz@pharma.com David Marks DMarks@RPSweb.com Dave Nickerson David.F.Nickerson@pfizer.com

×