Quality Metrics

10,908 views

Published on

Performance metrics are an essential element of the management review process.
Quality metrics may include elements such as customer satisfaction, supplier performance, manufacturing defects, complaints, cycle times and many other internal or external processes.
This presentation provides a framework for establishing right quality indicators for evaluating the performance of the quality system.

Published in: Health & Medicine, Business
1 Comment
25 Likes
Statistics
Notes
  • Please forward this PPT as download to me at abhayzbansal@gmail.com. I am a fresh Pharmacy Graduate,
    This PPT will be useful for me as a guidance for future reference.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
10,908
On SlideShare
0
From Embeds
0
Number of Embeds
74
Actions
Shares
0
Downloads
1
Comments
1
Likes
25
Embeds 0
No embeds

No notes for slide

Quality Metrics

  1. 1. This presentation is compiled by “ Drug Regulations” a non profit organization which provides free online resource to the Pharmaceutical Professional. Visit http://www.drugregulations.org for latest information from the world of Pharmaceuticals. 11/28/2015 1
  2. 2.  This presentation is compiled from freely available resources like the websites of FDA, EMA, WHO.  “Drug Regulations” is a non profit organization which provides free online resource to the Pharmaceutical Professional.  Visit http://www.drugregulations.org for latest information from the world of Pharmaceuticals. 11/28/2015 2 Drug Regulations : Online Resource for Latest Information
  3. 3.  This presentation has been updated based on the FDA Draft Guidance on Quality Metrics released in July 2015  Visit http://www.drugregulations.org for latest information from the world of Pharmaceuticals. 11/28/2015 3 Drug Regulations : Online Resource for Latest Information
  4. 4. ◦ Performance metrics are an essential element of the management review process. ◦ Quality metrics may include elements such as customer satisfaction, supplier performance, manufacturing defects, complaints, cycle times and many other internal or external processes. ◦ This presentation provides a framework for establishing right quality indicators for evaluating the performance of the quality system. 11/28/2015 4 Drug Regulations : Online Resource for Latest Information
  5. 5.  Accountability ◦ Metrics indicate how well an operation or project is performing  Alignment ◦ A core objective for a strategy focused organization is to align the entire team on a common set of strategic objectives. 11/28/2015 5 Drug Regulations : Online Resource for Latest Information
  6. 6.  Attention ◦ Metrics incorporated into management review provide a defined set of expectations of senior management and are reviewed regularly  Behaviour ◦ Setting the right metrics drives the right behaviour. 11/28/2015 6 Drug Regulations : Online Resource for Latest Information
  7. 7.  Expectations ◦ Metrics allow an entire organization a common understanding of what outcomes are most valued by leadership.  Feedback ◦ Metrics provide short and long term feedback on the effectiveness of projects and activities. 11/28/2015 7 Drug Regulations : Online Resource for Latest Information
  8. 8.  Motivation ◦ People like knowing their efforts are worthwhile. Metrics provide short and long term measurement of progress.  Objectivity ◦ Metrics form the basis for a fact-based organization 11/28/2015 8 Drug Regulations : Online Resource for Latest Information
  9. 9.  Problem Solving ◦ Metrics enable management to uncover problems earlier  Visibility ◦ A common set of metrics used throughout an organization creates transparency. 11/28/2015 9 Drug Regulations : Online Resource for Latest Information
  10. 10. ◦ There should be a standardized meaning of the measurement ◦ The data should help the decision making process ◦ It should provide worthwhile information ◦ It should be easy to install ◦ It can be benchmarked or used elsewhere 11/28/2015 10 Drug Regulations : Online Resource for Latest Information
  11. 11. ◦ Selecting metrics that take into account each of these factors is a challenge ◦ Easy metrics to collect offer little value ◦ More worthwhile information requires effort to establish a measurement process 11/28/2015 11 Drug Regulations : Online Resource for Latest Information
  12. 12. ◦ Many metrics are possible to fully characterize a quality system. ◦ Large number of metrics wastes resources and distracts leaders. ◦ Implementation of too many metrics may not result in a compliant quality system. ◦ Metrics must be carefully chosen to ensure they align with overall strategy. 11/28/2015 12 Drug Regulations : Online Resource for Latest Information
  13. 13. ◦ Good metrics system consists of leading and lagging indicators. ◦ Lagging indicators measure the outcomes of what already happened. ◦ Leading indicators provide information that may be able to predict future outcomes. ◦ More mature organizations will typically have a greater ratio of leading to lagging metrics. 11/28/2015 13 Drug Regulations : Online Resource for Latest Information
  14. 14. ◦ Most compliance metrics are inherently lagging indicators and are typically easier to define. ◦ Establishing predictive leading indicators can improve a quality system. ◦ Begin developing metrics with lagging indicators ◦ Then ask what actions or conditions cause this outcome to brainstorm leading indicators ◦ It is necessary to understand the cause and effect relationships. ◦ Possible to focus on impacting drivers which will ultimately change the outcomes. 11/28/2015 14 Drug Regulations : Online Resource for Latest Information
  15. 15. ◦ Important to consider whether the metric measures time, quality or quantity/cost. ◦ Time-based metrics are used to measure the duration of a process.  Metrics such as the percentage of CAPAs completed on-time is an example. ◦ Quantity/cost metrics track the frequency or productivity of the process.  In a balanced scorecard these could include the cost of the process. ◦ Using financial metrics for compliance is not recommended. 11/28/2015 15 Drug Regulations : Online Resource for Latest Information
  16. 16. ◦ Quality metrics should  Focus on effectiveness  Measure that the right things are being done correctly. ◦ Often the focus is on efficiency measured in financial terms. ◦ Effectiveness first eliminates waste & improves efficiency. ◦ Quality metrics are difficult to establish ◦ Provide excellent insight into the health of the quality system. 11/28/2015 16 Drug Regulations : Online Resource for Latest Information
  17. 17. ◦ Include representatives from all departments to design data system successfully ◦ Align performance measurements to overall company strategic plan  Quality objectives will support one or more elements of a strategic plan. ◦ Involve individuals from all levels ◦ Have a dialog between levels and across organization  Increase alignment,  Focus the organization  Develop buy-in 11/28/2015 17 Drug Regulations : Online Resource for Latest Information
  18. 18. ◦ Document the metrics definitions. ◦ Helps in collection of consistent metrics across organization over time. 11/28/2015 18 Drug Regulations : Online Resource for Latest Information
  19. 19. Definition One sentence explaining the Metric Per cent of CAPA completed within agreed time Benefit Quality objective to be achieved Are non conformances resolved in a timely manner ? Formula Specific formula to calculate the parameter Target Give desirable goal 90 % , 10 % improvement over last year ,….. Frequency Give desired frequency Monthly , Quarterly , Yearly ,…… 11/28/2015 19 Drug Regulations : Online Resource for Latest Information
  20. 20. Status Rules Colors could be used for performance Red ≤ 70 % , Yellow = 71% to 89% , Green =90% to 100% Data Source Indicate where the data is availble CAPA system RACI Responsible: person(s) influencing the metric performance; Accountable: person(s) ultimately responsible for the metric performance. Consulted: person(s) with knowledge in improving the metric performance. Informed: person(s) likely to benefit from the updates on the metric performance. Responsible: auditors; Accountable: Senior Audit Manager, Consulted: none; Informed Quality council, Quality Director. 11/28/2015 20 Drug Regulations : Online Resource for Latest Information
  21. 21. ◦ Set targets and goals in alignment with strategy ◦ Have realistic expectations ◦ Collect baseline data for several reporting periods ◦ Establish current performance level with baseline data ◦ Establish improvement targets ◦ Measure performance improvement against the baseline ◦ Targets =“SMART-A”  Specific, Measureable, Attainable, Realistic, Time-bound and Aligned. 11/28/2015 21 Drug Regulations : Online Resource for Latest Information
  22. 22.  Evolution of a Measure ◦ % of acceptable computer reports delivered on time ◦ Number of acceptable computer reports delivered on time ◦ Number of computer reports delivered on time ◦ Number of computer reports delivered 11/28/2015 22 Drug Regulations : Online Resource for Latest Information Extent Efficiency Effectiveness
  23. 23. ◦ Ensure that metric is performing as intended. ◦ Change the metric definition with caution as some changes may invalidate older measurements. ◦ Need for a metric may change over time. ◦ Remove the metrics if it does not add value 11/28/2015 23 Drug Regulations : Online Resource for Latest Information
  24. 24. ◦ Establish Quality performance metrics based on linkages to organizational objectives. ◦ Link to product or service performance, quality costs, quality improvement and customer satisfaction. ◦ Share the Quality goals across the organization. 11/28/2015 24 Drug Regulations : Online Resource for Latest Information
  25. 25. ◦ Quality metrics may be collected and reported by the quality function ◦ Ownership of the issue belongs to operating function ◦ Pareto principle applies to quality problems ◦ 80% of the quality problems are outside the control of the quality function, while 20% may be internal. 11/28/2015 25 Drug Regulations : Online Resource for Latest Information
  26. 26. ◦ Organization wide problems could include  Supplier quality requirements  R&D product designs  Distribution system problems ◦ Internal quality problems may include  Lab test issues  Auditing procedures  Sampling procedures 11/28/2015 26 Drug Regulations : Online Resource for Latest Information
  27. 27. ◦ Performance against regulatory agency inspections  Basic indicator of compliance  Commercialization of products  Manufacturing authorization. 11/28/2015 27 Drug Regulations : Online Resource for Latest Information
  28. 28. ◦ Internal Audit by an independent Auditor  Snapshot of performance .  On-going feedback on the health of the quality system.  Not an assurance of health of the quality system. ◦ Management Review to evaluate complete Quality System 11/28/2015 28 Drug Regulations : Online Resource for Latest Information
  29. 29. ◦ Favourable outcomes help confirm compliance and conformance. ◦ Unfavourable outcomes  significant business interruptions  Business loss  Impact public perceptions and  Decrease the value of a company’s brand and image 11/28/2015 29 Drug Regulations : Online Resource for Latest Information
  30. 30. ◦ Performance metrics against external inspections are reactive measures. ◦ Do not rely on Regulatory inspection as a single indicator. ◦ Conduct Internal audits more regularly  Will enable a proactive response to avoid an external finding.  Functional area self-audits offer an excellent opportunity for most proactive measurement. 11/28/2015 30 Drug Regulations : Online Resource for Latest Information
  31. 31. Outcome Measure Method Other Considerations % of inspections with no citations Trend Analyse citations according their nature and details Develop Pareto analysis of citations Evaluate whether there are relative differences between inspecting authorities % of audits with no adverse findings Trend Evaluate difference between Internal & External performance. Evaluate difference between sites , departments Frequency and severity of findings Trend Evaluate difference between Internal & External performance. Evaluate difference between sites , departments 11/28/2015 31 Drug Regulations : Online Resource for Latest Information
  32. 32. Process Measure Method Other Considerations % of inspections with Timeliness of internal audits against schedule Trend Evaluate whether there are relative differences between departments or auditors Indicators that audits covered all quality systems, sites and functions Trend Evaluate differences between auditors or departments Number/per cent of open inspection and audit actions Trend Probe for reasons for open items. Timeliness of closure of inspection and audit actions Trend Probe reasons for delay Evaluate whether there are relative differences between departments or Auditors 11/28/2015 32 Drug Regulations : Online Resource for Latest Information
  33. 33. ◦ Analyse the findings according their nature and details. ◦ Track with regulation citation number being violated. ◦ Develop a pareto of finding by citations. ◦ Focus of global improvement efforts should be the highest frequencies of findings ◦ Compare & calibrate internal audit group to external authorities 11/28/2015 33 Drug Regulations : Online Resource for Latest Information
  34. 34. ◦ Refine audit program if internal audits do not identify similar trends in findings as external inspections ◦ Perform external benchmarking. ◦ Compare with studies showing pareto analysis of finding by agencies such as the FDA. ◦ Carefully examine trends of external findings frequency. 11/28/2015 34 Drug Regulations : Online Resource for Latest Information
  35. 35. ◦ Increase emphasis on emerging issues ◦ Adjust the internal audit plan to include covering emerging issues ◦ Prepare organization for the frequent inspection areas. 11/28/2015 35 Drug Regulations : Online Resource for Latest Information
  36. 36. ◦ Crucial to develop appropriate metrics. ◦ Important window to compliance and the health of the quality system. ◦ Frequently a focus area for external inspectors. ◦ CAPA’s not addressed thoroughly & promptly raise red flag during inspection  Points to unresolved problems  Indicates unresponsive management 11/28/2015 36 Drug Regulations : Online Resource for Latest Information
  37. 37. Outcome Measure Method Other Considerations % Investigations which discover root cause Trend Confirm the level of reoccurrences, root cause and effectiveness and that there are not any adverse trends or patterns (current value against target, trending). Consider whether there are any products, processes or departments with a disproportionate number of issues. % of Nonconformance recurrence Trend 11/28/2015 37 Drug Regulations : Online Resource for Latest Information
  38. 38. Process Measure Method Other Considerations % Investigations completed within defined timelines Trend Evaluate any investigations that are open for extended periods of time. Probe the reasons for the delay % of actions completed within defined timelines Trend Evaluate actions that are open for extended period of time. Probe the reason for delay Frequency of Non conformances Trend Evaluate the location of the event, cause, product, process step 11/28/2015 38 Drug Regulations : Online Resource for Latest Information
  39. 39. ◦ Analyse additional CAPA system information. ◦ Categorize the nature and details of the nonconformities. Categorization by  Product family  Part number  Type of process  Equipment used ◦ Understand the cause and resolution of problems.  Equipment failure  Human error  Supplier 11/28/2015 39 Drug Regulations : Online Resource for Latest Information
  40. 40. ◦ Evaluate problems attributed to human error ◦ Indicates failure to identify true root cause ◦ Monitor non-conformances & CAPA that are classified as high risk ◦ Give priority for completion and confirmation ◦ When “non conforming” products are disposed off “as is” make the decision visible to management. ◦ Give higher priority to CAPA for recalls, and complaints. 11/28/2015 40 Drug Regulations : Online Resource for Latest Information
  41. 41. ◦ Systems critical to the product safety & patient. ◦ Regulators Evaluate these systems very closely ◦ Complaint and Product Action process measures determine the timeliness of key process steps ◦ Complaints open for a long period of time are a red flag  Points to a bigger problem  May also indicate lack of resources or focus ◦ Adverse events have defined reporting time frame 11/28/2015 41 Drug Regulations : Online Resource for Latest Information
  42. 42. Outcome Measure Method Other Considerations Frequency of complaints confirmed as a non-conformance Trend Pareto Evaluate by product and product family Frequency of product recalls Trend Impact on other batches 0r products Frequency of adverse events Trend Evaluate by product and product family 11/28/2015 42 Drug Regulations : Online Resource for Latest Information
  43. 43. Process Measure Method Other Considerations Frequency of complaints by product Trend Pareto Frequency of complaints by nature of complaint Trend Pareto % of complaints closed within defined timeframe Trend Evaluate any complaints that are open for extended periods of time Regulatory notification filing timeliness Trend 11/28/2015 43 Drug Regulations : Online Resource for Latest Information
  44. 44. ◦ Complexity of supply chain has increased ◦ Significant importance for supplier quality performance ◦ Rigorous Quality system essential for Supplier ◦ Suppliers may include any one or more  Providers of direct manufacturing materials including commodities,  Starting materials,  Chemicals,  Excipients,  API (active pharmaceutical ingredients),  Drug product,  Finished dosage and  Third party manufacturers of finished products. 11/28/2015 44 Drug Regulations : Online Resource for Latest Information
  45. 45. ◦ A separate metrics may used for  CRO for testing  CRO for clinical studies  Pest control service  Calibration services  IT services 11/28/2015 45 Drug Regulations : Online Resource for Latest Information
  46. 46. ◦ Part of a company’s quality system ◦ Act within their own quality system ◦ Monitor both perspectives with metrics ◦ Internal and external non-conformance are examples of supplier driven issues ◦ Look at a supplier as a whole quality system ◦ A scorecard could be developed for critical suppliers 11/28/2015 46 Drug Regulations : Online Resource for Latest Information
  47. 47. ◦ Scorecard  Audit outcomes  Lot failure rates  Delivery performance and  Responsiveness ◦ Prepare a composite metric for each supplier ◦ Prepare a composite metric from all critical supplier scorecards 11/28/2015 47 Drug Regulations : Online Resource for Latest Information
  48. 48. Outcome Measure Method Other Considerations Frequency of non-conformances due to supplier Trend Pareto Evaluate overall and by supplier Frequency of Incoming testing failures Trend Pareto Evaluate overall and by supplier Frequency of In-process failures due to supplier Trend Pareto Evaluate overall and by supplier 11/28/2015 48 Drug Regulations : Online Resource for Latest Information Process Measure Method Other Considerations Timeliness of internal supplier audits against schedule Trend Pareto
  49. 49. 11/28/2015 49 Drug Regulations : Online Resource for Latest Information  Analytical Lab metrics ◦ Frequency of OOS (out of specification)  Calibration metrics ◦ Frequency of calibration OOS (out of specification) ◦ Calibration schedule conformance  Cleanroom metrics ◦ Frequency of cleanroom OOS
  50. 50. 11/28/2015 50 Dilutes the effort.
  51. 51. 11/28/2015 51 The goal becomes to manipulate the number
  52. 52. 11/28/2015 52 Any improvement will not improve overall performance.
  53. 53. 11/28/2015 53 Exasperates involved people
  54. 54.  In July 2012, the Food and Drug Administration Safety and Innovation Act (FDASIA) was enacted.  This act requires the formation of a task force to develop and implement a strategic plan for enhancing the Agency's response to preventing and mitigating drug shortages. 11/28/2015 54 Drug Regulations : Online Resource for Latest Information
  55. 55.  The Task Force is specifically interested in seeking public input on amongst others :  “In an effort to address the major underlying causes of drug and biological product shortages, FDA is seeking new ideas to encourage high-quality manufacturing and to facilitate expansion of manufacturing capacity”  To assist in the evaluation of product manufacturing quality, FDA is exploring the broader use of manufacturing quality metrics. 11/28/2015 55 Drug Regulations : Online Resource for Latest Information
  56. 56. ◦ FDA is now required to do risk based inspections ◦ Several risk factors are to be considered in the evaluation (next slide) ◦ FDA can collect records otherwise available on inspection “in advance or in lieu of inspection” ◦ Quality Metrics are considered potentially the sort of information that could be provided in advance. 11/28/2015 56 Drug Regulations : Online Resource for Latest Information
  57. 57.  FDA plans to use its authority to collect records "in advance of or in lieu of" an inspection, under section 704(a)(4)(A) of the FD&C Act to gather various quality metrics data records.  The agency says it will use these records to "further develop [its] risk-based inspection scheduling." 11/28/2015 57 Drug Regulations : Online Resource for Latest Information
  58. 58. A. The compliance history of the establishment. B. The record, history, and nature of recalls linked to the establishment. C. The inherent risk of the drug manufactured, prepared, propagated, compounded, or processed at the establishment. D. The inspection frequency and history of the establishment, including whether the establishment has been inspected pursuant to section 704 within the last 4 years. E. Whether the establishment has been inspected by a foreign government or an agency of a foreign government recognized under section 809. F. Any other criteria deemed necessary and appropriate by the Secretary for purposes of allocating inspection resources. 11/28/2015 58 Drug Regulations : Online Resource for Latest Information
  59. 59.  Implications of Providing Quality Metrics : ◦ Identify lower risk products, processes & sites ◦ Regulatory flexibility of lower reporting  Prior approval supplement to an annual report ◦ Identify high quality site needing lower inspection ◦ Pathway to more objective inspections 11/28/2015 59 Drug Regulations : Online Resource for Latest Information
  60. 60. ◦ To assist in the evaluation of product manufacturing quality, FDA is exploring the broader use of manufacturing quality metrics. ◦ With that in mind, FDA would like input on the following issues:  What metrics do manufacturers currently use to monitor production quality?  To what extent do purchasers and prescribers use information about manufacturing quality when deciding how to purchase or utilize products?  What kinds of manufacturing quality metrics might be valuable for purchasers and prescribers when determining which manufacturers to purchase from or which manufacturers' products to prescribe?  What kinds of manufacturing quality metrics might be valuable for manufacturers when choosing a contract manufacturer?  How frequently would such metrics need to be updated to be meaningful? 11/28/2015 60 Drug Regulations : Online Resource for Latest Information
  61. 61.  The FDA released a draft guidance detailing how it plans to use quality metrics in July 2015  Based on this FDA proposes to ◦ Improve its ability to conduct risk-based inspections and ◦ Predict or mitigate potential drug shortages. 11/28/2015 61 Drug Regulations : Online Resource for Latest Information
  62. 62.  New draft guidance Request for Quality Metrics  FDA is giving more details to industry on its approach to quality metrics,  FDA is also asking industry for its input in specific areas that are still being considered. 11/28/2015 62 Drug Regulations : Online Resource for Latest Information
  63. 63.  For companies with robust quality metrics data, this information could be used to "reduce the inspection frequency at an establishment."  In cases where the data raises issues, FDA says having the quality metrics data will allow it to "improve the efficiency and effectiveness" of the inspections.  For now, the agency foresees its use of quality metrics to be used primarily for reducing inspection frequencies. 11/28/2015 63 Drug Regulations : Online Resource for Latest Information
  64. 64.  FDA says it will request the following 10 baseline quality metrics from companies as part of its analysis: ◦ The number of lots attempted of the product. ◦ The number of specification-related rejected lots of the product, rejected during or after manufacturing ◦ The number of attempted lots pending disposition for more than 30 days ◦ The number of out-of-specification (OOS) results for the product, including stability testing. ◦ The number of lot release and stability tests conducted for the product. 11/28/2015 64 Drug Regulations : Online Resource for Latest Information
  65. 65.  FDA says it will request the following 10 baseline quality metrics from companies as part of its analysis: ◦ The number of product quality complaints received for the product. ◦ The number of lots attempted which are released for distribution or for the next stage of manufacturing the product. ◦ If the associated annual product reviews (APRs) or product quality reviews (PQRs) were completed within 30 days of annual due date for the product ◦ The number of APRs or PQRs required for the product 11/28/2015 65 Drug Regulations : Online Resource for Latest Information
  66. 66.  The agency is also asking for input on five "additional, optional metrics as evidence of manufacturing robustness and a commitment to quality." ◦ Senior Management Engagement—Was each APR or PQR reviewed and approved by the following: (1) The head of the quality unit, (2) the head of the operations unit, (3) both, or (4) neither? ◦ Corrective Action and Preventive Action (CAPA) Effectiveness—What percentage of your corrective actions involved re-training of personnel (i.e., a root cause of the deviation is lack of adequate training)? ◦ Process Capability/Performance—A “yes” or “no” value of whether the establishment's management calculated a process capability or performance index for each critical quality attribute as part of that product's APR or PQR. 11/28/2015 66 Drug Regulations : Online Resource for Latest Information
  67. 67.  The agency is also asking for input on five "additional, optional metrics as evidence of manufacturing robustness and a commitment to quality.” ◦ Process Capability/Performance—A “yes” or “no” value of whether the establishment's management has a policy of requiring a CAPA at some lower process capability or performance index. ◦ Process Capability/Performance—If “yes” to the previous question—What is the process capability or performance index that triggers a CAPA? If “no” to the previous question—please do not respond. 11/28/2015 67 Drug Regulations : Online Resource for Latest Information
  68. 68.  Keep it simple  Improve objectivity  Potential for regulatory flexibility  Build consensus  Promote right behavior 11/28/2015 68 Drug Regulations : Online Resource for Latest Information
  69. 69.  This presentation was compiled from freely available resources like the websites of FDA, EMA, WHO.  “Drug Regulations” is a non profit organization which provides free online resource to the Pharmaceutical Professional.  Visit http://www.drugregulations.org for latest information from the world of Pharmaceuticals. 11/28/2015 69 Drug Regulations : Online Resource for Latest Information

×