Your SlideShare is downloading. ×
  • Like
  • Save
Quality Metrics
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Quality Metrics

  • 1,925 views
Published

Performance metrics are an essential element of the management review process. …

Performance metrics are an essential element of the management review process.
Quality metrics may include elements such as customer satisfaction, supplier performance, manufacturing defects, complaints, cycle times and many other internal or external processes.
This presentation provides a framework for establishing right quality indicators for evaluating the performance of the quality system.

Published in Health & Medicine , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Please forward this PPT as download to me at abhayzbansal@gmail.com. I am a fresh Pharmacy Graduate,
    This PPT will be useful for me as a guidance for future reference.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,925
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
1
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. This presentation is compiled by “ Drug Regulations” a non profit organization which provides free online resource to the Pharmaceutical Professional. Visit http://www.drugregulations.org for latest information from the world of Pharmaceuticals. 1/4/2014 1
  • 2. This presentation is compiled from freely available resources like the websites of FDA, EMA, WHO.  “Drug Regulations” is a non profit organization which provides free online resource to the Pharmaceutical Professional.  Visit http://www.drugregulations.org for latest information from the world of Pharmaceuticals.  Drug Regulations : Online Resource for Latest Information 1/4/2014 2
  • 3. ◦ Performance metrics are an essential element of the management review process. ◦ Quality metrics may include elements such as customer satisfaction, supplier performance, manufacturing defects, complaints, cycle times and many other internal or external processes. ◦ This presentation provides a framework for establishing right quality indicators for evaluating the performance of the quality system. Drug Regulations : Online Resource for Latest Information 1/4/2014 3
  • 4.  Accountability ◦ Metrics indicate how well an operation or project is performing  Alignment ◦ A core objective for a strategy focused organization is to align the entire team on a common set of strategic objectives. Drug Regulations : Online Resource for Latest Information 1/4/2014 4
  • 5.  Attention ◦ Metrics incorporated into management review provide a defined set of expectations of senior management and are reviewed regularly  Behaviour ◦ Setting the right metrics drives the right behaviour. Drug Regulations : Online Resource for Latest Information 1/4/2014 5
  • 6.  Expectations ◦ Metrics allow an entire organization a common understanding of what outcomes are most valued by leadership.  Feedback ◦ Metrics provide short and long term feedback on the effectiveness of projects and activities. Drug Regulations : Online Resource for Latest Information 1/4/2014 6
  • 7.  Motivation ◦ People like knowing their efforts are worthwhile. Metrics provide short and long term measurement of progress.  Objectivity ◦ Metrics form the basis for a fact-based organization Drug Regulations : Online Resource for Latest Information 1/4/2014 7
  • 8.  Problem Solving ◦ Metrics enable management to uncover problems earlier  Visibility ◦ A common set of metrics used throughout an organization creates transparency. Drug Regulations : Online Resource for Latest Information 1/4/2014 8
  • 9. ◦ There should be a standardized meaning of the measurement ◦ The data should help the decision making process ◦ It should provide worthwhile information ◦ It should be easy to install ◦ It can be benchmarked or used elsewhere Drug Regulations : Online Resource for Latest Information 1/4/2014 9
  • 10. ◦ Selecting metrics that take into account each of these factors is a challenge ◦ Easy metrics to collect offer little value ◦ More worthwhile information requires effort to establish a measurement process Drug Regulations : Online Resource for Latest Information 1/4/2014 10
  • 11. ◦ Many metrics are possible to fully characterize a quality system. ◦ Large number of metrics wastes resources and distracts leaders. ◦ Implementation of too many metrics may not result in a compliant quality system. ◦ Metrics must be carefully chosen to ensure they align with overall strategy. Drug Regulations : Online Resource for Latest Information 1/4/2014 11
  • 12. ◦ Good metrics system consists of leading and lagging indicators. ◦ Lagging indicators measure the outcomes of what already happened. ◦ Leading indicators provide information that may be able to predict future outcomes. ◦ More mature organizations will typically have a greater ratio of leading to lagging metrics. Drug Regulations : Online Resource for Latest Information 1/4/2014 12
  • 13. ◦ Most compliance metrics are inherently lagging indicators and are typically easier to define. ◦ Establishing predictive leading indicators can improve a quality system. ◦ Begin developing metrics with lagging indicators ◦ Then ask what actions or conditions cause this outcome to brainstorm leading indicators ◦ It is necessary to understand the cause and effect relationships. ◦ Possible to focus on impacting drivers which will ultimately change the outcomes. Drug Regulations : Online Resource for Latest Information 1/4/2014 13
  • 14. ◦ Important to consider whether the metric measures time, quality or quantity/cost. ◦ Time-based metrics are used to measure the duration of a process.  Metrics such as the percentage of CAPAs completed on-time is an example. ◦ Quantity/cost metrics track the frequency or productivity of the process.  In a balanced scorecard these could include the cost of the process. ◦ Using financial metrics for compliance is not recommended. Drug Regulations : Online Resource for Latest Information 1/4/2014 14
  • 15. ◦ Quality metrics should  Focus on effectiveness  Measure that the right things are being done correctly. ◦ Often the focus is on efficiency measured in financial terms. ◦ Effectiveness first eliminates waste & improves efficiency. ◦ Quality metrics are difficult to establish ◦ Provide excellent insight into the health of the quality system. Drug Regulations : Online Resource for Latest Information 1/4/2014 15
  • 16. ◦ Include representatives from all departments to design data system successfully ◦ Align performance measurements to overall company strategic plan  Quality objectives will support one or more elements of a strategic plan. ◦ Involve individuals from all levels ◦ Have a dialog between levels and across organization  Increase alignment,  Focus the organization  Develop buy-in Drug Regulations : Online Resource for Latest Information 1/4/2014 16
  • 17. ◦ Document the metrics definitions. ◦ Helps in collection of consistent metrics across organization over time. Drug Regulations : Online Resource for Latest Information 1/4/2014 17
  • 18. Definition One sentence explaining the Metric Per cent of CAPA completed within agreed time Benefit Quality objective to be achieved Are non conformances resolved in a timely manner ? Formula Specific formula to calculate the parameter Target Give desirable goal 90 % , 10 % improvement over last year ,….. Frequency Give desired frequency Monthly , Quarterly , Yearly ,…… Drug Regulations : Online Resource for Latest Information 1/4/2014 18
  • 19. Status Rules Colors could be used for performance Red ≤ 70 % , Yellow = 71% to 89% , Green =90% to 100% Data Source Indicate where the data is availble CAPA system RACI Responsible: person(s) influencing the metric performance; Accountable: person(s) ultimately responsible for the metric performance. Consulted: person(s) with knowledge in improving the metric performance. Informed: person(s) likely to benefit from the updates on the metric performance. Responsible: auditors; Accountable: Senior Audit Manager, Consulted: none; Informed Quality council, Quality Director. Drug Regulations : Online Resource for Latest Information 1/4/2014 19
  • 20. ◦ Set targets and goals in alignment with strategy ◦ Have realistic expectations ◦ Collect baseline data for several reporting periods ◦ Establish current performance level with baseline data ◦ Establish improvement targets ◦ Measure performance improvement against the baseline ◦ Targets =“SMART-A”  Specific, Measureable, Attainable, Realistic, Time-bound and Aligned. Drug Regulations : Online Resource for Latest Information 1/4/2014 20
  • 21.  Evolution of a Measure ◦ % of acceptable computer reports delivered on time ◦ Number of acceptable computer reports delivered on time Effectiveness Efficiency ◦ Number of computer reports delivered on time ◦ Number of computer reports delivered Drug Regulations : Online Resource for Latest Information Extent 1/4/2014 21
  • 22. ◦ Ensure that metric is performing as intended. ◦ Change the metric definition with caution as some changes may invalidate older measurements. ◦ Need for a metric may change over time. ◦ Remove the metrics if it does not add value Drug Regulations : Online Resource for Latest Information 1/4/2014 22
  • 23. ◦ Establish Quality performance metrics based on linkages to organizational objectives. ◦ Link to product or service performance, quality costs, quality improvement and customer satisfaction. ◦ Share the Quality goals across the organization. Drug Regulations : Online Resource for Latest Information 1/4/2014 23
  • 24. ◦ Quality metrics may be collected and reported by the quality function ◦ Ownership of the issue belongs to operating function ◦ Pareto principle applies to quality problems ◦ 80% of the quality problems are outside the control of the quality function, while 20% may be internal. Drug Regulations : Online Resource for Latest Information 1/4/2014 24
  • 25. ◦ Organization wide problems could include  Supplier quality requirements  R&D product designs  Distribution system problems ◦ Internal quality problems may include  Lab test issues  Auditing procedures  Sampling procedures Drug Regulations : Online Resource for Latest Information 1/4/2014 25
  • 26. ◦ Performance against regulatory agency inspections  Basic indicator of compliance  Commercialization of products  Manufacturing authorization. Drug Regulations : Online Resource for Latest Information 1/4/2014 26
  • 27. ◦ Internal Audit by an independent Auditor  Snapshot of performance .  On-going feedback on the health of the quality system.  Not an assurance of health of the quality system. ◦ Management Review to evaluate complete Quality System Drug Regulations : Online Resource for Latest Information 1/4/2014 27
  • 28. ◦ Favourable outcomes help confirm compliance and conformance. ◦ Unfavourable outcomes  significant business interruptions  Business loss  Impact public perceptions and  Decrease the value of a company‟s brand and image Drug Regulations : Online Resource for Latest Information 1/4/2014 28
  • 29. ◦ Performance metrics against external inspections are reactive measures. ◦ Do not rely on Regulatory inspection as a single indicator. ◦ Conduct Internal audits more regularly  Will enable a proactive response to avoid an external finding.  Functional area self-audits offer an excellent opportunity for most proactive measurement. Drug Regulations : Online Resource for Latest Information 1/4/2014 29
  • 30. Outcome Measure % of inspections with no citations Method Trend Other Considerations Analyse citations according their nature and details Develop Pareto analysis of citations Evaluate whether there are relative differences between inspecting authorities % of audits with no adverse findings Trend Evaluate difference between Internal & External performance. Evaluate difference between sites , departments Frequency and severity of findings Trend Evaluate difference between Internal & External performance. Evaluate difference between sites , departments Drug Regulations : Online Resource for Latest Information 1/4/2014 30
  • 31. Process Measure % of inspections with Timeliness of Method Trend internal audits against schedule Other Considerations Evaluate whether there are relative differences between departments or auditors Indicators that audits covered Trend all quality systems, sites and Evaluate differences between auditors or departments functions Number/per cent of open Trend Probe for reasons for open items. Trend Probe reasons for delay inspection and audit actions Timeliness of closure of inspection and audit actions Evaluate whether there are relative differences between departments or Auditors Drug Regulations : Online Resource for Latest Information 1/4/2014 31
  • 32. ◦ Analyse the findings according their nature and details. ◦ Track with regulation citation number being violated. ◦ Develop a pareto of finding by citations. ◦ Focus of global improvement efforts should be the highest frequencies of findings ◦ Compare & calibrate internal audit group to external authorities Drug Regulations : Online Resource for Latest Information 1/4/2014 32
  • 33. ◦ Refine audit program if internal audits do not identify similar trends in findings as external inspections ◦ Perform external benchmarking. ◦ Compare with studies showing pareto analysis of finding by agencies such as the FDA. ◦ Carefully examine trends of external findings frequency. Drug Regulations : Online Resource for Latest Information 1/4/2014 33
  • 34. ◦ Increase emphasis on emerging issues ◦ Adjust the internal audit plan to include covering emerging issues ◦ Prepare organization for the frequent inspection areas. Drug Regulations : Online Resource for Latest Information 1/4/2014 34
  • 35. ◦ Crucial to develop appropriate metrics. ◦ Important window to compliance and the health of the quality system. ◦ Frequently a focus area for external inspectors. ◦ CAPA‟s not addressed thoroughly & promptly raise red flag during inspection  Points to unresolved problems  Indicates unresponsive management Drug Regulations : Online Resource for Latest Information 1/4/2014 35
  • 36. Outcome Measure % Investigations which discover root Method Trend cause Other Considerations Confirm the level of reoccurrences, root cause and effectiveness and that % of Nonconformance recurrence Trend there are not any adverse trends or patterns (current value against target, trending). Consider whether there are any products, processes or departments with a disproportionate number of issues. Drug Regulations : Online Resource for Latest Information 1/4/2014 36
  • 37. Process Measure % Investigations completed within Method Trend defined timelines Other Considerations Evaluate any investigations that are open for extended periods of time. Probe the reasons for the delay % of actions completed within Trend defined timelines Evaluate actions that are open for extended period of time. Probe the reason for delay Frequency of Non conformances Trend Evaluate the location of the event, cause, product, process step Drug Regulations : Online Resource for Latest Information 1/4/2014 37
  • 38. ◦ Analyse additional CAPA system information. ◦ Categorize the nature and details of the nonconformities. Categorization by  Product family  Part number  Type of process  Equipment used ◦ Understand the cause and resolution of problems.  Equipment failure  Human error  Supplier Drug Regulations : Online Resource for Latest Information 1/4/2014 38
  • 39. ◦ Evaluate problems attributed to human error ◦ Indicates failure to identify true root cause ◦ Monitor non-conformances & CAPA that are classified as high risk ◦ Give priority for completion and confirmation ◦ When “non conforming” products are disposed off “as is” make the decision visible to management. ◦ Give higher priority to CAPA for recalls, and complaints. Drug Regulations : Online Resource for Latest Information 1/4/2014 39
  • 40. ◦ Systems critical to the product safety & patient. ◦ Regulators Evaluate these systems very closely ◦ Complaint and Product Action process measures determine the timeliness of key process steps ◦ Complaints open for a long period of time are a red flag  Points to a bigger problem  May also indicate lack of resources or focus ◦ Adverse events have defined reporting time frame Drug Regulations : Online Resource for Latest Information 1/4/2014 40
  • 41. Outcome Measure Method Other Considerations Frequency of complaints Trend Evaluate by product and product confirmed as a non-conformance Pareto family Frequency of product recalls Trend Impact on other batches 0r products Frequency of adverse events Trend Evaluate by product and product family Drug Regulations : Online Resource for Latest Information 1/4/2014 41
  • 42. Process Measure Method Frequency of complaints by Trend product Pareto Frequency of complaints by Trend nature of complaint Pareto % of complaints closed within Trend Other Considerations defined timeframe Regulatory notification filing Evaluate any complaints that are open for extended periods of time Trend timeliness Drug Regulations : Online Resource for Latest Information 1/4/2014 42
  • 43. ◦ Complexity of supply chain has increased ◦ Significant importance for supplier quality performance ◦ Rigorous Quality system essential for Supplier ◦ Suppliers may include any one or more  Providers of direct manufacturing materials including commodities,  Starting materials,  Chemicals,  Excipients,  API (active pharmaceutical ingredients),  Drug product,  Finished dosage and  Third party manufacturers of finished products. Drug Regulations : Online Resource for Latest Information 1/4/2014 43
  • 44. ◦ A separate metrics may used for  CRO for testing  CRO for clinical studies  Pest control service  Calibration services  IT services Drug Regulations : Online Resource for Latest Information 1/4/2014 44
  • 45. ◦ Part of a company‟s quality system ◦ Act within their own quality system ◦ Monitor both perspectives with metrics ◦ Internal and external non-conformance are examples of supplier driven issues ◦ Look at a supplier as a whole quality system ◦ A scorecard could be developed for critical suppliers Drug Regulations : Online Resource for Latest Information 1/4/2014 45
  • 46. ◦ Scorecard  Audit outcomes  Lot failure rates  Delivery performance and  Responsiveness ◦ Prepare a composite metric for each supplier ◦ Prepare a composite metric from all critical supplier scorecards Drug Regulations : Online Resource for Latest Information 1/4/2014 46
  • 47. Outcome Measure Method Frequency of non-conformances Trend due to supplier Pareto Frequency of Incoming testing Trend failures Pareto Frequency of In-process failures due Trend to supplier Other Considerations Pareto Process Measure Method Timeliness of internal supplier Evaluate overall and by supplier Evaluate overall and by supplier Other Considerations Trend audits against schedule Evaluate overall and by supplier Pareto Drug Regulations : Online Resource for Latest Information 1/4/2014 47
  • 48.  Analytical Lab metrics ◦ Frequency of OOS (out of specification)  Calibration metrics ◦ Frequency of calibration OOS (out of specification) ◦ Calibration schedule conformance  Cleanroom metrics ◦ Frequency of cleanroom OOS Drug Regulations : Online Resource for Latest Information 1/4/2014 48
  • 49. Dilutes the effort. 1/4/2014 49
  • 50. The goal becomes to manipulate the number 1/4/2014 50
  • 51. Any improvement will not improve overall performance. 1/4/2014 51
  • 52. Exasperates involved people 1/4/2014 52
  • 53.  In July 2012, the Food and Drug Administration Safety and Innovation Act (FDASIA) was enacted.  This act requires the formation of a task force to develop and implement a strategic plan for enhancing the Agency's response to preventing and mitigating drug shortages. Drug Regulations : Online Resource for Latest Information 1/4/2014 53
  • 54.  The Task Force is specifically interested in seeking public input on amongst others :  “In an effort to address the major underlying causes of drug and biological product shortages, FDA is seeking new manufacturing ideas and to to encourage facilitate high-quality expansion of manufacturing capacity”  To assist in the evaluation of product manufacturing quality, FDA is exploring the broader use of manufacturing quality metrics. Drug Regulations : Online Resource for Latest Information 1/4/2014 54
  • 55. ◦ FDA is now required to do risk based inspections ◦ Several risk factors are to be considered in the evaluation (next slide) ◦ FDA can collect records otherwise available on inspection “in advance or in lieu of inspection” ◦ Quality Metrics are considered potentially the sort of information that could be provided in advance. Drug Regulations : Online Resource for Latest Information 1/4/2014 55
  • 56. A. The compliance history of the establishment. B. The record, history, and nature of recalls linked to the establishment. C. The inherent risk of the drug manufactured, prepared, propagated, compounded, or processed at the establishment. D. The inspection frequency and history of the establishment, including whether the establishment has been inspected pursuant to section 704 within the last 4 years. E. Whether the establishment has been inspected by a foreign government or an agency of a foreign government recognized under section 809. F. Any other criteria deemed necessary and appropriate by the Secretary for purposes of allocating inspection resources. Drug Regulations : Online Resource for Latest Information 1/4/2014 56
  • 57.  Implications of Providing Quality Metrics : ◦ Identify lower risk products, processes & sites ◦ Regulatory flexibility of lower reporting  Prior approval supplement to an annual report ◦ Identify high quality site needing lower inspection ◦ Pathway to more objective inspections Drug Regulations : Online Resource for Latest Information 1/4/2014 57
  • 58. ◦ To assist in the evaluation of product manufacturing quality, FDA is exploring the broader use of manufacturing quality metrics. ◦ With that in mind, FDA would like input on the following issues:  What metrics do manufacturers currently use to monitor production quality?  To what extent do purchasers and prescribers use information about manufacturing quality when deciding how to purchase or utilize products?  What kinds of manufacturing quality metrics might be valuable for purchasers and prescribers when determining which manufacturers to purchase from or which manufacturers' products to prescribe?  What kinds of manufacturing quality metrics might be valuable for manufacturers when choosing a contract manufacturer?  How frequently would such metrics need to be updated to be meaningful? Drug Regulations : Online Resource for Latest Information 1/4/2014 58
  • 59. ◦ Available under 704 w/ same frequency ◦ Meaningful relative to 705 and 706 ◦ Viable across all industry sectors (brand, biotech, generics), and objective ◦ Individually be minimally intrusive, leveraging existing industry data ◦ Collectively be balanced (limit unintended consequences), understandable to lay person, and linked to patient needs (i.e. product meets specifications) ◦ Individually relevant to product or unit operation, leveraging new or existing ◦ Industry data, and focused on specifications or state of control – Collectively, provides predictive capability to future product quality issues Drug Regulations : Online Resource for Latest Information 1/4/2014 59
  • 60.  Keep it simple  Improve objectivity  Potential for regulatory flexibility  Build consensus  Promote right behavior Drug Regulations : Online Resource for Latest Information 1/4/2014 60
  • 61. ◦ Batch Failure Rate ◦ Right First Time ◦ Out of Specification/ Laboratory Failure Investigation rates ◦ Yield Drug Regulations : Online Resource for Latest Information 1/4/2014 61
  • 62. Metric FDA System Leading / Lagging Rationale Batch Rejection Production, Lagging Already collected by about 95%* of Rate Packaging & companies in some form. Labelling A measure of production quality or packaging and labelling quality. Rework and Reprocessing Rate Production Lagging These are not normally reported as a stand-alone quality metric, but may be captured as a financial metric or as part of Annual Product Review. This metric is proposed as a corollary to the Batch Rejection Rate to provide a fuller picture of the production capability. Drug Regulations : Online Resource for Latest Information 1/4/2014 62
  • 63. Metric FDA System Leading / Lagging Rationale Confirmed out of Production Lagging Number of OOS is tracked by most Specification (OOS) companies*. Would expect that Rate batches that are OOS would end up being included in „Batch Rejection Rate‟ metric. Useful sub-division of Batch Rejection Rate. A measure of production quality. Unconfirmed OOS Rate Laboratory Lagging A measure of laboratory performance and potentially has a link to the quality system Drug Regulations : Online Resource for Latest Information 1/4/2014 63
  • 64. Metric FDA System Leading / Lagging Rationale “Critical” Quality Lagging Strong link to product quality and Complaints quality system performance, and importantly to the patient. May overlap with Field Alerts or other Health Authority reporting. This metric could be considered amongst the first to be analyzed as product-based % Annual Product Quality Lagging High and consistent values are a Quality Reviews leading indicator of a quality Completed on system performance. Product- time based metric with strong links to quality system performance including resource allocation. 1/4/2014 64
  • 65.  This presentation was compiled from freely available resources like the websites of FDA, EMA, WHO.  “Drug Regulations” is a non profit organization which provides free online resource to the Pharmaceutical Professional.  Visit http://www.drugregulations.org for latest information from the world of Pharmaceuticals. Drug Regulations : Online Resource for Latest Information 1/4/2014 65