Quality Metrics

This presentation is compiled by “ Drug Regulations” a
non profit organization which provides free online
resource to the Pharmaceutical Professional.
Visit http://www.drugregulations.org for latest
information from the world of Pharmaceuticals.
11/28/2015 1

 This presentation is compiled from freely
available resources like the websites of FDA,
EMA, WHO.
 “Drug Regulations” is a non profit
organization which provides free online
 Visit http://www.drugregulations.org for
latest information from the world of
Pharmaceuticals.
11/28/2015 2
Drug Regulations : Online
Resource for Latest Information

 This presentation has been updated based on
the FDA Draft Guidance on Quality Metrics
released in July 2015
Pharmaceuticals.
11/28/2015 3

◦ Performance metrics are an essential element of the
management review process.
◦ Quality metrics may include elements such as customer
satisfaction, supplier performance, manufacturing defects,
complaints, cycle times and many other internal or external
processes.
◦ This presentation provides a framework for establishing
right quality indicators for evaluating the performance of
the quality system.
11/28/2015 4
Drug Regulations : Online Resource for Latest
Information

 Accountability
◦ Metrics indicate how well an operation or project is
performing
 Alignment
◦ A core objective for a strategy focused organization
is to align the entire team on a common set of
strategic objectives.
11/28/2015 5
Information

 Attention
◦ Metrics incorporated into management review
provide a defined set of expectations of senior
management and are reviewed regularly
 Behaviour
◦ Setting the right metrics drives the right behaviour.
11/28/2015 6
Information

 Expectations
◦ Metrics allow an entire organization a common
understanding of what outcomes are most valued
by leadership.
 Feedback
◦ Metrics provide short and long term feedback on
the effectiveness of projects and activities.
11/28/2015 7
Information

 Motivation
◦ People like knowing their efforts are worthwhile.
Metrics provide short and long term measurement
of progress.
 Objectivity
◦ Metrics form the basis for a fact-based
organization
11/28/2015 8
Information

 Problem Solving
◦ Metrics enable management to uncover problems
earlier
 Visibility
◦ A common set of metrics used throughout an
organization creates transparency.
11/28/2015 9
Information

◦ There should be a standardized meaning of the
measurement
◦ The data should help the decision making process
◦ It should provide worthwhile information
◦ It should be easy to install
◦ It can be benchmarked or used elsewhere
11/28/2015 10
Information

◦ Selecting metrics that take into account each of
these factors is a challenge
◦ Easy metrics to collect offer little value
◦ More worthwhile information requires effort to
establish a measurement process
11/28/2015 11
Information

◦ Many metrics are possible to fully characterize a quality
system.
◦ Large number of metrics wastes resources and distracts
leaders.
◦ Implementation of too many metrics may not result in a
compliant quality system.
◦ Metrics must be carefully chosen to ensure they align
with overall strategy.
11/28/2015 12
Information

◦ Good metrics system consists of leading and lagging
indicators.
◦ Lagging indicators measure the outcomes of what
already happened.
◦ Leading indicators provide information that may be able
to predict future outcomes.
◦ More mature organizations will typically have a greater
ratio of leading to lagging metrics.
11/28/2015 13
Information

◦ Most compliance metrics are inherently lagging indicators and are
typically easier to define.
◦ Establishing predictive leading indicators can improve a quality
system.
◦ Begin developing metrics with lagging indicators
◦ Then ask what actions or conditions cause this outcome to
brainstorm leading indicators
◦ It is necessary to understand the cause and effect
relationships.
◦ Possible to focus on impacting drivers which will ultimately
change the outcomes.
11/28/2015 14
Information

◦ Important to consider whether the metric measures time, quality
or quantity/cost.
◦ Time-based metrics are used to measure the duration of a
process.
 Metrics such as the percentage of CAPAs completed on-time is an
example.
◦ Quantity/cost metrics track the frequency or productivity of the
process.
 In a balanced scorecard these could include the cost of the process.
◦ Using financial metrics for compliance is not recommended.
11/28/2015 15
Information

◦ Quality metrics should
 Focus on effectiveness
 Measure that the right things are being done correctly.
◦ Often the focus is on efficiency measured in financial
terms.
◦ Effectiveness first eliminates waste & improves
efficiency.
◦ Quality metrics are difficult to establish
◦ Provide excellent insight into the health of the quality
system.
11/28/2015 16
Information

◦ Include representatives from all departments to design data
system successfully
◦ Align performance measurements to overall company strategic
plan
 Quality objectives will support one or more elements of a strategic plan.
◦ Involve individuals from all levels
◦ Have a dialog between levels and across organization
 Increase alignment,
 Focus the organization
 Develop buy-in
11/28/2015 17
Information

◦ Document the metrics definitions.
◦ Helps in collection of consistent metrics across
organization over time.
11/28/2015 18
Information

Definition One sentence explaining the Metric
Per cent of CAPA completed within agreed time
Benefit Quality objective to be achieved
Are non conformances resolved in a timely manner ?
Formula Specific formula to calculate the parameter
Target Give desirable goal
90 % , 10 % improvement over last year ,…..
Frequency Give desired frequency
Monthly , Quarterly , Yearly ,……
11/28/2015 19
Information

Status Rules Colors could be used for performance
Red ≤ 70 % , Yellow = 71% to 89% , Green =90% to 100%
Data Source Indicate where the data is availble
CAPA system
RACI Responsible: person(s) influencing the metric performance;
Accountable: person(s) ultimately responsible for the metric
performance.
Consulted: person(s) with knowledge in improving the metric
performance.
Informed: person(s) likely to benefit from the updates on the metric
performance.
Responsible: auditors; Accountable: Senior Audit Manager,
Consulted: none; Informed Quality council, Quality Director.
11/28/2015 20
Information

◦ Set targets and goals in alignment with strategy
◦ Have realistic expectations
◦ Collect baseline data for several reporting periods
◦ Establish current performance level with baseline data
◦ Establish improvement targets
◦ Measure performance improvement against the baseline
◦ Targets =“SMART-A”
 Specific, Measureable, Attainable, Realistic, Time-bound and
Aligned.
11/28/2015 21
Information

 Evolution of a Measure
◦ % of acceptable computer reports delivered
on time
◦ Number of acceptable computer reports
delivered on time
◦ Number of computer reports delivered on
time
◦ Number of computer reports delivered
11/28/2015 22
Information
Extent
Efficiency
Effectiveness

◦ Ensure that metric is performing as intended.
◦ Change the metric definition with caution as some
changes may invalidate older measurements.
◦ Need for a metric may change over time.
◦ Remove the metrics if it does not add value
11/28/2015 23
Information

◦ Establish Quality performance metrics based on
linkages to organizational objectives.
◦ Link to product or service performance, quality
costs, quality improvement and customer
satisfaction.
◦ Share the Quality goals across the organization.
11/28/2015 24
Information

◦ Quality metrics may be collected and reported by
the quality function
◦ Ownership of the issue belongs to operating
function
◦ Pareto principle applies to quality problems
◦ 80% of the quality problems are outside the control
of the quality function, while 20% may be internal.
11/28/2015 25
Information

◦ Organization wide problems could include
 Supplier quality requirements
 R&D product designs
 Distribution system problems
◦ Internal quality problems may include
 Lab test issues
 Auditing procedures
 Sampling procedures
11/28/2015 26
Information

◦ Performance against regulatory agency inspections
 Basic indicator of compliance
 Commercialization of products
 Manufacturing authorization.
11/28/2015 27
Information

◦ Internal Audit by an independent Auditor
 Snapshot of performance .
 On-going feedback on the health of the quality
system.
 Not an assurance of health of the quality system.
◦ Management Review to evaluate complete Quality
System
11/28/2015 28
Information

◦ Favourable outcomes help confirm compliance and
conformance.
◦ Unfavourable outcomes
 significant business interruptions
 Business loss
 Impact public perceptions and
 Decrease the value of a company’s brand and image
11/28/2015 29
Information

◦ Performance metrics against external inspections
are reactive measures.
◦ Do not rely on Regulatory inspection as a single
indicator.
◦ Conduct Internal audits more regularly
 Will enable a proactive response to avoid an external
finding.
 Functional area self-audits offer an excellent
opportunity for most proactive measurement.
11/28/2015 30
Information

Outcome Measure Method Other Considerations
% of inspections with no citations Trend Analyse citations according their
nature and details
Develop Pareto analysis of citations
Evaluate whether there are relative
differences between inspecting
authorities
% of audits with no adverse findings Trend Evaluate difference between Internal &
External performance.
Evaluate difference between sites ,
departments
Frequency and severity of
findings
Trend Evaluate difference between Internal &
External performance.
Evaluate difference between sites ,
departments
11/28/2015 31
Information

Process Measure Method Other Considerations
% of inspections with Timeliness of
internal audits against schedule
Trend Evaluate whether there are relative
differences between departments or
auditors
Indicators that audits covered
all quality systems, sites and
functions
Trend Evaluate differences between auditors
or departments
Number/per cent of open
inspection and audit actions
Trend Probe for reasons for open items.
Timeliness of closure of
inspection and audit actions
Trend Probe reasons for delay
Evaluate whether there are relative
differences between departments or
Auditors
11/28/2015 32
Information

◦ Analyse the findings according their nature and
details.
◦ Track with regulation citation number being
violated.
◦ Develop a pareto of finding by citations.
◦ Focus of global improvement efforts should be the
highest frequencies of findings
◦ Compare & calibrate internal audit group to
external authorities
11/28/2015 33
Information

◦ Refine audit program if internal audits do not
identify similar trends in findings as external
inspections
◦ Perform external benchmarking.
◦ Compare with studies showing pareto analysis of
finding by agencies such as the FDA.
◦ Carefully examine trends of external findings
frequency.
11/28/2015 34
Information

◦ Increase emphasis on emerging issues
◦ Adjust the internal audit plan to include covering
emerging issues
◦ Prepare organization for the frequent inspection
areas.
11/28/2015 35
Information

◦ Crucial to develop appropriate metrics.
◦ Important window to compliance and the health of
the quality system.
◦ Frequently a focus area for external inspectors.
◦ CAPA’s not addressed thoroughly & promptly raise
red flag during inspection
 Points to unresolved problems
 Indicates unresponsive management
11/28/2015 36
Information

% Investigations which discover root
cause
Trend Confirm the level of reoccurrences,
root cause and effectiveness and that
there are not any adverse trends or
patterns (current value against target,
trending).
Consider whether there are any
products, processes or departments
with a disproportionate number of
issues.
% of Nonconformance recurrence Trend
11/28/2015 37
Information

% Investigations completed within
defined timelines
Trend Evaluate any investigations that are
open for extended periods of time.
Probe the reasons for the delay
% of actions completed within
defined timelines
Trend Evaluate actions that are open for
extended period of time.
Probe the reason for delay
Frequency of Non conformances Trend Evaluate the location of the event,
cause, product, process step
11/28/2015 38
Information

◦ Analyse additional CAPA system information.
◦ Categorize the nature and details of the nonconformities.
Categorization by
 Product family
 Part number
 Type of process
 Equipment used
◦ Understand the cause and resolution of problems.
 Equipment failure
 Human error
 Supplier
11/28/2015 39
Information

◦ Evaluate problems attributed to human error
◦ Indicates failure to identify true root cause
◦ Monitor non-conformances & CAPA that are classified as
high risk
◦ Give priority for completion and confirmation
◦ When “non conforming” products are disposed off “as is”
make the decision visible to management.
◦ Give higher priority to CAPA for recalls, and complaints.
11/28/2015 40
Information

◦ Systems critical to the product safety & patient.
◦ Regulators Evaluate these systems very closely
◦ Complaint and Product Action process measures
determine the timeliness of key process steps
◦ Complaints open for a long period of time are a red
flag
 Points to a bigger problem
 May also indicate lack of resources or focus
◦ Adverse events have defined reporting time frame
11/28/2015 41
Information

Frequency of complaints
confirmed as a non-conformance
Trend
Pareto
Evaluate by product and product
family
Frequency of product recalls Trend Impact on other batches 0r products
Frequency of adverse events Trend Evaluate by product and product
family
11/28/2015 42
Information

Frequency of complaints by
product
Trend
Pareto
Frequency of complaints by
nature of complaint
Trend
Pareto
% of complaints closed within
defined timeframe
Trend Evaluate any complaints that are open
for extended periods of time
Regulatory notification filing
timeliness
Trend
11/28/2015 43
Information

◦ Complexity of supply chain has increased
◦ Significant importance for supplier quality performance
◦ Rigorous Quality system essential for Supplier
◦ Suppliers may include any one or more
 Providers of direct manufacturing materials including commodities,
 Starting materials,
 Chemicals,
 Excipients,
 API (active pharmaceutical ingredients),
 Drug product,
 Finished dosage and
 Third party manufacturers of finished products.
11/28/2015 44
Information

◦ A separate metrics may used for
 CRO for testing
 CRO for clinical studies
 Pest control service
 Calibration services
 IT services
11/28/2015 45
Information

◦ Part of a company’s quality system
◦ Act within their own quality system
◦ Monitor both perspectives with metrics
◦ Internal and external non-conformance are
examples of supplier driven issues
◦ Look at a supplier as a whole quality system
◦ A scorecard could be developed for critical
suppliers
11/28/2015 46
Information

◦ Scorecard
 Audit outcomes
 Lot failure rates
 Delivery performance and
 Responsiveness
◦ Prepare a composite metric for each supplier
◦ Prepare a composite metric from all critical supplier
scorecards
11/28/2015 47
Information

Frequency of non-conformances
due to supplier
Trend
Pareto
Evaluate overall and by supplier
Frequency of Incoming testing
failures
Trend
Pareto
Frequency of In-process failures due
to supplier
Trend
Pareto
11/28/2015 48
Information
Timeliness of internal supplier
audits against schedule
Trend
Pareto

11/28/2015 49
Information
 Analytical Lab metrics
◦ Frequency of OOS (out of specification)
 Calibration metrics
◦ Frequency of calibration OOS (out of specification)
◦ Calibration schedule conformance
 Cleanroom metrics
◦ Frequency of cleanroom OOS

11/28/2015 50
Dilutes the
effort.

11/28/2015 51
The goal
becomes to
manipulate the
number

11/28/2015 52
Any improvement will
not improve overall
performance.

11/28/2015 53
Exasperates involved
people

 In July 2012, the Food and Drug
Administration Safety and Innovation Act
(FDASIA) was enacted.
 This act requires the formation of a task force
to develop and implement a strategic plan for
enhancing the Agency's response to
preventing and mitigating drug shortages.
11/28/2015 54
Information

 The Task Force is specifically interested in seeking
public input on amongst others :
 “In an effort to address the major underlying causes
of drug and biological product shortages, FDA is
seeking new ideas to encourage high-quality
manufacturing and to facilitate expansion of
manufacturing capacity”
 To assist in the evaluation of product manufacturing
quality, FDA is exploring the broader use of
manufacturing quality metrics.
11/28/2015 55
Information

◦ FDA is now required to do risk based inspections
◦ Several risk factors are to be considered in the evaluation
(next slide)
◦ FDA can collect records otherwise available on inspection
“in advance or in lieu of inspection”
◦ Quality Metrics are considered potentially the sort of
information that could be provided in advance.
11/28/2015 56
Information

 FDA plans to use its authority to collect records
"in advance of or in lieu of" an inspection,
under section 704(a)(4)(A) of the FD&C Act to
gather various quality metrics data records.
 The agency says it will use these records to
"further develop [its] risk-based inspection
scheduling."
11/28/2015 57
Information

A. The compliance history of the establishment.
B. The record, history, and nature of recalls linked to the establishment.
C. The inherent risk of the drug manufactured, prepared, propagated,
compounded, or processed at the establishment.
D. The inspection frequency and history of the establishment, including
whether the establishment has been inspected pursuant to section 704
within the last 4 years.
E. Whether the establishment has been inspected by a foreign
government or an agency of a foreign government recognized under
section 809.
F. Any other criteria deemed necessary and appropriate by the Secretary
for purposes of allocating inspection resources.
11/28/2015 58
Information

 Implications of Providing Quality Metrics :
◦ Identify lower risk products, processes & sites
◦ Regulatory flexibility of lower reporting
 Prior approval supplement to an annual report
◦ Identify high quality site needing lower inspection
◦ Pathway to more objective inspections
11/28/2015 59
Information

◦ To assist in the evaluation of product manufacturing quality, FDA
is exploring the broader use of manufacturing quality metrics.
◦ With that in mind, FDA would like input on the following issues:
 What metrics do manufacturers currently use to monitor production
quality?
 To what extent do purchasers and prescribers use information about
manufacturing quality when deciding how to purchase or utilize
products?
 What kinds of manufacturing quality metrics might be valuable for
purchasers and prescribers when determining which manufacturers to
purchase from or which manufacturers' products to prescribe?
 What kinds of manufacturing quality metrics might be valuable for
manufacturers when choosing a contract manufacturer?
 How frequently would such metrics need to be updated to be
meaningful?
11/28/2015 60
Information

 The FDA released a draft guidance detailing
how it plans to use quality metrics in July
2015
 Based on this FDA proposes to
◦ Improve its ability to conduct risk-based
inspections and
◦ Predict or mitigate potential drug shortages.
11/28/2015 61
Information

 New draft guidance Request for Quality
Metrics
 FDA is giving more details to industry on its
approach to quality metrics,
 FDA is also asking industry for its input in
specific areas that are still being
considered.
11/28/2015 62
Information

 For companies with robust quality metrics data,
this information could be used to "reduce the
inspection frequency at an establishment."
 In cases where the data raises issues, FDA says
having the quality metrics data will allow it to
"improve the efficiency and effectiveness" of the
inspections.
 For now, the agency foresees its use of quality
metrics to be used primarily for reducing
inspection frequencies.
11/28/2015 63
Information

 FDA says it will request the following 10 baseline quality
metrics from companies as part of its analysis:
◦ The number of lots attempted of the product.
◦ The number of specification-related rejected lots of the product,
rejected during or after manufacturing
◦ The number of attempted lots pending disposition for more than 30
days
◦ The number of out-of-specification (OOS) results for the product,
including stability testing.
◦ The number of lot release and stability tests conducted for the
product.
11/28/2015 64
Information

 FDA says it will request the following 10 baseline quality
metrics from companies as part of its analysis:
◦ The number of product quality complaints received for the product.
◦ The number of lots attempted which are released for distribution or
for the next stage of manufacturing the product.
◦ If the associated annual product reviews (APRs) or product quality
reviews (PQRs) were completed within 30 days of annual due date for
the product
◦ The number of APRs or PQRs required for the product
11/28/2015 65
Information

 The agency is also asking for input on five "additional, optional
metrics as evidence of manufacturing robustness and a
commitment to quality."
◦ Senior Management Engagement—Was each APR or PQR reviewed and approved
by the following: (1) The head of the quality unit, (2) the head of the operations
unit, (3) both, or (4) neither?
◦ Corrective Action and Preventive Action (CAPA) Effectiveness—What percentage
of your corrective actions involved re-training of personnel (i.e., a root cause of
the deviation is lack of adequate training)?
◦ Process Capability/Performance—A “yes” or “no” value of whether the
establishment's management calculated a process capability or performance
index for each critical quality attribute as part of that product's APR or PQR.
11/28/2015 66
Information

 The agency is also asking for input on five "additional,
optional metrics as evidence of manufacturing robustness
and a commitment to quality.”
◦ Process Capability/Performance—A “yes” or “no” value of whether the
establishment's management has a policy of requiring a CAPA at some
lower process capability or performance index.
◦ Process Capability/Performance—If “yes” to the previous question—What is
the process capability or performance index that triggers a CAPA? If “no”
to the previous question—please do not respond.
11/28/2015 67
Information

 Keep it simple
 Improve objectivity
 Potential for regulatory flexibility
 Build consensus
 Promote right behavior
11/28/2015 68
Information

 This presentation was compiled from freely
available resources like the websites of FDA,
EMA, WHO.
 “Drug Regulations” is a non profit
organization which provides free online
Pharmaceuticals.
11/28/2015 69

Quality Metrics

More Related Content

What's hot

Viewers also liked

Similar to Quality Metrics

More from GMP EDUCATION : Not for Profit Organization

Recently uploaded

Quality Metrics