This document discusses quality indicators, their history, definitions, and examples. It describes how quality indicators can be used to monitor performance, determine quality of services, and identify areas for improvement. The document provides examples of quality indicators collected by various laboratories and organizations. It also outlines best practices for developing, presenting, and using quality indicators effectively.

1. Quality Indicators:
Past and Present
Michael A Noble MD FRCPC
Professor
Medical Microbiology and Infection Control, Vancouver Coastal Health
and
Chair, Clinical Microbiology Proficiency Testing program,
Chair, Program Office for Laboratory Quality Management
Department of Pathology and Laboratory Medicine
University of British Columbia

2. Quality Indicators:
Past and Present
• History
• Quality Indicators and ISO
• Characteristics of Indicators – strong and weak
• Quality Indicator Inventories – USA and BC
• Examples of Quality Indicators
• Summary

3. A really
good,
inexpensive
reference
book
Amazon.com
$30.00

4. Two excellent (essential) references from CSA

5. A Short History of Metrics in
Quality Management
Innovator Date Cycle
Walter A Shewart 1920’s Plan-Do-SEE
J Edwards Deming 1940’s Plan-Do-CHECK-Act
Bob Galvin 1980’s Define-MEASURE-
Analyze-Improve-
Control

6. The Quality Cycle
Plan
Do
Act
CHECK
Each step is
essential to keep
the quality cycle
cycling

7. Quality Indicators
A workable definition
• Established measures used to
determine how well an organization
meets needs and operational and
performance expectations.
– Objective
– Measurable
– Repeatable

8. Metrics and ISO 9001:2000
Factual Approach to Decision Making
5.4.1
• Top management should ensure that quality objectives,
including those needed to meet requirements for product,
are established at relevant functions and levels within the
organization. The quality objectives shall be measurable
and consistent with the quality policy.

9. Metrics and ISO 9001:2000 (2)
8.4
The organization shall determine, and collect and analyze appropriate
data to demonstrate suitability and effectiveness of the quality
management system and evaluate where continual improvement of the
effectiveness of the quality management system can be made. This
shall include data generated as a result of monitoring and measurement
and from other relevant sources.
The analysis of data shall provide information relating to:
– Customer satisfaction
– Conformity to product requirements
– Characteristics and trends of processes and products including
opportunities for preventive actions, and
– suppliers

10. Metrics and ISO 15189:2003
• 4.12.4
Laboratory management shall implement quality
indicators for systematically monitoring and
evaluating the laboratory’s contribution to patient
care. When this program identifies opportunities
for improvement, laboratory management shall
address them regardless of where they occur.
Laboratory management shall ensure that the
medical laboratory participates in quality
improvement activities that deal with relevant
areas and outcomes of patient care.

11. So…
Quality Indicators are measured information that
• Indicates the performance of a process.
• determines quality of services.
• highlights potential quality concerns,
• identifies areas that need further study and
investigation, and
• track changes over time.

12. Measuring Performance
Mark Graham Brown
• Fewer is better.
• Link measures to the factors needed for success.
• Measures should be based around customer and stakeholder
needs.
• Measures should start at the top and flow down to all levels
of employees.
• Measures should change as the environment and strategy
changes
• Measures should have targets or goals established that are
based on research rather than arbitrary values.

13. Keeping Score
Using the Right Metrics to Drive World Class
Performance
1996
Many organizations spend thousands of
hours collecting and interpreting data.
However many of these hours are nothing
more than wasted time because they
analyze the wrong measurements,
leading to inaccurate decision making.
– Mark Graham Brown.

14. Indicators?
You want Indicators?
We’ve got LOTS of Indicators!
AHRQ
IQLM
American Nurses Association
American Psychiatric Association
RAND
WHO
Leapfrog
National Quality Forum
JCAHO
ISQua
OECD
ASQ

15. Characteristics of Good Metrics
Timed
short and long term
Engaging
all levels
Balanced
full cycle
Actionable
action oriented
Interpretable
specific
Achievable
contained
Measurable
objective
Good
Metrics

16. Indicators of Good Indicators
Measurable Can you count it, time it, record it?
Achievable Can you actually capture it?
Interpretable When you’ve got it, what does it mean?
Actionable Can you do something about it?
Timed Does your set cover both the short and long term?
Engaging Does your set involve all laboratory personnel?
Balanced Does your set cover the full cycle of events?

17. Assessing Quality Indicators
• Importance Potential for Improvement
• Scientific Acceptability Reliability and Validity
• Feasibility Implementation and cost
• Usefulness Comprehensive
Having Quality Quality Indicators

18. Total Testing Cycle for
Medical Laboratories
Report Interpretation
Report Creation
Data Capture
Report Transport
Analysis
Quality Control
Analytic Post-Analytic
Pre-Analytic
Menu
Ordering Rules
Patient ID
Acceptance Criteria
Collection
Transport

19. Baldrige Award Criteria
• Balanced Metrics
– Customer satisfaction
– Employee satisfaction
– Financial performance
– Operational performance
– Product and Service quality
– Supplier performance
– Safety and environment and public responsibility
Most organizations focus 80% of metrics on
finance and operations.

20. IQLM Indicators
• Diabetes monitoring (system)
• Hyperlipidemia screening (system)
• Test Order Accuracy and Appropriateness
• Patient Identification (pre-analytic)
• Adequacy and Accuracy of Specimen Information (pre-analytic)
• Blood Culture Contamination (pre-analytic / system)
• Accuracy of point-of-care testing (analytic)
• Cervical cytology/biopsy correlation (analytic)
• Critical Values Reporting
• Turnaround time (postanalytic)
• Clinician satisfaction (system/postanalytic)
• Clinician followup (system/postanalytic)

21. CMPT Metrics Scorecard
• Balanced Metrics Percent
– Customer satisfaction 25
– Employee satisfaction 5
– Financial performance 10
– Operational performance 20
– Product and Service quality 30
– Supplier performance 5
– Safety /environment /
public responsibility 5

22. Characteristics of Weak
Metrics
• Focus only on measures easy to count
• Focus only on measures easy to achieve.
• Metrics with arbitrary targets.
• Measures that don’t change with experience

23. Computer Nonsense Metrics
[urine culture] * [glucose] * [INR]
[NUPA hr] * [Telephone minutes]
X100
Just because a computer
can calculate a value,
doesn’t mean that it
should.

24. Computerese Quality Indicators
• Unit Producing Activity per Paid Hour
• Unit Producing Activity per Worked Hour
• Unit Producing Activity per Total FTE
• Non-Unit Producing Activity per Paid Hour
• Non-Unit Producing Activity per Worked Hour
• Non-Unit Producing Activity per Total FTE
• Crude Turn-Around-Time

25. A Cautionary Note
• Measures that drive the wrong performance.
Measuring professionals is tough because
intellectual work is difficult to measure
objectively. Looking for factors that can be
counted may not be what is really
important. Meaningful outputs such as
ideas, information, and problems avoided
may be difficult but more relevant.
Mark Graham Brown

26. Caution about
patient outcome indicators
Theoretically, outcomes best assess quality,
but they are the most difficult to measure
– too many variables and confusers
• Age, underlying conditions, therapy, circumstance
– require high volumes of detailed data
– Need long collection periods.
David Hsia
Medicare Quality Improvement Bad Apples or
Bad Systems?
JAMA. 2003;289:354-356.

27. Are you an Indicator Glutton?
Monitoring more
than 10-12
indicators is rarely
successful
Mark Graham Brown
1996

28. Quality Inventory:
US Medical Laboratories
2004
• In 2004 the Institute for Quality in Laboratory
Medicine (IQLM) and the Clinical Laboratory
Managers Association (CLMA) undertook an on-
line quality inventory of laboratories with CLMA
members.
• Approximately 400 laboratories responded.
• The study was voluntary, self-reported, with a
validated questionnaire.
• Information provided was not verified by a second
method

30. In British Columbia…
The Program Office for Laboratory Quality
Management and the Provincial Laboratory
Coordinating Office have organized to
perform a similar, but improved inventory
in 2005.

31. 10 Most Common Procedures Monitored
BC Quality Inventory 2005
76
78
80
82
84
86
88
90
92
94
96
Q
C
E
Q
A
P
T
I
D
C
o
n
t
a
i
n
e
r
O
K
S
a
m
p
l
e
I
n
t
e
g
r
i
t
y
S
a
m
p
l
e
I
D
L
a
b
I
n
j
u
r
i
e
s
W
r
i
t
t
e
n
O
r
d
e
r
s
C
o
l
l
e
c
t
i
o
n
T
i
m
i
n
g
S
p
e
c
i
m
e
n
S
t
o
r
a
g
e
Pre-Analytic
System
Analytic

32. Post Analytic Procedures Monitored
BC Quality Inventory 2005
62
64
66
68
70
72
74
76
78
Report Accuracy
Report Critical Values
Turn Around Time

33. Satisfaction Monitoring
BC Quality Inventory 2005
0
10
20
30
40
50
60
P
a
t
i
e
n
t
P
a
t
i
e
n
t
(
P
l
e
b
o
t
o
m
y
)
P
h
y
s
i
c
i
a
n
E
m
p
l
o
y
e
e

34. Other Achievable Indicators
• Blood culture volumes:
Blood culture false negative results occur
when bottles contain insufficient (<3 mL)
or excessive (>15 mL) blood.
Insufficient or excessive blood collection is
a collection non-conformity.

35. Under and Overfill Blood Cultures
2001-2004
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
5.0%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Percent Over
Percent Under

36. Underfill Blood Collections
(As a percent of collections per site)
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
RF2
RF1
ER
ICU
Phlebotomists

37. Baldrige Award Criteria
• Balanced Metrics
– Customer satisfaction
– Employee satisfaction
– Financial performance
– Operational performance
– Product and Service quality
– Supplier performance
– Safety and environment and public responsibility
Most organizations focus 80% of metrics on
finance and operations.

38. Eight Steps to Developing
Successful Indicators
1. Objective
2. Methodology
3. Limits
4. Interpretation
5. Limitations
6. Presentation
7. Action plan
8. Exit plan

39. Developing Indicators
Objective What are you trying to measure
Methodology 1. How to capture the data
2. Who (or what) to capture the data
3. How often to capture the data
Limits Acceptable, Concern, Unacceptable Critical
Presentation Graphic or Text
Interpretation What does it mean?
Does it reflect on YOUR quality?
Limitations Unintended variables
Action Plan What will I do if it indicates acceptable performance?
What will I do if it does not?
Exit Plan When can I stop measuring?

40. Presenting
Quality Indicator Information
0
10
20
30
40
50
60
70
80
1st Qtr 2ndQtr 3rdQtr 4th Qtr
High
Low
Average

41. The BIG SECRET for
Quality Indicator Team
Engage the folks
who do the work,
because they
know what
they do!

42. Microbiology Indicators
Collected and Monitored by Vancouver
General Hospital Division of Medical
Microbiology and Infection Control
Many thanks to:
Diane Roscoe
Anita Kwong
Medical Microbiology team

43. Contmination Rate: Blood Culture Sets
0.0%
1.0%
2.0%
3.0%
4.0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time Period
Percent
2002-2003 2003-2004 2004-2005

44. Contmination Rate: Blood Culture Sets
0.0%
1.0%
2.0%
3.0%
4.0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time Period
Percent
2002-2003 2003-2004 2004-2005
Limits: Below 2%
Interpretation: Meeting accepted limits all the time
Limitations Definition may include some true infections and
may miss others
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.
Exit plan: Reactivate with cause
Objective: to ensure that blood
culture results reflect
sepsis.
Methodology: Count single bottle
positives of common skin
flora/ total sets

45. Underfill Blood Collections
(As a percent of collections per site)
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
RF2
RF1
ER
ICU
Phlebotomists

46. Limits: Below 2% (?)
Interpretation: Wards with inexperienced collectors have problems
Limitations Some frail and elder people have very weak veins and may be
impossible to collect
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.
Exit: Continue on selective basis
Objective: to ensure that blood
culture are properly filled.
Methodology: Count underfilled bottles /
total bottles collected
Underfill Blood Collections
(As a percent of collections per site)
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
RF2
RF1
ER
ICU
Phlebotomists

47. Certification Performance
Year Event Measures MAJOR
NC
Minor
NC
2002 Pre-Certification (EI) 100 1 2
2002 Certification (E) 100 0 2
2003 Pre-Certification (EI) 100 0 0
2003 Certification (E) 100 0 0
2004 Certification (E) 100 0 0
2005 Pre-certification (EI) 100 0 1
2005 Re-Certification (E) 100 0 0

48. Limits: No Major above 1Below 2%; No Minor above 3
Interpretation: Meeting accepted limits all the time
Limitations May indicate things are better than they are if
inspector is not diligent
Presentation: Linear time table
Action plan: Maintain program, respond through OFI and Corrective
Actions
Exit plan: Compile with each inspection
Objective: to monitor CMPT quality
preparedness
Methodology: Monitoring External
assessment values
Year Event Measures MAJOR
NC
Minor
NC
2002 Pre-Certification (EI) 100 1 2
2002 Certification (E) 100 0 2
2003 Pre-Certification (EI) 100 0 0
2003 Certification (E) 100 0 0
2004 Certification (E) 100 0 0
2005 Pre-certification (EI) 100 0 1
2005 Re-Certification (E) 100 0 0

49. Composite Indicators
• Reflecting a single subject with a number of
sub-components
When the finished value is greater than just the
sum of the parts

50. Creating Composite
Quality Indicators
1
2
3
4
5
25
üIdentify individual components
üWeight the components
üDefine Limits
üMeasure and Combine
üMonitor for trend

51. CMPT Client Satisfaction
Composite Score
Factor Weighting
Survey Score +10
Open Comments –Positive +5
Open Comments – Negative -10
New Contracts +10
Contract Renewals +25
Contract Cancellations -100
Consults +5
Complaints -10

52. Limits
Survey
Positive
Opinions
Negative
Opinions
New
Contracts
Contract
Renewals
Contract
Cancellations
Consults
Complaints
VALUE
90 30 5 1 1 0 1 0 105
80 30 16 0 0 0 0 1 76

53. Year
Survey
Positive
Opinions
Negative
Opinions
New
Contracts
Contract
Renewals
Contract
Cancellations
Consults
Complaints
2002-2003 90 24 6 0 0 0 5 2
2003-2004 85 22 10 4 0 0 5 0
2004-2005 85 22 6 6 0 0 3 0
2005-2006 85 20 2 2 1 0 4 1

54. CMPT Client Satisfaction
Composite Score
96.5 96.5
103.5
100
70
80
90
100
110
2002-2003 2003-2004 2004-2005 2005-2006

55. CMPT Composite Satisfaction Score
• Objectives: To indicate customer satisfaction
• Methodology: Examination of 5 independent variables
• Presentation: Composite score
• Interpretation: Score associated with satisfaction
• Limits: 76-105 calculated weighted score
• Limitations: Arbitrary
• Action plan: Root Cause Analysis of deficiencies
• Exit: Annual for 5 years and evaluate

56. In Summary
• Quality Quality Indicators are a required
component of a quality management system.
• Quality Quality Indicators can be characterized
and distinguished from Weak and Terrible Quality
Indicators.
– Watch out for the weak ones
– Avoid the terrible ones
• Quality Quality Indicators provide the information
and opportunity essential for POSITIVE action.

57. Setting Relevant Ranges
• Set Objectively
• Validate by Study
• Clinical Relevancy
• Customer Expectation
• Matched Benchmarks
• Regulation 60 minutes
Relevant or Easy?

58. Quality Indicators and Timing
Use an indicator
only as long as
it provides
you with
useful
information.
Don’t get tied to
your indicators

59. Caution about
patient outcome indicators
Theoretically, outcomes best assess quality,
but they are the most difficult to measure
– too many variables and confusers
• Age, underlying conditions, therapy, circumstance
– require high volumes of detailed data
– Need long collection periods.
David Hsia
Medicare Quality
Improvement Bad Apples
or Bad Systems?
JAMA. 2003;289:354-
356.

60. The BIG SECRET for
Quality Indicator Team
Engage the folks
who do the work,
because they
know what
they do!

61. Timed