Quality Metrics

Developing Meaningful Metrics:
Driving Action for Results!
Presented to

ASQ St. Petersburg-Tampa-Section 1508

08 February 2010

by

T. M. Kubiak, President
Performance Improvement Solutions

Conquering the Challenge!™ 1 Copyright T. M. Kubiak 1996-2010

If you don’t have confidence in
the diagnosis, you won’t have
confidence in the prescription.

— Steven Covey


Can any of you help me with
metrics? My boss wants a
whole new set by the end of
the week and I don’t even
know where to start! HELP!


Overview
 The Driving Force  Characteristics of Measures

 Definitions  Development Considerations

 Why Measure?  Principles of Measurement

 Barriers to Measurement  Where Should You Measure?

 Pitfalls to Avoid  Goals, Targets, & Benchmarks

 What Measures Measure  Evolution of a Measure

 The “E’s” of Measurement  Examples of Measures

 Different Views of Measures  Some Final Words...

A Few Definitions, Of Course!
 Measure (Metric)
 Refers to numerical information that quantify
input, output, and performance of processes,
products, and services

 Indicator
 Measurement that relates to performance, but
is not a direct or exclusive measure of such
performance
 Measure that is a predicator of some more
significant performance

Measure, Metrics, and Indicators! Oh my!

Why Measure?
 If you cannot measure it, you cannot control it.
If you cannot control it, you cannot manage it.
If you cannot manage it, you cannot improve it.
 What gets measured and rewarded is what gets
done. Corollary: If the measurements don’t
change, neither do the results.
 Measurements allow us to track where we have
been, where we are, and where we are going
 Shows how effectively we use our resources

Measurement: The Roadmap to Progress

More Reasons for Measuring
 Focuses attention on factors contributing to
achieving the organization’s mission
 Assists in setting goals and monitoring
trends and progress
 Provides input for analyzing root cause and
sources of errors
 Identifies opportunities for on-going
improvement
 Provides a means of knowing whether we’re
winning or losing
Measurement: The Roadmap to Progress

The Driving Force
Integrated System of Metrics

Customer Operational
Requirements Processes
Lean Six Sigma drives
improvement at the individual
process level.

Corporate Business Business Support
Objectives Strategy Requirements Processes

Corporate Are these processes capable of Supplier
Strategy Processes
meeting business requirements?

Driving Customer Satisfaction and Business Success!
Conquering the Challenge!™ Copyright T. M. Kubiak 1996-2010

Barriers to Measurement
 Thinking some jobs can’t be measured
 Thinking there isn’t enough time to measure
 The process of accomplishing a task is not
understood
 Fear that the measurement will become a club


Pitfalls to Avoid!
 Using the measure as a carrot or a stick
Reason: the goal becomes to manipulate the number


Pitfalls to Avoid!
 Measuring everything that is possibly helpful
Reason: dilutes the effort


Pitfalls to Avoid!
 Measuring parameters where the organization excels
Reason: reduces the effort to a publicity campaign


Pitfalls to Avoid!
 Requiring precision in the data beyond the
requirements of the decision
Reason: exasperates all involved


What Measures Measure
 Quality (acceptable, good, in-spec, etc.)

 Quantity (Number of...)

 Time (Cycle time, rates)

 Cost

The “E’s” of Measurement!
 Extent (volume-based or deployment)
 How much?
 Efficiency (productivity-based, usually
expressed as a rate)
 How fast?
 Effectiveness (customer-based or
objective-based)
 How well?

If It Were Only That Easy!

Different Views of Measures
 Organizational level
 Operational measures (invisible to the customer)
 Predictive measures (in line of sight of the customer)

 Process Level
 In-process measures (evaluate the method that
creates results)

W IPM W IPM W IPM W EOPM

 End-of-process measures (measure the results of a
method)
W IPM W IPM W IPM W EOPM


Characteristics of Measures

 Actionable (So What?)  Relevant

 Easily measurable  Specific

 Economical to collect  Timely

 Focused on processes  Traceable

 Objective  Understandable

If It Ain’t Actionable...

Development Considerations
 How often will the measure be computed and
posted?
 What about data stratification/aggregation?
 Where will the data come from?
 Is the data source reliable?
 Can/should the data be normalized?
 How will the measure be presented?
 Who will see them?
Critical Questions Require Thoughtful Answers

Operational Definition
 Answers: Who, How, What, Where, When
 Defines: formulas and terms
 Defines: interpretation (e.g., up is good)
 Example:

What is a “month?”

Establishes a common language and understanding

Cash for Clunkers Example Source: WSJ 09-08-05

 Issue: 78 cars bumped from the list
 Threshold: 18 mpg
 Reason: More precise data (4 decimals) caused the
revisions (Note: 0.0001 miles is ½ drop of fuel based on
18 mpg)
 Measurement Methods:
 Older: Dynamometer, fuel consumption measured,
measured to 4 decimals, numbers rounded
 Newer: Tailpipe emissions, measured to 4 decimals
 Fix: Old data updated to be “effectively equivalent”
 Quote: “Repeatability and accuracy is something we
spend a lot of time on.” EPA scientist

Hey, Joe! Gotta light?

Cancer Deaths Example Source: WSJ 09-02-09

Does this chart support
the conclusion?
Normalized

•Gaps in age
stratification
•Aggregated across
Stratified by age gender
•Aggregated across
cancer type


Pareto Chart Example
20 20
18 18
16 16
14 14
12 12
10
10
8
8
6
6
4
4
2
0 2
Def Def Def Def Def Def 0
1 3 5 7 9 11 Def 1 Def 2 Def 3 Def 4 Other

You can’t fix stupid!

Air Travel Example Source: WSJ 1--01-07

 What is your understanding of an “on-time arrival?”
 How does plus or minus 15 minutes of the scheduled
arrival time sound?
 Is this a good metric or a bad metric?
 How many customers are aware of it?
 Related metrics (by carrier):
 Percentage of flights arriving on-time
 Percentage of canceled flights
 Mishandled bag reports per 1,000 passengers
 Complaint reports per 1,000,000 passengers
 Bumped passengers per 10,000

Do key players understand the metrics?

Principles of Measurement
 Measurements must be specific
 Measure the outputs of highest value to
the customer
 Measures can be applied to all performance
dimensions - external as well as internal
 Measure the process as well as the results
 Understand the game before you decide
how you’ll keep score
 There is no single perfect measure
Extent Effectiveness
Efficiency


Where Should You Measure?
 Early in the process to promote prevention
 At the point where a cause-and-effect
relationship can be established
At critical processes
 Functional boundaries
 Points of convergence
 Points of divergence


Goals, Targets, & Benchmarks


If you can’t win the race, make
sure the guy ahead of you
breaks the record.

— Steve Prefontaine (1951-
1975), American runner


Evolution of a Measure
 Percent of acceptable computer Effectiveness
reports delivered on time

 Number of acceptable computer
reports delivered on time
Efficiency
 Number of computer reports
delivered on time

 Number of computer reports
delivered Extent


Examples of Measures
 Measure: Trouble calls received/week

 The value of this measure is unclear. It is not
customer-focused nor does it measure the
ability of an organization to deal with incoming
trouble calls since no processing of trouble
calls is evident in the metric. Essentially, this
metric measures nothing more than the
volume of incoming trouble calls. The fact that
it is expressed in the form of a rate is
misleading and may fool one into thinking it is
a measure of efficiency.
 Type of Measure: Extent


 Measure: Job Scheduled

 This metric focuses on measuring nothing
more than the volume of jobs scheduled. Other
than that, it provides little insight into the
underlying process of “scheduling jobs.” The
metric is not actionable.



 Measure: Actual versus planned sales

 Measures the planning/forecasting process.

 Type of Measure: Effectiveness


 Measure: Sales/employee

 This metric is highly misleading and often
used for organization to organization
comparisons. It falsely assumes that all
employees generate sales and does not
consider the product being sold or the
associated supporting infrastructure required
to produce and sell the product.

 Type of Measure: Efficiency


 Measure: Number of training hours/employee

 This metric measures the deployment of
training hours to the workforce.



 Measure: Revenue/sales training hour

 This metric measures the impact of training the
sales workforce so long as a clear cause-and-
effect relationship can be established between
training and revenue.



 Measure: Engineering change orders/drawing

 This metric measure the design process from
the point of view of the drawings. However,
note that this metric does not normalize or
adjust for the complexity of a given drawing.



Some Final Words...
 Measure, Control, Manage, Improve
 Extent, Efficiency, Effectiveness
 Performance measurement is
irrelevant without a frame of
reference
 Measure twice! But cut once!


People without information are
unable to change; those with
information are compelled to
change.


Quality Metrics

More Related Content

What's hot

Similar to Quality Metrics

More from Haroon Abbu

Recently uploaded

Quality Metrics