Quality Metrics

12,062 views

Published on

Published in: Education

Quality Metrics

  1. 1. Developing Meaningful Metrics: Driving Action for Results! Presented to ASQ St. Petersburg-Tampa-Section 1508 08 February 2010 by T. M. Kubiak, President Performance Improvement Solutions Conquering the Challenge!™ 1 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  2. 2. If you don’t have confidence in the diagnosis, you won’t have confidence in the prescription. — Steven Covey Conquering the Challenge!™ 2 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  3. 3. Conquering the Challenge!™ 3 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  4. 4. Can any of you help me with metrics? My boss wants a whole new set by the end of the week and I don’t even know where to start! HELP! Conquering the Challenge!™ 4 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  5. 5. Overview  The Driving Force  Characteristics of Measures  Definitions  Development Considerations  Why Measure?  Principles of Measurement  Barriers to Measurement  Where Should You Measure?  Pitfalls to Avoid  Goals, Targets, & Benchmarks  What Measures Measure  Evolution of a Measure  The “E’s” of Measurement  Examples of Measures  Different Views of Measures  Some Final Words... Conquering the Challenge!™ 5 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  6. 6. A Few Definitions, Of Course!  Measure (Metric)  Refers to numerical information that quantify input, output, and performance of processes, products, and services  Indicator  Measurement that relates to performance, but is not a direct or exclusive measure of such performance  Measure that is a predicator of some more significant performance Measure, Metrics, and Indicators! Oh my! Conquering the Challenge!™ 6 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  7. 7. Why Measure?  If you cannot measure it, you cannot control it. If you cannot control it, you cannot manage it. If you cannot manage it, you cannot improve it.  What gets measured and rewarded is what gets done. Corollary: If the measurements don’t change, neither do the results.  Measurements allow us to track where we have been, where we are, and where we are going  Shows how effectively we use our resources Measurement: The Roadmap to Progress Conquering the Challenge!™ 7 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  8. 8. More Reasons for Measuring  Focuses attention on factors contributing to achieving the organization’s mission  Assists in setting goals and monitoring trends and progress  Provides input for analyzing root cause and sources of errors  Identifies opportunities for on-going improvement  Provides a means of knowing whether we’re winning or losing Measurement: The Roadmap to Progress Conquering the Challenge!™ 8 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  9. 9. The Driving Force Integrated System of Metrics Customer Operational Requirements Processes Lean Six Sigma drives improvement at the individual process level. Corporate Business Business Support Objectives Strategy Requirements Processes Corporate Are these processes capable of Supplier Strategy Processes meeting business requirements? Driving Customer Satisfaction and Business Success! Conquering the Challenge!™ Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  10. 10. Barriers to Measurement  Thinking some jobs can’t be measured  Thinking there isn’t enough time to measure  The process of accomplishing a task is not understood  Fear that the measurement will become a club Conquering the Challenge!™ 10 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  11. 11. Pitfalls to Avoid!  Using the measure as a carrot or a stick Reason: the goal becomes to manipulate the number Conquering the Challenge!™ 11 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  12. 12. Pitfalls to Avoid!  Measuring everything that is possibly helpful Reason: dilutes the effort Conquering the Challenge!™ 12 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  13. 13. Pitfalls to Avoid!  Measuring parameters where the organization excels Reason: reduces the effort to a publicity campaign Conquering the Challenge!™ 13 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  14. 14. Pitfalls to Avoid!  Requiring precision in the data beyond the requirements of the decision Reason: exasperates all involved Conquering the Challenge!™ 14 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  15. 15. What Measures Measure  Quality (acceptable, good, in-spec, etc.)  Quantity (Number of...)  Time (Cycle time, rates)  Cost Conquering the Challenge!™ 15 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  16. 16. The “E’s” of Measurement!  Extent (volume-based or deployment)  How much?  Efficiency (productivity-based, usually expressed as a rate)  How fast?  Effectiveness (customer-based or objective-based)  How well? If It Were Only That Easy! Conquering the Challenge!™ 16 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  17. 17. Different Views of Measures  Organizational level  Operational measures (invisible to the customer)  Predictive measures (in line of sight of the customer)  Process Level  In-process measures (evaluate the method that creates results) W IPM W IPM W IPM W EOPM  End-of-process measures (measure the results of a method) W IPM W IPM W IPM W EOPM Conquering the Challenge!™ 17 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  18. 18. Characteristics of Measures  Actionable (So What?)  Relevant  Easily measurable  Specific  Economical to collect  Timely  Focused on processes  Traceable  Objective  Understandable If It Ain’t Actionable... Conquering the Challenge!™ 18 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  19. 19. Development Considerations  How often will the measure be computed and posted?  What about data stratification/aggregation?  Where will the data come from?  Is the data source reliable?  Can/should the data be normalized?  How will the measure be presented?  Who will see them? Critical Questions Require Thoughtful Answers Conquering the Challenge!™ 19 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  20. 20. Operational Definition  Answers: Who, How, What, Where, When  Defines: formulas and terms  Defines: interpretation (e.g., up is good)  Example: What is a “month?” Establishes a common language and understanding Conquering the Challenge!™ 20 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  21. 21. Cash for Clunkers Example Source: WSJ 09-08-05  Issue: 78 cars bumped from the list  Threshold: 18 mpg  Reason: More precise data (4 decimals) caused the revisions (Note: 0.0001 miles is ½ drop of fuel based on 18 mpg)  Measurement Methods:  Older: Dynamometer, fuel consumption measured, measured to 4 decimals, numbers rounded  Newer: Tailpipe emissions, measured to 4 decimals  Fix: Old data updated to be “effectively equivalent”  Quote: “Repeatability and accuracy is something we spend a lot of time on.” EPA scientist Hey, Joe! Gotta light? Conquering the Challenge!™ 21 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  22. 22. Cancer Deaths Example Source: WSJ 09-02-09 Does this chart support the conclusion? Normalized •Gaps in age stratification •Aggregated across Stratified by age gender •Aggregated across cancer type Conquering the Challenge!™ 22 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  23. 23. Pareto Chart Example 20 20 18 18 16 16 14 14 12 12 10 10 8 8 6 6 4 4 2 0 2 Def Def Def Def Def Def 0 1 3 5 7 9 11 Def 1 Def 2 Def 3 Def 4 Other You can’t fix stupid! Conquering the Challenge!™ 23 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  24. 24. Air Travel Example Source: WSJ 1--01-07  What is your understanding of an “on-time arrival?”  How does plus or minus 15 minutes of the scheduled arrival time sound?  Is this a good metric or a bad metric?  How many customers are aware of it?  Related metrics (by carrier):  Percentage of flights arriving on-time  Percentage of canceled flights  Mishandled bag reports per 1,000 passengers  Complaint reports per 1,000,000 passengers  Bumped passengers per 10,000 Do key players understand the metrics? Conquering the Challenge!™ 24 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  25. 25. Principles of Measurement  Measurements must be specific  Measure the outputs of highest value to the customer  Measures can be applied to all performance dimensions - external as well as internal  Measure the process as well as the results  Understand the game before you decide how you’ll keep score  There is no single perfect measure Extent Effectiveness Efficiency Conquering the Challenge!™ 25 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  26. 26. Where Should You Measure?  Early in the process to promote prevention  At the point where a cause-and-effect relationship can be established At critical processes  Functional boundaries  Points of convergence  Points of divergence Conquering the Challenge!™ 26 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  27. 27. Goals, Targets, & Benchmarks Conquering the Challenge!™ 27 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  28. 28. If you can’t win the race, make sure the guy ahead of you breaks the record. — Steve Prefontaine (1951- 1975), American runner Conquering the Challenge!™ 28 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  29. 29. Evolution of a Measure  Percent of acceptable computer Effectiveness reports delivered on time  Number of acceptable computer reports delivered on time Efficiency  Number of computer reports delivered on time  Number of computer reports delivered Extent Conquering the Challenge!™ 29 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  30. 30. Examples of Measures  Measure: Trouble calls received/week  The value of this measure is unclear. It is not customer-focused nor does it measure the ability of an organization to deal with incoming trouble calls since no processing of trouble calls is evident in the metric. Essentially, this metric measures nothing more than the volume of incoming trouble calls. The fact that it is expressed in the form of a rate is misleading and may fool one into thinking it is a measure of efficiency.  Type of Measure: Extent Conquering the Challenge!™ 30 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  31. 31. Examples of Measures  Measure: Job Scheduled  This metric focuses on measuring nothing more than the volume of jobs scheduled. Other than that, it provides little insight into the underlying process of “scheduling jobs.” The metric is not actionable.  Type of Measure: Extent Conquering the Challenge!™ 31 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  32. 32. Examples of Measures  Measure: Actual versus planned sales  Measures the planning/forecasting process.  Type of Measure: Effectiveness Conquering the Challenge!™ 32 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  33. 33. Examples of Measures  Measure: Sales/employee  This metric is highly misleading and often used for organization to organization comparisons. It falsely assumes that all employees generate sales and does not consider the product being sold or the associated supporting infrastructure required to produce and sell the product.  Type of Measure: Efficiency Conquering the Challenge!™ 33 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  34. 34. Examples of Measures  Measure: Number of training hours/employee  This metric measures the deployment of training hours to the workforce.  Type of Measure: Extent Conquering the Challenge!™ 34 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  35. 35. Examples of Measures  Measure: Revenue/sales training hour  This metric measures the impact of training the sales workforce so long as a clear cause-and- effect relationship can be established between training and revenue.  Type of Measure: Effectiveness Conquering the Challenge!™ 35 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  36. 36. Examples of Measures  Measure: Engineering change orders/drawing  This metric measure the design process from the point of view of the drawings. However, note that this metric does not normalize or adjust for the complexity of a given drawing.  Type of Measure: Effectiveness Conquering the Challenge!™ 36 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  37. 37. Some Final Words...  Measure, Control, Manage, Improve  Extent, Efficiency, Effectiveness  Performance measurement is irrelevant without a frame of reference  Measure twice! But cut once! Conquering the Challenge!™ 37 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  38. 38. People without information are unable to change; those with information are compelled to change. Conquering the Challenge!™ 38 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  39. 39. Conquering the Challenge!™ 39 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions
  40. 40. Conquering the Challenge!™ 40 Copyright T. M. Kubiak 1996-2010 Performance Improvement Solutions

×