Successfully reported this slideshow.
Your SlideShare is downloading. ×

Establishing Performance Standards

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 75 Ad
Advertisement

More Related Content

Recently uploaded (20)

Advertisement

Establishing Performance Standards

  1. 1. Establishing Performance Standards: A Practical Approach Establishing Performance Standards: A Practical Approach by David G. Rhoads, Ph.D., DABCC Copyright 2005 by David G. Rhoads Associates, Inc.
  2. 2. 2 Topics Topics z Define performance standards and show why they are important. z List sources for calculating performance standards z Show formulae for those specific calculations z Define process for figuring out how to calculate a performance standard. z Show examples for specific performance standards.
  3. 3. 3 What are Performance Standards? What are Performance Standards? z Performance Standards define the required quality of the basic product of the clinical laboratory, patient results. z Performance Standards are used in two ways: – To define allowable systematic error (i.e. bias) – To define allowable random error (target SD values for routine QC). z These two metrics define the quality specifications of our product.
  4. 4. 4 Status of Performance Standards Status of Performance Standards z There is no consensus in our industry that we need performance standards except perhaps for the CLIA ’88 PT limits. z There is no consensus on how to set performance standards. z There is no consensus on what those performance standards should be. z Achieving a consensus on these issues may take a decade. We need to start the process now.
  5. 5. 5 What are Performance Standards? What are Performance Standards? z Performance Standards for Clinical Laboratory Tests is expressed in terms of Experimental Error. – I like the term Performance Standards because it is positive. I use it instead of Total Allowable Error (TEa) because the latter has negative connotations and is technical. z Understanding experimental error is essential to establishing and maintaining a program of good quality assurance and good quality control. z We will see the consequences of QA failure in the next few slides.
  6. 6. 6 Regulatory Effect of QC Failure Regulatory Effect of QC Failure
  7. 7. 7 Clinical Effect of QC Failure Clinical Effect of QC Failure z Many clinical protocols are triggered by results exceeding a certain medical decision point. They assume reasonably accurate results. z What happens if – Some specimens with true results below this value in fact exceed the MDP? – Some specimens with true results above this value in fact do NOT exceed the MDP? z Both can cause incorrect diagnoses plus extra cost and pain for both the health care system and the patient as the system deals with consequences of poor lab results.
  8. 8. 8 Healthcare System Costs – Incorrect Calibration Healthcare System Costs – Incorrect Calibration z If Calcium has a bias to the high side, a whole series of follow-up tests may be ordered in order to confirm the initial diagnosis of hypercalcemia. z The costs of the follow-up lab tests alone ranges from $60 to 199 million per year nationally depending on the magnitude of the calibration error. – Clin Lab News, Aug 2004. How much does Test Calibration Error Cost? z This is a very conservative estimate based only on the Medicare costs of possible follow-up tests. It does not include patient or physician costs for the follow-ups.
  9. 9. 9 A Documented Instance of Error A Documented Instance of Error z Dr. George Klee at Mayo Clinic did an experiment in which he evaluated the change in the number of patients with cholesterol values exceeding the cutoff value in the presence of varying amounts of bias. z If bias changed 1%, there was a 5% change in the number of patients exceeding the 200 mg/dL cutoff. z If bias changed 3%, the corresponding change was 15%.
  10. 10. 10 Costs of Quality Failure Costs of Quality Failure z If lab results causing incorrect diagnoses amounted to 0.12% of the results for a lab producing 1 million tests a year, that would mean 1,200 patients per year would be incorrectly diagnosed (i.e. about 3.5 per day). z If the average additional cost for each of these incorrectly diagnosed patients was only $1,000, then these incorrect results would result in an additional $1.2 million cost per year for the health care system. z This cost is a significant fraction of the total budget for a lab performing this volume of work. The budget for a lab this size is approximately $3 million per year. z Plebani and Carraro reported 49 medically significant errors from 40,490 tests (0.12%). A medically significant error is one which resulted in an inappropriate care or evaluation of a patient. Ref: CCJ 43, 1348 (1997)
  11. 11. 11 Business Costs of QC/QA Failure – 1 Business Costs of QC/QA Failure – 1 z Loss of business – Poor quality of chemistry results in one Southern hospital caused physicians to send their patients to another hospital in the same city. • Private communication to DG Rhoads – April, 2004 z Loss of jobs – At Maryland General Hospital in Baltimore, after QC failures for HIV testing were widely publicized, hospital and lab management were replaced. – A “vast majority” of 460 patient results were repeated. 99.6% were unchanged. (Two did change.) • Testimony before a subcommittee of the House Committee of Government Reform, May 18, 2004
  12. 12. 12 Business Costs of QC/QA Failure – 2 Business Costs of QC/QA Failure – 2 z Hospital failed!! – At St. Agnes Medical Center in Philadelphia, 3 patients died from inaccurate hemostasis results in 2001. Hospital closed in 2004.
  13. 13. 13 Key Role of Clinical Laboratory Key Role of Clinical Laboratory z Clinical laboratory costs are 3-4% of the total U.S. health-care budget. z At Mayo Clinic, 94% of the objective data in patients’ charts were from the laboratory. R. Forsman, Clinical Lab News, July 2004, p 12. z Lab data leverages 60-70% of all critical decisions. R. Forsman, Clinical Chemistry, 42, 813 (1996) z In other words, while we are but a small component of the total healthcare system, our data control a large majority of the critical decisions.
  14. 14. 14 Scope of TEa Problem Scope of TEa Problem z Object is to obtain values which are useful, defensible and attainable. z Many analytes (>1000) z Many classes of analytes. – Endogenous - Chemistry – Hematology (CBC, Diff) – Tox – TDM and DAU – PCR type tests (Viral loads) – Serology tests (Quant -> Qual result) – Urinalysis (Qual and semi-quant)
  15. 15. 15 Performance Standards Definitions Performance Standards Definitions z Three classes – Clinical – Establishment – Deployment
  16. 16. 16 Clinical Definition Clinical Definition z These define the clinical goals without specifying how they are to be accomplished. – Definition #1: The amount of error that can be tolerated without invalidating the medical usefulness of the analytical result" (Carey and Garber - 1989). – Definition #2: The amount of error which will allow one to calculate the probability that a change in clinical condition has occurred since that patient was tested previously. (Biological variation)
  17. 17. 17 Establishment Definitions Establishment Definitions z These definitions are used to define TEa values. (preferred definitions are higher on list.) – Medical Requirements – Biological Variation – Reference Interval – Regulatory Requirements (CLIA ’88 or EQAS) – Achievable Error • Peer Group Survey Results • Process SD • CLSI (formerly NCCLS) EP 21 – Manufacturer’s Claim for Method
  18. 18. 18 Deployment Definitions Deployment Definitions z These definitions define algorithms which convert a TEa into target SD’s for daily QC and allowable bias. – Allocation by fixed fraction • 25% Rule • Six Sigma – Specification of QC Rules based on magnitude of systematic and random errors. • Westgard Approach – Biological Variation
  19. 19. 19 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  20. 20. 20 Medical Requirements Theory Medical Requirements Theory z Several possible approaches to establishing quality specifications: – For specific clinical situations – Based on medical opinion – Based on national or international groups – Based on institutional or expert individuals
  21. 21. 21 Approach – Based on Medical Interpretation Requirements Approach – Based on Medical Interpretation Requirements z From NCEP – Cholesterol, HDL, and LDL z From the American Diabetic Association – Glucose and HbA1c z From NKEP – Creatinine z Others
  22. 22. 22 TEa from Med Req -- Examples TEa from Med Req -- Examples z Cholesterol – 5% vs 10% from CLIA ‘88 z Glucose – 10% (ADA??)
  23. 23. 23 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  24. 24. 24 Biological Variation - Explained Biological Variation - Explained Taken from Callum Fraser’s “Biological Variation: From Principle to Practice”
  25. 25. 25 Approach based on Biological Variation Approach based on Biological Variation z Are for inter- and intra- individual variations. z For most analytes, the intra-individual variation is less than the inter-individual variation. For a few tests, it is larger. z Due to the nature of the calculations, they are expressed in percent terms (% CV).
  26. 26. 26 BV TEa – Advantages BV TEa – Advantages z If one uses this approach, it is possible to calculate the probability that a significant clinical change has occurred in a patient based on the difference between his serial results. z Many workers in the field consider this to be the preferred approach.
  27. 27. 27 BV TEa – Issues BV TEa – Issues z Specifications currently available only for about 280 analytes out of a total lab menu of >1000 tests. z Assumes that care-givers know significance of this specification and are able to use it. z Not applicable for many analytes (i.e. drugs, certain disease markers, qualitative tests). z Some TEa’s are too large or too small. z Use of BV makes the assumption that it is constant across all populations.
  28. 28. 28 BV Sources BV Sources z Two sources (about 280 analytes): – Ricos et al (Westgard’s website – 270 tests) – Lacher et al (CCJ – Feb, 2005 – 42 tests)
  29. 29. 29 BV TEa – Examples BV TEa – Examples Desirable TEa (%) Ricos et al Lacher et al ALT 32.1 33.4 Albumin 3.9 4.6 Total Bilirubin 31.1 32.0 Calcium 2.4 4.2 Cholesterol 9.0 12.7 Glucose 6.9 10.6 Sodium 0.9 1.6
  30. 30. 30 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  31. 31. 31 TEa from Reference Intervals TEa from Reference Intervals z I have found two rules to calculate TEa as a fraction of a reference interval. z Tonk’s Rule TEa = 25% * RI z CLIA ’88 Rule TEa = 50% * RI – I propose that this 50% RI rule be established as the maximum value for TEa.
  32. 32. 32 Reference Intervals – Issues Reference Intervals – Issues z There are many tests for which RI’s are not available. In particular, these have cutoff values. Examples include Troponin, CKMB and HbA1c. z Most reference intervals are not industry standard. They vary between labs.
  33. 33. 33 TEa from Ref Intervals – Example TEa from Ref Intervals – Example Calcium TEa (mg/dL) Reference Interval Range Tonks Rule 25% of Range CLIA Rule 50% of Range 8.5 – 10.5 (Traditional) 2.0 0.5 mg/dL (0.5/9.5 = 5.3%) 1.0 (10.5%) 8.9 – 10.1 (Recent) 1.2 0.3 (3.2%) 0.6 (6.3%) z Major implications of this table with respect to setting routine QC target SD values, especially if 25% Rule is applied. – 1 SD would range from 0.8% (0.08 units) to 2.5% (0.25 units).
  34. 34. 34 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (i.e. CLIA ’88) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  35. 35. 35 PT Approach PT Approach z Values are available for about 75 analytes in USA. z Values were set administratively in early ’90’s so they may not be the best z However since these values exist and they are not grossly out of line with appropriate values, they are a good starting point. z When available, CLIA ’88 and EQAS values set upper limits on TEa.
  36. 36. 36 TEa from CLIA ’88 Regs – Examples TEa from CLIA ’88 Regs – Examples Analyte TEa - CLIA ’88 Limits Erythrocyte count (RBC) +/- 6% Prothrombin time +/- 15% Calcium +/- 1.0 mg/dL ALT (SGPT) +/- 20% Blood gas pO2 +/- 3 SD Glucose +/- 6 mg/dL or 10% (greater) HCG +/- 3 SD Digoxin +/- 20% or 0.2 ng/mL (greater)
  37. 37. 37 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  38. 38. 38 Achievable Total Error - 1 Achievable Total Error - 1 z This represents the actual observed total error of a method as compared with a goal established using clinical or population data. z This is the fall-back approach when the other methods are not available.
  39. 39. 39 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  40. 40. 40 TEa from Peer Group Surveys TEa from Peer Group Surveys z This concept is based on the PT specification of target +/- 3SD. z Preferable source is PT or EQAS results. Need number of eligible specimens to be at least 5, more can be used if available. z Fundamental advantage of this approach is its accessibility. It is available for almost every test performed in most clinical laboratories. z Disadvantage is that it is based on what can be done, not on medical or BV requirements.
  41. 41. 41 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  42. 42. 42 Process SD or CV Process SD or CV z Represents the actual performance of the process. z One must be careful to select values which truly represent the long-term precision of the method as obtained by conscientious workers. – It is entirely possible to use values here which mis- represent the precision of the system, so one must be very careful about which values to use.
  43. 43. 43 TEa from Process SD or CV values TEa from Process SD or CV values z Calculate TEa from a Process SD or CV. z This is the SD or CV observed over a period of time as judged by routine monthly QC results. z This is NOT necessarily the target SD used on the Levy-Jennings charts.
  44. 44. 44 TEa from Process SD or CV values – 2 TEa from Process SD or CV values – 2 z This is the lower limit for TEa. It defines what is routinely attainable by your method. z Keep in mind that this is what is attainable. Whether it is the ideal value is another issue.
  45. 45. 45 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  46. 46. 46 CLSI EP21 CLSI EP21 z This document defines an approach similar to an EP9 method comparison experiment to calculate a Total Achievable Error. z If you are interested in this approach, you may buy a spreadsheet implementing this document from CLSI for $350.
  47. 47. 47 Establishment Definitions Establishment Definitions z Medical Requirements z Biological Variation z Reference Interval z Regulatory Requirements (CLIA ’88 or EQAS) z Achievable Error – Peer Group Survey Results – Process SD – CLSI (formerly NCCLS) EP 21 z Manufacturer’s Claim for Method
  48. 48. 48 TEa from Manufacturer’s Claims TEa from Manufacturer’s Claims z Calculate TEa from manufacturer’s specifications for accuracy and precision. z This approach is to be used only when nothing else is available.
  49. 49. 49 Calculation Details Calculation Details z We will show calculation details for several data sources. z Issue: – What multiplier factor (Z score) should be used? TEa = SE + (Z * RE) • What confidence level should it reference? Some possibilities are 95%, 99.7%, 99.9997%. • Traditionally, our industry has used 95% confidence which corresponds to 50,000 errors per million. Six Sigma (99.9997%) places that rate at 3 errors per million.
  50. 50. 50 TEa from Medical Requirements TEa from Medical Requirements z If the Medical Requirement (MedReq) specifications include both an accuracy component (SE) and a precision component (RE), use the formula TEa = SE + (2 * RE) z If the specifications only include a precision component, use the formula TEa = 3 * RE – This is not as satisfactory as the first, but often those who create the specifications neglect the accuracy component.
  51. 51. 51 BV Calculations BV Calculations z The general equation is: TEa = f * ((0.125 * B) + (1.65 * 0.25 * P)) • Where B and P represent Bias and Precision components respectively. B and P are calculated from intra- and inter- individual biological variation. z 3 specifications: – Optimal (f= 1) – Desirable (f= 2) (Quoted on Westgard’s site) – Minimal. (f= 3)
  52. 52. 52 Error Profile Error Profile z Definition: The amount of random error observed or expected over the analytical range of an instrument. z It is expected that the amount of error will vary as the concentration varies.
  53. 53. 53 Idealized Error Profile Idealized Error Profile 0 5 10 15 20 25 30 10 100 1000 Concentration SD 0 5 10 15 20 25 CV SD CV
  54. 54. 54 Idealized Error Profile Idealized Error Profile 0 5 10 15 20 25 30 10 100 1000 Concentration SD 0 5 10 15 20 25 CV SD CV Concentration Percent
  55. 55. 55 TEa from Peer Group Surveys – 2 TEa from Peer Group Surveys – 2 z For each of a series of specimens for a given instrument, calculate the CV. Want the mean result to be in the mid- to upper-range for that analyte. Determine the median result. TEa = 3 * (median CV) – Use this approach to calculate the percent component of TEa.
  56. 56. 56 TEa from External QC results – 3 TEa from External QC results – 3 z Sometimes it is important to determine the concentration component of TEa. In that case, determine the median or representative SD for results near the low end of the reportable range. TEa = 3 * median SD
  57. 57. 57 TEa from External QC – CAP Survey Example TEa from External QC – CAP Survey Example HCG (VITROS ECi) n=63 Spec ID Mean SD CV C-11 26.97 1.65 6.1 C-12 68.29 4.54 6.6 C-13 90.61 6.39 7.1 C-14 52.13 3.57 6.8 C-15 82.47 4.84 5.9 Median CV
  58. 58. 58 Error Profile – Plot of CAP Survey Data Error Profile – Plot of CAP Survey Data HCG (VITROS ECi) 0 2 4 6 8 10 1 10 100 1000 Concentration (mIU/mL) SD 0.0 2.0 4.0 6.0 8.0 10.0 CV SD Range CV Reportable Range
  59. 59. 59 TEa from External QC – Example TEa from External QC – Example NYS PT Results - 2002-04 0 2 4 6 8 10 12 14 16 18 1 10 100 1000 HCG (mIU/mL) CV or SD SD % CV 1.6 units 5.6%
  60. 60. 60 TEa from External QC – Example TEa from External QC – Example z From CAP Results – TEa = 3 * Median CV = 3 * 6.6% = 19.8% = 20% z From NYS Results – TEa = 3 * Median CV = 3 * 5.6% =16.8% = 17% – TEa = 3 * Median SD = 3 * 1.6 = 4.8 units TEa = 5 units or 17% whichever is greater
  61. 61. 61 TEa from Manufacturer’s Claims TEa from Manufacturer’s Claims z From published specifications for an instrument or method. TEa = SE + (1.65 * RE) (95% confidence) TEa = 3 * RE (when SE is not specified)
  62. 62. 62 Calculating TEa – Rigorous approach Calculating TEa – Rigorous approach 1. Calculate minimum and maximum TEa’s. 2. Calculate TEa from the available resource highest on the list. 3. If selected TEa (sTEa) is between the minimum and maximum values then use it. – If sTEa is < minimum, use the minimum value. – If sTea is > maximum, use the maximum value.
  63. 63. 63 Calculating TEa – Easier Approach Calculating TEa – Easier Approach z If analyte has nationally specified medical requirements, use them!! z Otherwise, use TEa’s based on either CLIA PT limits or PGS whichever is smaller.
  64. 64. 64 Calculating the Minimum TEa Calculating the Minimum TEa z The absolute minimum TEa TEa = 3.0 * process SD or CV
  65. 65. 65 Calculating the Maximum TEa Calculating the Maximum TEa z TEa should never exceed 50% of the RI or the CLIA ’88 PT limits whichever is smaller.
  66. 66. 66 Our Data Sources Our Data Sources z Biological Variation (2 sources) – Ricos et al (Westgard’s website) 270 tests – Lacher et al (CCJ paper – Feb, 2005) 42 tests z External QC – Used NY State PT results for the instrument with the highest population for a given test. – TEa was calculated as 3 * median CV across all 5 specimens. We used results from mid- to upper- range specimens. z Process CV – Object was to obtain results which could be reasonably expected from a responsible clinical laboratory. – Median of monthly QC results for 1 - 10 Canadian labs over a period of more than 1 year. Data shown here are for the mid- or high- level control.
  67. 67. 67 Case Study: ALT Case Study: ALT z Medical Requirements: n/a z Biological Variation: – TEa: 32% z Reference Interval: – 10 to 50 IU/L z CLIA ’88 PT Limits: 20% z 3 * Peer CV: 4.6% (NYS PT Survey) z Process SD: 1.6%
  68. 68. 68 ALT TEa - Calculation ALT TEa - Calculation z Maximum TEa calculation – 50% of RI = (40 / 2) = 20 units or 67% at 30 units – CLIA ’88 PT limit: 20% – Since CLIA ’88 limit is smaller, • TEa (max) = 20% z Minimum TEa calculation – TEa (min) = Process CV * 3 = 3 * 1.6% = 4.8%. z BV TEa: 32% (greater than max TEa) z Set TEa to: 20%
  69. 69. 69 Case Study: Calcium Case Study: Calcium z Medical Requirements: n/a z Biological Variation: – TEa: 2.4% z Reference Interval: 8.9 – 10.1 mg/dL (Recent) z CLIA ’88 PT Limits: 1.0 mg/dL (10.6% at 9.5 mg/dL) z 3 * Peer CV: 7% (NYS PT Survey) z Process SD: 1.1%
  70. 70. 70 Calcium TEa - Calculation Calcium TEa - Calculation z TEa (max) = 50% of RI or PT value 50% of RI = 0.6 mg/dL (6.3%) PT value = 1.0 mg/dL (10.5%) z TEa (min) = 3 * Process SD = 3 * 1.1% = 3.3% z BV Goals – TEa = 2.4% z Set point for TEa 3.3% (round up to 4%?)
  71. 71. 71 Case Study – Total Protein Case Study – Total Protein z Medical Requirement: n/a z CLIA ’88 PT Limit: 10% z BV TEa: 3.4% z Reference interval: 6.7 to 8.1 g/dL z 3 * Peer CV: 5.3% z 3 * Process SD: 3.8%
  72. 72. 72 Total Protein – Calculations (Rigorous) Total Protein – Calculations (Rigorous) z TEa (max) = 50% of RI or PT limit 50% of RI = (8.1 – 6.7)/2 =1.4/2 = 0.7 = 0.7 / 7.4 = 9.5% PT limit = 10% z TEa (min) = 3 * Process CV = 3.8% z BV TEa: = 3.4% z BV TEa is < TEa (min) Set TEa to: 3.8%. Maybe round to 4.0%.
  73. 73. 73 Total Protein – Calculations (Easier) Total Protein – Calculations (Easier) z Medical Reqs: n/a z CLIA PT Limit: 10% z 3 * PGS: 5.3% z Set TEa to: 5.3% – Round up to: 6%
  74. 74. 74 Final Thoughts Final Thoughts z My object in this exercise has been to start a discussion on the approach that should be taken to defining performance standards for the general case. z I make no claim that what I propose here is the final answer. z Remember each lab serves its own customers, so the clinical use to which its data is used may be different from yours. z I welcome your ideas and discussion. z I trust that our collective wisdom will produce a far better solution than any one of us.
  75. 75. 75 Discussion Discussion z To discuss this presentation, send email to dgrhoads@dgrhoads.com z If there is sufficient interest, we will set up a discussion forum on our web site.

×