This document defines various terms related to method evaluation in clinical laboratories, including accuracy, precision, bias, systematic and random errors, linearity, and medical decision levels. It discusses the purpose of method evaluation for laboratory, manufacturer and medical requirements. Key aspects of evaluating a new method include assessing stability, linearity, and sources of random and systematic error through experiments. The goal is to determine if a method's total error meets performance criteria for clinical use.

2. Dr Gilani STAS CHP AFIP Dec 09

3. Basic terminologies
 Accuracy
 Agreement between the mean estimate of a quantity and its true value
 Inaccuracy
 The systematic error, estimated from the mean of a set of data relative to the true value or
difference between the observed value and true or assigned value
 Precision
 Repeatability: closeness of agreement between results of successive measurements
carried out under the same conditions (within-run precision)
 Intermediate precision (long term)
 Reproducibility: closeness of agreement between results of measurements performed
under changed conditions of measurements (interlaboratory)
 Imprecision
 The standard deviation or coefficient of variation in the set of replicate measurement. Or a
measure of the dispersion of random errors
 Limit of detection (LOD)
 The minimum concentration of analyte whose presence can be quantitatively detected
under defined conditions.
 Replication experiment
 An evaluation experiment that estimates analytical error. Measurements are made aliquots
of a stable sample over specified periods of time, as within a run, within a day, or over a
period of days

4. Precision & Accuracy

5. Basic terminologies(Cont…)
 Bias
 The difference between two quantities or a measure of inaccuracy
 Constant systematic error
 An error which is always in the same direction and magnitude, even as the concentration of analyte
changes
 Interference
 The effect of a component on the accuracy of measurement of the desired analyte
 Interference experiment
 An evaluation experiment that estimates the systematic error in a method resulting from interference or
lack of specificity
 Linear regression
 An approach to choose a single line that best subscribe the relation between two subsets of two methods
 Proportional systematic error
 An error that is always in one direction and whose magnitude is a percentage of the concentration of
analyte being measured
 Recovery Experiment
 An evaluation experiment that estimates proportional systematic error. The amount of analyte recovered
is divided by the amount of analyte added to a sample, and the ratio is expressed as the percentage of
recovery. The deviation of the percentage of recovery from 100% is the proportional error
 Comparative method
 The analytical method to which the test method is compared
 Comparison of methods experiment
 An evaluation experiment in method which a series of patient samples are analyzed by both the test
method and comparative method. The results are assessed to determine whether differences exist
between the two methods.

6. Basic terminologies(Cont…)
 Total error
 Random + Systematic analytical error
 Allowable error(EA)____5%
 The amount of error that can be tolerated without
invalidating the medical usefulness of the analytical
result
 Medical decision level(Xc)
 A concentration of analyte at which some medical
action is indicated

7. Total Analytical error TEA.
RE
SE
TEA = RE + SE
TEA

8. Evaluation of methods
 Purpose
 Laboratory requirements
 Manufacturer requirements
 Medical requirements
 Selection of methods
 Evaluation of need
 Define requirements
 Application characteristics
 Method characteristics
 Analytical performance characteristics
 Review literature
 Select candidate method
 Laboratory evaluation of a method
 Familiarization
 Stability
 Linearity
 Random and systematic error
 Experiments to estimate magnitude of specific errors
 Random error replication studies
 Constant error interference
 Proportional error a recovery experiment
 Final evaluation experiments
 Between day replication experiment
 Comparison of methods experiment

9. Purpose of method evaluation
 Laboratory requirements
 New analytical methods are usually develop to improve accuracy or precision
over existing methods
 To assess inherit analytical errors of the methods and relate them to medical
requirements
 Manufacturer requirements
 When a manufacture develops a new method and prepares to market it, the
manufacture is required by the FDA to make claim about analytical
performance of the method, its precision and accuracy
 This claim must be supported by experimental method evaluation data
 Method evaluation studies are also performed by some organization to verify a
manufacture claim of analytical performance of a method (FDA, NCCLS)
 Most method evaluations are performed by users
 Medical requirements
 Ability of the method to meet the requirements of the user, the physician who is
interpreting the results of a laboratory test of a patient
 The error of the test result is excessive if it causes a misdiagnosis
 Performance standard may be formulated, consisting of the decision level
concentration(Xc), and the allowable error(EA )
 Medical decision levels concentrations proposed that allowable error be either
¼ of the normal range or 10%, whichever is less

10. Establish need
Method selection
Definition of
quality goal
Method
evaluation
Method
development
Implementation
Routine
analysis
Quality control
practices
Submission of
specimen
Result
report

11. Selection of methods
 Evaluation of need
 Medical requirement for a new test
 Advances in laboratory practice
 Age and lack of operational reliability of the present analyzer
 Define requirements
 Application characteristics
 Practical features required of the method
 Sample size, turnaround time, sample throughput rate, specimen type, automated
calibration, online quality control review, self diagnostics, laboratory space required,
reagent storage facility required, availability and skill of lab staff, time available, cost per
test, safety and environmental hazards
 Method characteristics
 Chemical specificity
 Chemical sensitivity
 To use primary aqueous standards for calibration
 The choice of reagents temperature, reaction time, measurement time, measurement
approach
A source of recommended principal method has been developed by NCCLS
 Analytical performance characteristics
Overall goals for analytical performance have been discussed in terms of
 Allowable error, working range of a method(linearity), stability of the reagents, ability
of the analyzer to detect reagent depletion in the case of enzyme substrates, expected
reference range, amount of error caused by interfering substances, precision and
accuracy of the method
 Review literature
 Select candidate method

12. Factors in method selection

13. The analytical goal can be defined as
“the allowable total error “

15. Analytical Goals of various analytes

16. Analytical Goals
Analyte Acceptable
performance
criteria
(CLIA 88)
Decision level
XC
Allowable
error
(CLIA 88)
Maximum
sd
(CLIA 88)
(CV%)
Medically based
maximum sd
(Fraser)
(CV%)
Albumin ± 10% 35 g/L 3.5 0.9 (2.6%) 0.5 (1.43%)
Cholesterol ± 10% 5.2 mmol/L 0.52 0.13 (2.5%) 0.14 (2.7%)
Creatinine ± 15%
88 µmol/L
265 µmol/L
26
40
7.0 (8%)
9.7 (3.7%)
1.8 (2.0%)
6.2 (2.3%)
Glucose ± 10%
2.75 mmol/L
6.9 mmol/L
11.0 mmol/L
0.33
0.69
1.10
0.08 (2.9%)
0.18 (2.6%)
0.28
(2.55%)
0.06 (2.2%)
0.15 (2.2%)
0.24 (2.2%)
Hb A1C 7.0% 0.35% 0.14%
K ± 0.5 mmol/L
3.0 mmol/L
6.0 mmol/L
0.50
0.50
0.12 (4%)
0.12 (2%)
0.07 (2.33%)
0.14 (2.33%)
ALP ± 30% 150 U/L 45 11 (7.3%) 5.1 (3.4%)
CK ± 30% 200 U/L 60 15 (7.5%) 40 (20%)
Analytical Goals of various analytes

17. Laboratory evaluation of a method
 Familiarization
 Stability
 Reagents, calibrators and control material
 Expiry dates
 One should perform preliminary studies with crossover analysis
 One can test the observed difference by use of a t test
 Linearity
 The range of concentration or other quantity in the specimen over
which the method is applicable without modification
 The absolute minimum no of different concentrations that must be
measured is 3, better is 5
 Duplicate measurement should be made on each concentration sample
 Construct a dilution series
 Each sample should be prepared by direct dilution from the original
high sample pool
 Data points should be plotted for visual inspection of linear
performance

18. Laboratory evaluation of a
method(Cont…)
 Random and systematic error
Errors that effect the performance of the analytical procedures are random,
systematic
 Random error
 RE are those that effect the reproducibility of the measurements. These include
 Instability of the instrument
 Variation in temperature
 Variation in reagents and calibrators
 Variability in handling techniques such as pipetting, mixing and timing
 Variability in operators
 Within-run component of variation (σwr) is caused by
 pipetting, precision and short term variation in the temperature and stability of the
instrument
 Within-day, between-run variation (σbr) is caused by
 differences in recalibration, longer-term variations in the instruments, small changes in
the condition of the calibrator and reagents, changes in the condition of the laboratory
during the day and fatigue of the laboratory staff
 Between-day component of variation (σbd) is caused by
 Variation in the instrument that occur over days, changes in calibrators and reagents (new
vials)and changes in staff
 Total variance of method (σt
2 )
σt
2 = σwr + σbr +σbd
 Term used to indicate random error include precision, imprecision, reproducibility and
repeatability. In each case the referred to the random dispersion of results or
measurements around some point of central tendency

19. Total Variance of a Method (t
2)
t
2 = wr
2 + br
2 + bd
2
RE = t

20.  Systematic errors
 Constant systematic error
 Error that is consistently high or low by the same amount, regardless of
the concentration of analyte
 Interfering substances
 Reaction B/W Interfering substances and the reagents caused by a lack of
specificity
 Interfering substance that effect the reaction b/w the analyte and the
reagent(enzymatic methods)
 Improper blanking of sample or reagent
 Proportional systematic error
 Error that is consistently high or low
by an amount proportional to the
concentration of the analyte
 Erroneous caliberation
 Side reaction for the analyte
Laboratory evaluation of a
method(Cont…)

22. Experiments to estimate
magnitude of specific errors
 The aim of this section is to describe specific experiments that will enable one
to estimate the magnitude of a specific error
 One can then compare the size of the error to the allowable error to determine
the acceptability of the method

23. Experiments to estimate magnitude
of specific errors(Cont…)
 Random error estimated from REPLICATION
STUDIES
 Within-run replication experiment
 Assessment of precision
 Short term performance
 Replication study should be performed with
 Aqueous solution of calibrator and then repeat with samples
whose matrices is as similar as possible to that of the intended
patients sample
 Concentration should be near the medical concentration
 Repeated analysis of the same specimen
 If the estimate of the random error is less than allowable error,
it is acceptable

24. Experiments to estimate magnitude
of specific errors(Cont…)
 Constant error estimated from INTERFERENCE STUDIES
 The interference study measures the constant error caused by the
presence of a substance suspected of interfering with the test method
 A sample is spiked with the substance suspected of interfering . To
compensate for the dilution of the spiked sample a baseline sample
should be prepared by addition of the solvent used for the interferent,
to another aliquot of the sample. The 2 samples should be analyzed at
least in duplicate the difference b/w the results in the 2 samples is
attributable to an interference caused by the added substance
 Example of the Interfering substances are haemolysed, lipemic and
icteric samples
 If the error is too large at the maximum concentration of interfering
substance it may be appropriate to test the interference at lower
concentration
 The overall average difference(bias) is called a constant error because it
is independent of the analyte conc.
 If the constant error is less than the allowable error the constant error
caused by the interference is judged acceptable

25. Experiments to estimate magnitude
of specific errors(Cont…)
 Proportional error estimated from a RECOVERY
EXPERIMENT
 Sample is divided into two aliquot 1 is spiked with analyte and
equivalent amount of diluent is added to the second(baseline
sample)
 Two samples are then analyzed
 The difference b/w the spiked sample and the baseline sample
indicates the amount of analyte “recovered”
 The volume of the analyte added to the sample should be less than
10 % to avoid major disruption
 Analysis of this sample with the comparison method is
recommended
 The ratio of the amount recovered to the amount added and is given
as percentage the difference b/w the calculated percentage of
recovery and 100% recovery is the percentage of the proportional
error, the standard deviation of the percentage of the recovery is a
measure of the uncertainty of percentage of proportional error
 If the proportional error is less than the allowable error, the
proportional error is acceptable

26. Final evaluation experiments
 B/w-day replication experiment
 Expansion of the within-run experiment over many days usually 20
 Comparison of methods experiment
 Systematic error of the test method and a comparative method
 A group of patient specimens are analyzed by both the test method and a comparative
method
 Systematic differences b/w the two methods are interpreted as the error of the test method
 Methods may be classified in terms of the quality of their performance
 Definitive, reference or routine methods
 Comparative method Should be of the highest quality possible
 At least 40 and preferably 100 specimens should be analyzed
 Analyte concentration throughout the analytical range
 Specimens are analyzed for duplicate in each method
 The test and comparative method should be run at the same time or as closely in time to
each other as possible
 The comparison of method experiment is usually combined with the b/w-day replication
experiment
 Patients specimens should be evenly spread over atleast 5 runs and preferably all 20 runs
 Both methods must be maintained in acceptable quality control during the period

27. Final evaluation experiments(Cont..)
 Comparison of methods experiment(Cont..)
t-Test statistics
 Bias
 Systematic differences b/w the test and the comparative methods are most easily
estimated from the comparison of method data by the bias
 Bias is the difference b/w the average result by the test method and the average result
by the comparative method or is the magnitude of the systematic error b/w the two
methods
 Bias= ∑(yi-xi)
N
 yi and xi are the analyte concentrations of the individuals specimens by the test method
and comparative method respectively and N is the no of paired results compared
 Standard deviation
SD = ∑(yi-xi -Bias)2
N-1
 t-Test
 The statistical significance of the bias, that is weather it really differs from 0 or no bias,
is determine by the use of t value
 t value is the ratio of the systematic error term(bias) to a random error term (SD)
t= bias N
SD

28. Comparison of methods experiment(Cont..)
t-Test statistics
If the calculated t value exceeds the critical t value(t table), statistically real bias
exist b/w the two methods
If bias is less than allowable error the systematic error is acceptable
Final evaluation
experiments(Cont..)

29. Recommended Minimum Studies for comparison of
methods experiment.
1. Select 40 patient specimens to cover the full working
range of the method.
2. Analyze 8 specimens a day within 2 hours by the test and
comparative methods.
3. Graph results immediately on a difference plot and
inspect for discrepancies.
4. Reanalyze specimens that give discrepant results.
5. Continue the experiment for 5 days if no discrepant
results are observed.

30. Recommended Minimum Studies for comparison of methods
experiment.
6. Continue for another 5 days if discrepancies are observed
during the first 5 days.
7. Prepare a comparison plot of all the data to assess the
range, outliers, and linearity.
8. Calculate the correlation coefficient and if 0.99 or greater,
calculate simple linear regression statistics and estimate
the systematic error at medical decision concentrations.
9. Use the medical decision chart to combine the estimates of
SE and RE and make judgment on the total error observed
for the method.

31. Dr.Sarma 57
Thanks
Everyone