Skip to main content
Chemistry LibreTexts

4.41: Methods Evaluation

  • Page ID
    123345
  • \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

    A new procedure is being introduced in your laboratory to replace one in current use. The new assay is considered to be more accurate based upon literature reports as well as survey data (i.e. CAP and AACC surveys). The clinicians who will most likely utilize this assay have suggested that at a concentration of the analyte (Xc) of 1000 mg/L, i.e. a medical decision level, they would like a precision of 75 mg/L (EA) since, at changes of this magnitude, they will begin to treat the patient. The upper limit of the reference range of this analyte is 0.9 gm/L and you would like a range of linearity up to 5 gm/L. A series of experiments were run to validate the method.

    Experiment 1. The following data were obtained on aqueous standards:

    Concentration in mg/L Absorbance Units
    400 0.075
    800 0.148
    1200 0.227
    2000 0.370
    4000 0.650
    6000 0.950

    QUESTIONS

    1. What is the linearity of this assay? If the linearity is unacceptable to you, how can the assay be modified to increase the linearity?
    2. Experiment 2. A quality control pool was analyzed 26 times within a single run. The calculated standard deviation (SD or STM, see p 346, Chapter 19 or p 422, Chapter 22) was 28 mg/L at a mean analyte value of 1000 mg/L. Is the within-run precision acceptable at a 95% confidence limit?
    3. Without doing a calculation, do you consider the between-run and within-run data to be consistent? If so, why? If not, why not?
    4. Experiment 3. A clear non-hemolyzed serum pool ("ZERO") is spiked with Intralipid and a stock hemolysate to give a series of samples with increasing concentration of interferent. The samples are analyzed as six time replicates with the following results:
    Sample Mean
    (mg/L)
    Standard Deviation
    (mg/L)
    Difference from ZERO
    (mg/L)
    hemolysate, Intralipid 950 30
    Slightly hemolyzed 928 29 -22
    Moderately hemolyzed 897 31 -53
    Grossly hemolyzed 827 28 -153
    Sample Mean
    (mg/L)
    Standard Deviation
    (mg/L)
    Difference from ZERO
    (mg/L)
    Slightly turbid 24 15
    Moderately turbid 983 37 33
    Grossly turbid 1025 45 75
    Slightly icteric 937 29 -13
    Moderately icteric 968 30 18
    Grossly icteric 952 34 2

    Do any of these substances cause a significant interference? If so, at what level does significant interference occur?

    1. Experiment 4. Fifty patient samples are analyzed by the current glucose method and the data (in mg/L) are given below. All these specimens will then be analyzed by the current method in a correlation experiment.

      720, 1000, 430, 1900, 450, 550, 520, 800, 4100, 880, 630, 405, 500, 460, 420, 304, 400, 599, 563, 338, 445, 1167, 2700, 470, 425, 575, 820, 500, 640, 450, 538, 419, 495, 360, 955, 470, 378, 745, 1355, 401, 525, 705, 355, 480, 359, 437, 900, 446, 410, 525

      Should all these samples be used? If so, why? If not, why not?
    2. What further steps should be taken to increase the confidence of the correlation data?

    Question to Consider

    1. How can one determine the linearity of the assay?
    2. How is the total random error of the assay determined and evaluated?
    3. The same sample is analyzed on 20 successive days, and the between-run precision is calculated to be 26 mg/L. Do the between-run precision data meet your criteria for error at a 95% confidence limit?
    4. How can one determine the extent of an interference?
    Answer
    1. The linearity of this assay does NOT meet the expected value of 5 g/L and therefore is unacceptable. To increase the linearity, decrease the sample size or increase the total volume of the reaction mixture by increasing the reagent volume in the reaction mixture.
    2. $$\begin{split} RE &= 4\; S_{TM} \\ &= 4 (28\; mg/L) \\ &= 112\; mg/L \end{split}$$This calculated point estimate random error is greater than the allowable error of 75 mg/L and therefore the point estimate of the random error is not acceptable. The 95% confidence limit of the acceptability of this error is derived by calculating the 5% and 95% interval around the STM from a one-sided confidence limit table and again taking two standard deviations around these values (see example, p 422, Chapter 22).$$\begin{split} S_{TM1}&= S_{TM} (A\; 0.05) \\ &= 28.0\; mg/L (0.8149) \\ &= 22.8\; mg/L \end{split}$$$$\begin{split} S_{TMu}&= S_{TM} (A\; 0.95) \\ &= 28.0\; mg/L (1.308) \\ &= 36.6\; mg/L \end{split}$$$$\begin{split} RE_{1} &= 4\; S_{TM1} \\ &= 4 (22.8\; mg/L) \\ &= 91.2\; mg/L \end{split}$$$$\begin{split} RE_{u} &= 4\; S_{TMu} \\ &= 4 (36.6\; mg/L) \\ &= 146.4\; mg/L \end{split}$$Both the upper and lower limits of the RE values are greater than the desired EA and therefore one is 95% confident that the estimated error is greater than the allowable limits.
    3. No! The within-run component of the random error should be smaller than the between-day error component since there are less variables to introduce error within a single run than over a many-day period. Remember, the between-day error includes the within-run component. Therefore, one should look at this data with a critical eye and assume that one of the data sets is incorrect and should be repeated. Since the calcualted estimates of random error exceed teh desired limits, the laboratory either need to speak with the clinicians about revising the limit or else speak with the manufacturer of the assay to discuss ways to improve the assay.
    4. Hemolysis constant error due to interference by hemolysis exceeds the acceptable limit of 75 mg/L with the gross hemolysis sample, and therefore grossly hemolyzed samples would be unacceptable for analysis. The remaining question requires determination of a confidence limit for acceptance of the slightly hemolyzed and moderately hemolyzed samples. These can be calculated using a confidence limits ( p. 422-425): $$\begin{split} CE_{U} &= CE + \frac{ts}{\sqrt{N}} \\ &= 53\; mg/L + \frac{2.089 (31\; mg/L)}{\sqrt{6}} \\ &= 79\; mg/L \end{split}$$$$\begin{split} CE_{1} &= CE - \frac{ts}{\sqrt{N}} \\ &= 53\; mg/L - \frac{2.089 (31\; mg/L)}{\sqrt{6}} \\ &= 27\; mg/L \end{split}$$Moderate hemolysis conclusion: Since the CEu exceeds the acceptable limits but the CE1 does not, one is not 95% confident that the moderately hemolyzed samples do not cause significant interference. The question of sample acceptance cannot be statistically answered at this point. The question may be resolved by repeating the experiment with a larger number of analyses or on clinical grounds.

      Slight hemolysis conclusion: One can accept slightly hemolyzed samples with a 95% confidence limit that the error is within the allowable limits (75 mg/L).

      Turbidity: Grossly turbid samples are obviously unacceptable as the point error introduced is at the limit of acceptable error (75 mg/L). For moderate turbidity, the calculation is as follows: $$\begin{split} CE_{u} &= CE + ts \\ &= 33\; mg/L +\frac{ 2.089 (37\; mg/L)}{\sqrt{6}} \\ &= 64.6\; mg/L \end{split}$$Since the upper limit of error is within the acceptable error limit, one can be 95% confident that the error introduced by moderate turbidity is acceptable.

      Slight turbidity. Same conclusion as above for slight hemolysis.
    5. No! When collecting sample for this type of experiment it is important to include samples from many types of patients (see pp 414-415) and with analyte levels over as wide a range as possible (see p 416). That is, the samples should not be choosen at random, but purposely picked with these criteria in mind. The bulk of the data presented for this experiment falls below the upper limit of the reference range (900 mg/L) and does not include enough patient samples above this point. Insufficient data points within the entire dynamic working range can result in incorrect regression analysis including a falsely low correlation coefficient (r, see p 416).
    6. Go back and pick out a number of specimens whose values fall between 1000 and 5000 mg/L. One approach may be to divide the desired range of sample over the linear range of the assays, that is into perhaps 8 to 10 groups. In this case, if the linear range were up to 5000 mg/L, one might divide it into groups of 500 mg/L. Thus, one would choose samples in the range of 0-500, 501-1000, 1001-1500, etc. for analysis by the new method. Use of 10 samples in each of these groups would give an ideal range of data for analysis. If fewer samples are readily available in one or more groups, this must be taken into consideration when analyzing the data.

    Answers to Questions to Consider

    1. There are two ways of determining the linearity of an assay. One can perform a linear regression analysis (Chapters 19, pp 357-361, and 22, pp 415-418). Usually a regression coefficient ("r") of > 0.999 indicates a linear relationship. However, it is easy to be mislead by the linear regression analysis (see p 416), and one should always plot and visually inspect the data, looking for a deviation from a straight line.


      The "r" value for the data by regression analysis was 0.9987, a fairly high value that would suggest good linearity. However, a plot the actual results obtained from the analysis vs. the expected results of the standards suggests a possible problem.

    ed475e717d0e877e8adce5c30dd782b47.png

    The region of the curve that deviates from the straight line of the initial data points is the concentration at which the method loses its linear response; that is, between 2000 to 4000 mg/L. One might want to redo the experiment with some points between 2 and 4 mg/L to get a more specific estimate of the upper limit of linearity.

    1. The point estimate of the random error (RE) is calculated by taking two standard deviations around the mean SD (Chapter 22): RE = 1.96 (STM). It is suggested (Table 22-4, p 421) that acceptable RE be: 4STM < EA. To determine the 95% confidence limit about this point estimate of random error, use the equations found on p 419, Chapter 22): $$S_{TMl} = S_{TM} (A\; 0.05) S_{TMu} = S_{TM} (A\; 0.95) \ldotp$$
    2. The point estimate of a constant error ('bias') is simply the DIFFERENCE between the spiked and unspiked ('ZERO') samples. The confidence limits of the point estimate is calculated as shown for the example on p 422-423, Chapter 22.
    3. The point estimate of error exceeds the allowable limit of error (75 mg/L) and one would have to assume that the new method has an unacceptable bias compared to the older method. However, the currently used method is not a reference method and therefore one cannot assume that the bias between the methods is an accurate criteria for rejection of this method. One would need additional information concerning the two methods in which both are compared to an acceptable reference method. At this point, one can accept the method on other grounds.

    This page titled 4.41: Methods Evaluation is shared under a CC BY-NC-SA 4.0 license and was authored, remixed, and/or curated by Lawrence Kaplan & Amadeo Pesce.

    • Was this article helpful?