Use and Interpretation of Common Statistical Tests in Method-Comparison Studies

TLDR

Decisions on method acceptability should be based on judgments of tolerable errors. The study simulated various error types in test data to assess statistical parameter sensitivity and applied tests to provide specific error estimates. Least‑squares analysis provides specific estimates of proportional, constant, and random errors and should precede t‑tests, which only estimate constant and random errors when proportional error is absent; the correlation coefficient is sensitive only to random error and, together with r, t, and F, is not useful for deciding method acceptability.

Abstract

Abstract We have studied the usefulness of common statistical tests as applied to method comparison studies. We simulated different types of errors in test sets of data to determine the sensitivity of different statistical parameters. Least-squares parameters (slope of least-squares line, its y intercept, and the standard error of estimate in the y direction) provide specific estimates of proportional, constant, and random errors, but comparison data must be presented graphically to detect limitations caused by nonlinearity and errant points. t-test parameters ( bias, standard deviation of difference) provide estimates of constant and random errors, but only when proportional error is absent. Least-squares analysis can estimate proportional error and should be considered a prerequisite to t-test analysis. The correlation coefficient (r) is sensitive only to random error, but is not easily interpreted. Values for r, t, and F are not useful in making decisions on the acceptability of performance. These decisions should be judgments on the errors that are tolerable. Statistical tests can be applied in a manner that provides specific estimates of these errors