Three Coefficients for Analyzing the Reliability and Validity of Ratings

TLDR

Three numerical coefficients (V, R, and H) are introduced for assessing the validity and reliability of ratings. The study aims to apply these coefficients to validity, reliability, item analysis, agreement, and factor analysis of rating‑scale data. The coefficients are calculated as ratios of observed to maximum rating differences (or 1 minus that ratio), with probability tables generated by provided programs and z‑score approximations for large item or rater counts. The authors provide computer programs and tables of probabilities for the coefficients across various rating categories and item or rater counts.

Abstract

Three numerical coefficients (V, R, and H) for analyzing the validity and reliability of ratings are described. Each coefficient, which ranges in value from 0 to 1, is computed as the ratio of an obtained to a maximum sum of differences in ratings, or as 1 minus that ratio. Computer programs for calculating the coefficients, their associated individual and cumulative right-tail probabilities, and the population mean and standard deviation of each coefficient are available. Tables of the individual and right-tail probabilities for selected values of the three coefficients are included for 2 to 7 rating categories and 2 to 25 items or raters. More complete probability tables can be generated for any value of c, m, or n with one of the computer programs provided. When the number of items or raters is large (>25), the right-tail probability associated with any value of a coefficient may be estimated by a z-score procedure. The three coefficients are applicable not only to validity and reliability (test-retest and internal consistency) determinations but also to the item analysis, agreement analysis, and cluster or factor analysis of rating-scale data.