Rater Reliability: Short‐ and Long‐Term Effects of Calibration Training

Abstract

The purpose of this investigation was to evaluate the immediate effects of calibration on inter-rater agreement to a gold standard (GS) and to determine whether the effects can be sustained over a ten-week period. Valid criteria for a Class II amalgam preparation, a three-point rating scale, and a grade form were developed. Three tests were administered: prior to calibration training, immediately following training, and ten weeks later. Each test consisted of faculty independently evaluating ten prepared teeth. Agreement with GS scores for most of the grading criteria improved as a result of training and did not deteriorate over time. The overall percent agreement was 54.5, 66.9, and 64.6 percent across test periods. The most impressive gains in agreement occurred when the criteria evaluated had a GS score of either "standard not met" or "ideal." There was very little gain when the gold standard score was "acceptable." It is concluded that, with training, inter-rater agreement with a gold standard can improve and such improvement is reasonably resistant to deterioration after ten weeks. Nevertheless, future training ought to consider the use of a mastery approach in calibration training to ensure that a satisfactory degree of agreement with the GS is obtained.