Journal of Educational Measurement · 2002 · 34 citations · 16 references
EducationEducational AssessmentEducational MeasurementPerformance AssessmentPsychometricsStandard-settingCut Score InterpretationFeedback In AssessmentStandard-setting ReliabilityStandard-setting ValidityStandards-based AssessmentAngoff Method InterpretationClassical Angoff ProcedureCriterion-referenced Cut ScoresCut Score GeneralizabilityCut Score PrecisionCut Score Standard SettingDecision Cut ScoresMultivariate Generalizability Analysis
Cut scores, estimated using the Angoff procedure, are routinely used to make high‐stakes classification decisions based on examinee scores. Precision is necessary in estimation of cut scores because of the importance of these decisions. Although much has been written about how these procedures should be implemented, there is relatively little literature providing empirical support for specific approaches to providing training and feedback to standard‐setting judges. This article presents a multivariate generalizability analysis designed to examine the impact of training and feedback on various sources of error in estimation of cut scores for a standard‐setting procedure in which multiple independent groups completed the judgments. The results indicate that after training, there was little improvement in the ability of judges to rank order items by difficulty but there was a substantial improvement in inter‐judge consistency in centering ratings. The results also show a substantial group effect. Consistent with this result, the direction of change for the estimated cut score was shown to be group dependent.
16
Educational measurement, 3rd ed.
Robert L. Linn · 1989 · 2.9K citations
Education, Educational Assessment, Educational Measurement +6
Lincoln E. Moses, E. F. Lindquist · Journal of the American Statistical Association · 1951 · 1.7K citations
Certification of student competence.
Richard M. Jaeger · 1989 · 182 citations