Educational Assessment · 2015 · 17 citations · 22 references
EducationEducational AssessmentEducational MeasurementAssessment FairnessStandard-settingSummative AssessmentAngoff MethodAssessment ValidityStandard-setting For Certification ExaminationsStandard-setting For Licensure ExaminationsStandard-setting ReliabilityStandard-setting ValidityAngoff Method InterpretationAngoff Panel ConsistencyAssessmentsPanel Judgment ConsistencyPassing ScoresRecommended Passing ScoreStandard Setting PanelsStandard-setting Validity Evidence
Angoff-based standard setting is widely used, especially for high-stakes licensure assessments. Nonetheless, some critics have claimed that the judgment task is too cognitively complex for panelists, whereas others have explicitly challenged the consistency in (replicability of) standard-setting outcomes. Evidence of consistency in item judgments and passing scores is necessary to justify using the passing scores for consequential decisions. Few studies, however, have directly evaluated consistency across different standard-setting panels. The purpose of this study was to investigate consistency of Angoff-based standard-setting judgments and passing scores across 9 different educator licensure assessments. Two independent, multistate panels of educators were formed to recommend the passing score for each assessment, with each panel engaging in 2 rounds of judgments. Multiple measures of consistency were applied to each round of judgments. The results provide positive evidence of the consistency in judgments and passing scores.
22