Ability of Panelists to Estimate Item Performance for a Target Group of Candidates: An Issue in Judgmental Standard Setting

Abstract

Recent researchers (Impara & Plake, 1998; National Research Council, 1999; Shepard, 1995) have called into question the ability of judges to make accurate item performance estimates for target subgroups of candidates, such as minimally competent candidates. The purpose of this study was to examine both the reliability and accuracy of item performance estimates from an Angoff (1971) standard setting application. Results provide evidence that item performance estimates were both reasonable and reliable. Factors that might have influenced these results are discussed.