Setting Response Time Thresholds for a CAT Item Pool: The Normative Threshold Method

Abstract

Identification of rapid-guessing test-taking behavior is important because it indicates the presence of noneffortful item responses that can negatively bias examinee proficiency estimates. Identifying rapid guesses requires the specification of a time threshold for each item. This requirement can be practically challenging for computerized adaptive tests (CATs), which use item pools that may contain thousands of items. This study investigated a new method—termed the normative threshold method—for identifying item thresholds. The results showed that the normative threshold method could markedly outperform a common three-second threshold (which has been used in previous research) in identifying non-effortful behavior.