Taking the Time to Improve the Validity of Low‐Stakes Tests: The Effort‐Monitoring CBT

Abstract

The attractiveness of computer‐based tests (CBTs) is due largely to their capability to expand the ways we conduct testing. A relatively unexplored application, however, is actively using the computer to reduce construct‐irrelevant variance while a test is being administered. This investigation introduces the effort‐monitoring CBT, in which the computer monitors examinee effort (based on item response time) in a low‐stakes test and displays warning messages to those exhibiting rapid‐guessing behavior. The results of an experimental study are presented, which showed that an effort‐monitoring CBT increased examinee effort and yielded more valid test scores than a conventional CBT. Thus, unlike previous research that has focused on identifying rapid‐guessing behavior after it has occurred, the effort‐monitoring CBT proactively attempts to suppress rapid‐guessing behavior. This innovative testing procedure extends the capabilities of measurement practitioners to manage the psychometric challenges posed by unmotivated examinees.