A Comparison of Web-Based Standard Setting and Monitored Standard Setting

Abstract

Standard setting, when carefully done, can be an expensive and time-consuming process. The modified Angoff method and the benchmark method, as utilized in this study, employ representative panels of judges to provide recommended passing scores to standard setting decision-makers. It has been considered preferable to have the judges meet in a central location to complete the standard setting and take part in the discussion. The use of a central location, however, adds travel costs and time. The World Wide Web allows for training, discussion, and data collection without requiring judges and the measurement of professionals running the study (facilitators) to travel to a central location. A Web-based standard setting system was developed to offset the costs of travel for a typical, monitored standard setting, and to improve the standardization of the training materials. A study using the Web-based system was compared to a similar study using a monitored session. The results of this study are generally favorable toward using a Web-based standard setting system, and suggest that recommended passing scores from an Internet study will be similar to those from a monitored study. A future version of the system should improve on the communication between the judges, and between the judges and facilitator. However, the teachers in this study felt similarly about the overall experience, regardless of whether it was an Internet or face-to-face study. (AEF) Reproductions supplied by EDRS are the best that can be made from the original document.