Threats to the Valid Use of Assessments

TLDR

Validity is the most important quality of an assessment, yet its evaluation is often neglected. The authors present a step‑by‑step approach that offers structured guidance to validators of educational assessments, with illustrative threat sets that are not exhaustive. Validity is conceptualized as a chain of eight linked stages—administration, scoring, aggregation, generalization, extrapolation, evaluation, decision, and impact—each of which is examined for specific threats. The model shows that validity is constrained by the weakest link, making over‑strengthening other stages wasteful or harmful, and it proves useful for planning assessments.

Abstract

Abstract Validity is the most important quality of an assessment, but its evaluation is often neglected. The step‐by‐step approach suggested here provides structured guidance to validators of educational assessments. Assessment is depicted as a chain of eight linked stages: administration, scoring, aggregation, generalization, extrapolation, evaluation, decision and impact. Evaluating validity requires careful consideration of threats to validity associated with each link. Several threats are described and exemplified for each link. These sets of threats are intended to be illustrative rather than comprehensive. The chain model suggests that validity is limited by the weakest link, and that efforts to make other links particularly strong may be wasteful or even harmful. The chain model and list of threats is also shown to be valuable when planning assessments.