Certification Testing as an Illustration of Argument-Based Validation

Abstract

The theories of validity developed over the past 60 years are quite sophisticated, but the methodology of validity is not generally very effective. The validity evidence for major testing programs is typically much weaker than the evidence for more technical characteristics such as reliability. In addition, most validation efforts have a strong confirmationist bias. The argument-based approach to validation is intended to provide a systematic approach to validation that can be applied to a wide range of interpretations and uses of test scores. This approach involves two kinds of argument, an interpretive argument and a validity argument. The interpretive argument specifies the proposed interpretation and use of the test scores. The validity argument provides a critical evaluation of the overall plausibility of the interpretive argument and, where necessary, empirical checks on its inferences and assumptions. The interpretive argument is an informal, presumptive argument that employs qualitative inferences as well as formal deduction and mathematical analysis. The validity argument can validate the proposed interpretation and use by providing adequate backing for the inferences in the interpretive argument, or it can invalidate the proposed interpretation by showing that at least one inference in the interpretive argument does not work. The argument-based approach to validation is illustrated by a detailed analysis of the validity of test-based certification decisions.