Generalizability Analysis for Performance Assessments of Student Achievement or School Effectiveness

TLDR

School and individual achievement scores carry uncertainty that must be quantified, and generalizability analysis offers a standard‑error based indicator while accounting for pupil nesting and population scope. The study extends generalizability analysis to accommodate non‑traditional assessment designs. They show that what is often attributed to pupil‑by‑task interaction is partly due to unstable pupil performance.

Abstract

Evidence of the uncertainty attached to school and individual scores is required to avoid over interpretation of results. Generalizability analysis provides in the standard error a suitable indicator of uncertainty. Assessments depart from traditional measurements in ways that require extensions and re interpretatons of generalizability analysis. The authors find, for example, that in many analyses the variance component interpreted as Pupil x Task interaction actually arises in part from instability in pupil performance. It is necessary to recognize in the school-level analysis that pupils are nested in classes and schools. Whether to treat the population of a school's pupils as infinite or as limited to the student body assessed also requires careful consideration.