Essay Reliability: Form and Meaning.

Doug Shale

1986 · 25 citations · 15 references

Concepts

Abstract

This study is an attempt at a cohesive characterization of the concept of essay reliability. As such, it takes as a basic premise that previous and current practices in reporting estimates for essay tests have certain shortcomings. The study provides an analysis of these shortcomings--partly to encourage a fuller understanding of the concept of as applied to essay testing, and partly to build the case that the framework of generalizabili'cy theory offers a much more satisfactory way of characterizing the concept. The study applies generalizability theory to existing research to illustrate that this approach is an improvement over the usual methods of estimating essay reliability. The paper also argues that the classical approach to has lead to a preoccupation with inter-marker agreement which '_ri turn has lead to a formulation of the reliability problem that makes it not susceptible of solution. Conceptual grounds for tol,...-sating inter-marker disagreement are advanced and the paper discusses conditions under which this may be so. However, the paper explains how generaliLability theory remains an appropriate framework for estimating the of essay scores whatever assumptions one chooses to make regarding inter-marker istency. cons

References

15