Measures for the comparison of information retrieval systems

Abstract

Abstract A definite need exists for an absolute and uniform means of comparison and evaluation of certain characteristics of various information retrieval systems. This paper is concerned only with the system's ability to satisfy queries; the effort and time used, and their relative costs, are not considered. Some past procedures that have been used are reviewed and discussed, along with some of the difficulties that arise when these procedures are followed. We present some desirable characteristics for a meaningful measure of effectiveness, then analyze in detail one such measure that may be used to compare the retrieval capabilities of a large variety of systems. This measure, called the “normalized sliding ratio,” has a relation to the classical recall and precision ratios and may be looked on as a generalization of them.