A THEORETICAL BASIS FOR THE USE OF CO‐OCCURRENCE DATA IN INFORMATION RETRIEVAL

TLDR

Index terms are usually assumed independent to simplify weighting, but in reality they are likely dependent. The paper aims to improve automatic retrieval effectiveness by developing a practical method that removes the independence assumption in index term weighting. The authors measure inter‑term dependence to build a non‑linear weighting function, estimate its parameters from small document samples, and recommend a specific estimation rule. The study demonstrates that the required computations for the non‑linear weighting scheme are feasible.

Abstract

Abstract This paper provides a foundation for a practical way of improving the effectiveness of an automatic retrieval system. Its main concern is with the weighting of index terms as a device for increasing retrieval effectiveness. Previously index terms have been assumed to be independent for the good reason that then a very simple weighting scheme can be used. In reality index terms are most unlikely to be independent. This paper explores one way of removing the independence assumption. Instead the extent of the dependence between index terms is measured and used to construct a non‐linear weighting function. In a practical situation the values of some of the parameters of such a function must be estimated from small samples of documents. So a number of estimation rules are discussed and one in particular is recommended. Finally the feasibility of the computations required for a non‐linear weighting scheme is examined.