Parameter estimation for probabilistic document-retrieval models

Abstract

A probabilistic document-retrieval system may be seen as a sequential learning process, in which the system learns the characteristics of relevant documents, or more formally, it learns the parameters of probability distributions describing the frequencies of feature occurrences in relevant and nonrelevant documents. Probability distributions that may be used to describe the distribution of features include binary and Poisson distributions. Techniques for estimating the parameters of distributions are suggested. We have tested a proposal that parameters of distributions describing the distribution of features in nonrelevant documents be estimated from the parameters of the corresponding distributions of the entire database; the confidence parameter of such an estimate resulting in the highest average precision is given. Tests of several methods for estimating the parameters of distributions describing the distribution of features in relevant documents suggest that small values of the confidence parameter be used in our initial estimates of parameters for relevant documents. © 1988 John Wiley & Sons, Inc.