Exploiting the maximum entropy principle to increase retrieval effectiveness

Abstract

Abstract Several of the drawbacks of conventional information retrieval systems can be overcome by a design approach in which queries consist of sets of terms, either unweighted or weighted with subjective term precision estimates, and retrieval outputs are ranked by probability of usefulness estimated in accordance with the so‐called “maximum entropy principle.” A system organized along these lines combines the convenience of a simple input language with a powerful probabilistic inference mechanism capable of exploiting kinds of statistical clues not ordinarily used in systems of traditional design. The sensitivity of the maximum entropy principle to the frequencies and joint frequencies with which terms have been assigned to documents in the collection results in a system design of increased power and expressiveness without a concomitant increase in the complexity of the request language. It incorporates the more important search capabilities of both Boolean and conventional weighted‐request languages and facilitates the use of unconventional search clues.