Effective information retrieval using genetic algorithms based matching functions adaptation

Abstract

Knowledge intensive organizations have vast array of information contained in large document repositories. With the advent of E-commerce and corporate intranets/extranets, these repositories are expected to grow at a fast pace. This explosive growth has led to huge, fragmented, and unstructured document collections. Although it has become easier to collect and store information in document collections, it has become increasingly difficult to retrieve relevant information from these large document collections. This paper addresses the issue of improving retrieval performance (in terms of precision and recall) for retrieval from document collections. There are three important paradigms of research in the area of information retrieval (1R): Probabilistic IR, Knowledge-based IR, and, Artificial Intelligence based techniques like neural networks and symbolic learning. Very few researcher have tried to use evolutionary algorithms like genetic algorithms (GAs). Previous attempts at using GAs have concentrated on modifying document representations or modifying query representations. This work looks at the possibility of applying GAs to adapt various matching functions. It is hoped that such an adaptation of the matching functions in lead to a better retrieval performance than that obtained by using a single matching function. An overall matching function is treated as an weighted combination of scores produced by individual matching functions. This overall score is asked to rank and retrieve documents. Weights associated with individual functions are searched using Genetic Algorithms. The idea is tested on a real document collection called the Cranfield collection. The results look very encouraging.