Corpus-based stemming using cooccurrence of word variants

Jinxi Xu, W. Bruce Croft

ACM Transactions on Information Systems · 1998 · 303 citations · 17 references

DOIFull text

Open access

Concepts

TL;DR

Stemming reduces variant word forms to common roots in IR, but existing methods ignore corpus‑specific language use, potentially causing retrieval failures. The study aims to develop a stemmer that leverages corpus‑based word variant cooccurrence statistics. The proposed approach modifies or creates a stemmer by applying cooccurrence statistics derived from the target corpus. Experiments on English newspaper, legal, and Spanish texts show the technique is viable and outperforms conventional morphology‑only stemmers.

Abstract

Stemming is used in many information retrieval (IR) systems to reduce variant word forms to common roots. It is one of the simplest applications of natural-language processing to IR and is one of the most effective in terms of user acceptance and consistency, though small retrieval improvements. Current stemming techniques do not, however, reflect the language use in specific corpora, and this can lead to occasional serious retrieval failures. We propose a technique for using corpus-based word variant cooccurrence statistics to modify or create a stemmer. The experimental results generated using English newspaper and legal text and Spanish text demonstrate the viability of this technique and its advantages relative to conventional approaches that only employ morphological rules.

References

17