Indexing by latent semantic analysis

TLDR

The paper introduces a new automatic indexing and retrieval method that leverages semantic structure to improve document relevance detection. The method uses singular‑value decomposition of a term‑by‑document matrix into 100 orthogonal factors, representing documents and queries as 100‑dimensional vectors and retrieving documents with cosine similarity above a threshold. Initial tests show the automatic retrieval method to be promising. © 1990 John Wiley & Sons, Inc.

Abstract

A new method for automatic indexing and retrieval is described. The approach is to take advantage of implicit higher-order structure in the association of terms with documents ("semantic structure") in order to improve the detection of relevant documents on the basis of terms found in queries. The particular technique used is singular-value decomposition, in which a large term by document matrix is decomposed into a set of ca. 100 orthogonal factors from which the original matrix can be approximated by linear combination. Documents are represented by ca. 100 item vectors of factor weights. Queries are represented as pseudo-document vectors formed from weighted combinations of terms, and documents with supra-threshold cosine values are returned. Initial tests find this completely automatic method for retrieval to be promising. © 1990 John Wiley & Sons, Inc.