2005 · 64 citations · 14 references
ComputingComputer ScienceInformation SystemsAlgorithmsDatabasesInformation RetrievalData CompressionQuery OptimizationQuery ProcessingSearch SystemsSearching AlgorithmsWeb SearchCompressionIndex CompressionIndexingPrecisionPruningRetrievalSearch EfficiencyStatic Pruning
The unarguably fast, and continuous, growth of the volume of indexed (and indexable) documents on the Web poses a great challenge for search engines. This is true regarding not only search effectiveness but also time and space efficiency. In this paper we present an index pruning technique targeted for search engines that addresses the latter issue without disconsidering the former. To this effect, we adopt a new pruning strategy capable of greatly reducing the size of search engine indices. Experiments using a real search engine show that our technique can reduce the indices' storage costs by up to 60% over traditional lossless compression methods, while keeping the loss in retrieval precision to a minimum. When compared to the indices size with no compression at all, the compression rate is higher than 88%, i.e., less than one eighth of the original size. More importantly, our results indicate that, due to the reduction in storage overhead, query processing time can be reduced to nearly 65% of the original time, with no loss in average precision. The new method yields significative improvements when compared against the best known static pruning method for search engine indices. In addition, since our technique is orthogonal to the underlying search algorithms, it can be adopted by virtually any search engine.
14
Indexing by latent semantic analysis
Scott Deerwester, Susan Dumais, George W. Furnas et al. · Journal of the American Society for Information Science · 1990 · 12.8K citations
Authoritative sources in a hyperlinked environment
Jon Kleinberg · Symposium on Discrete Algorithms · 1998 · 1.8K citations
Ronald Fagin, Ravi Kumar, D. Sivakumar · 2003 · 768 citations
A large-scale study of the evolution of web pages
Dennis Fetterly, Mark S. Manasse, Marc Najork et al. · 2003 · 347 citations
The use of phrases and structured queries in information retrieval
W. Bruce Croft, Howard R. Turtle, David Lewis · 1991 · 266 citations · Full text