Australasian Database Conference · 2004 · 18 citations · 8 references
ComputingComputer ScienceInformation SystemsData ManagementDistributed SystemsInformation RetrievalEvaluation Of Retrieval SystemsQuery ProcessingRetrieval ModelsSearch SystemsSearching AlgorithmsWeb SearchCost-aware SearchIndexingLocal SearchSearch Engine CrawlingSearch Engine IndexingSearch Engine Query LogsSearch IndexingWeb Crawling
Web search engines crawl the web to fetch the data that they index. In this paper we re-examine that need, and evaluate the network costs associated with data acquisition, and alternative ways in which a search service might be supported. As a concrete example, we make use of the Research Finder search service provided at http://rf.panopticsearch.com, and information derived from its crawl and query logs. Based upon an analysis of the Research Finder system we introduce a hybrid arrangement, in which queries are evaluated partially by reference to a centrally maintained index representing a subset of the collection, and partially by referring them on to the local search services maintained by the balance of the collection. We also examine various ways in which crawling costs can be reduced.
8
Analysis of a very large web search engine query log
Craig Silverstein, Hannes Marais, Monika Henzinger et al. · ACM SIGIR Forum · 1999 · 1.2K citations
A large-scale study of the evolution of web pages
Dennis Fetterly, Mark S. Manasse, Marc Najork et al. · 2003 · 347 citations