ACM Transactions on the Web · 2008 · 106 citations · 25 references
ComputingComputer ScienceComputer SystemsDatabasesInformation RetrievalStorage SystemsCache Behavior AnalysisCachingCaching StrategiesQuery ProcessingSearch Log AnalysisSearch SystemsCache HierarchiesCache OptimizationCaching MechanismsData AccessHierarchical CachingQuery Log AnalysisQuery Performance OptimizationSearch Caching
In this article we study the trade-offs in designing efficient caching systems for Web search engines. We explore the impact of different approaches, such as static vs. dynamic caching, and caching query results vs. caching posting lists. Using a query log spanning a whole year, we explore the limitations of caching and we demonstrate that caching posting lists can achieve higher hit rates than caching query answers. We propose a new algorithm for static caching of posting lists, which outperforms previous methods. We also study the problem of finding the optimal way to split the static cache between answers and posting lists. Finally, we measure how the changes in the query log influence the effectiveness of static caching, given our observation that the distribution of the queries changes slowly over time. Our results and observations are applicable to different levels of the data-access hierarchy, for instance, for a memory/disk layer or a broker/remote server layer.
25
Analysis of a very large web search engine query log
Craig Silverstein, Hannes Marais, Monika Henzinger et al. · ACM SIGIR Forum · 1999 · 1.2K citations
Cost-aware WWW proxy caching algorithms
Pei Cao, Sandy Irani · Minds at UW (University of Wisconsin) · 1997 · 924 citations · Full text
Real life information retrieval: a study of user queries on the Web
Bernard J. Jansen, Amanda Spink, Judy Bateman et al. · ACM SIGIR Forum · 1998 · 621 citations · Full text
UbiCrawler: a scalable fully distributed Web crawler
Paolo Boldi, Bruno Codenotti, Massimo Santini et al. · Software Practice and Experience · 2004 · 564 citations