Analysis of a very large web search engine query log

TLDR

The AltaVista query log contains about 285 million user sessions, each representing an attempt to satisfy a single information need. The study analyzes a six‑week AltaVista query log of ~1 billion entries to examine individual queries, duplication, sessions, and term interactions. The authors aggregate queries, identify duplicates and sessions, and compute correlations between terms within queries. The analysis shows that web users issue short queries, focus on the first ten results, rarely refine them, and that highly correlated terms tend to form phrases, implying that traditional IR methods may be inadequate and that search engines should treat terms as phrase components even when not explicitly indicated.

Abstract

In this paper we present an analysis of an AltaVista Search Engine query log consisting of approximately 1 billion entries for search requests over a period of six weeks. This represents almost 285 million user sessions, each an attempt to fill a single information need. We present an analysis of individual queries, query duplication, and query sessions. We also present results of a correlation analysis of the log entries, studying the interaction of terms within queries. Our data supports the conjecture that web users differ significantly from the user assumed in the standard information retrieval literature. Specifically, we show that web users type in short queries, mostly look at the first 10 results only, and seldom modify the query. This suggests that traditional information retrieval techniques may not work well for answering web search requests. The correlation analysis showed that the most highly correlated items are constituents of phrases. This result indicates it may be useful for search engines to consider search terms as parts of phrases even if the user did not explicitly specify them as such.