Searching the Web

TLDR

The paper provides an overview of current Web search engine design. The authors describe a generic search engine architecture and analyze its components—crawling, storage, indexing, and link‑analysis—using literature and an experimental testbed. Performance analyses comparing different design choices reveal key trade‑offs and demonstrate the impact of link‑analysis on search effectiveness.

Abstract

We offer an overview of current Web search engine design. After introducing a generic search engine architecture, we examine each engine component in turn. We cover crawling, local Web page storage, indexing, and the use of link analysis for boosting search performance. The most common design and implementation techniques for each of these components are presented. For this presentation we draw from the literature and from our own experimental search engine testbed. Emphasis is on introducing the fundamental concepts and the results of several performance analyses we conducted to compare different designs.