Effective site finding using link anchor information

TLDR

Link-based ranking methods have been described in the literature and applied in commercial Web search engines. The study investigates the task of finding the main entry point of a specific Web site. Results were obtained using two sets of 100 queries on an 18.5‑million‑document collection and another set of 100 on a 0.4‑million‑document collection. Recent TREC experiments show link-based ranking is no better than content-based methods, yet our experiments demonstrate that ranking by link anchor text is twice as effective as content-based ranking using BM25, explaining why many search engines adopt link methods and opening a new area for improvement where traditional methods fail.

Abstract

Link-based ranking methods have been described in the literature and applied in commercial Web search engines. However, according to recent TREC experiments, they are no better than traditional content-based methods. We conduct a different type of experiment, in which the task is to find the main entry point of a specific Web site. In our experiments, ranking based on link anchor text is twice as effective as ranking based on document content, even though both methods used the same BM25 formula. We obtained these results using two sets of 100 queries on a 18.5 million document set and another set of 100 on a 0.4 million document set. This site finding effectiveness begins to explain why many search engines have adopted link methods. It also opens a rich new area for effectiveness improvement, where traditional methods fail.