2007 · 610 citations · 50 references
ComputingComputer ScienceInformation TechnologyData MiningInformation RetrievalWeb AnalyticsDuplicate DetectionSearch SystemsSearching AlgorithmsWeb SearchDuplicate Detection EvaluationFingerprintingFingerprinting TechniquesSimilarity DetectionWeb Crawling
Near-duplicate web documents are abundant. Two such documents differ from each other in a very small portion that displays advertisements, for example. Such differences are irrelevant for web search. So the quality of a web crawler increases if it can assess whether a newly crawled web page is a near-duplicate of a previously crawled web page or not. In the course of developing a near-duplicate detection system for a multi-billion page repository, we make two research contributions. First, we demonstrate that Charikar's fingerprinting technique is appropriate for this goal. Second, we present an algorithmic technique for identifying existing f-bit fingerprints that differ from a given fingerprint in at most k bit-positions, for small k. Our technique is useful for both online queries (single fingerprints) and all batch queries (multiple fingerprints). Experimental evaluation over real data confirms the practicality of our design.
50
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Indexing by latent semantic analysis
Scott Deerwester, Susan Dumais, George W. Furnas et al. · Journal of the American Society for Information Science · 1990 · 12.8K citations
Authoritative sources in a hyperlinked environment
Jon Kleinberg · Journal of the ACM · 1999 · 9.1K citations · Full text