2014 · 77 citations · 20 references
ComputingComputer EngineeringComputer ScienceCluster ComputingDistributed SystemsParallel ComputingBatch ProcessingDistributed ComputingDistributed Data ProcessingHigh-throughput ComputingLarge-scale Web CrawlingMapreduceMapreduce AlgorithmsPoliteness SystemsWeb Crawling
Although web crawlers have been around for twenty years by now, there is virtually no freely available, open-source crawling software that guarantees high throughput, overcomes the limits of single-machine tools and at the same time scales linearly with the amount of resources available. This paper aims at filling this gap.
20
Space/time trade-offs in hash coding with allowable errors
Burton H. Bloom · Communications of the ACM · 1970 · 7.5K citations · Full text
Similarity estimation techniques from rounding algorithms
Moses Charikar · 2002 · 2.2K citations
Syntactic clustering of the Web
Andrei Broder, S. Glassman, Mark S. Manasse et al. · Computer Networks and ISDN Systems · 1997 · 1.4K citations