2006 · 40 citations · 14 references
ComputingComputer ScienceInformation RetrievalText AnalyticsWeb AnalyticsDocument ClusteringDocument Image AnalysisDocument LayoutRetrieval ModelsSearch SystemsWeb SearchHuge PagesLink AnalysisSearch Engine AlgorithmsSearch Engine IndexingTemplate Removal
Templates in web sites hurt search engine retrieval performance, especially in content relevance and link analysis. Current template removal methods suffer from processing speed and scalability when dealing with large volume web pages. In this paper, we propose a novel two-stage template detection method, which combines template detection and removal with the index building process of a search engine. First, web pages are segmented into blocks and blocks are clustered according to their style features. Second, similar contents sharing the common layout style are detected during the index building process. The blocks with similar layout style and content are identified as templates and deleted. Our experiment on eight popular web sites shows that our method achieves 20-40% faster than shingle and SST methods with close accuracy.
14
Space/time trade-offs in hash coding with allowable errors
Burton H. Bloom · Communications of the ACM · 1970 · 7.5K citations · Full text
Extracting structured data from Web pages
Arvind Arasu, Héctor García-Molina · 2003 · 701 citations
Automatic web news extraction using tree edit distance
Davi C. Reis, Paulo B. Golgher, Ana S. C. Silva et al. · 2004 · 340 citations
Eliminating noisy information in Web pages for data mining
Discovering informative content blocks from Web documents
Shian-Hua Lin, Jan-Ming Ho · 2002 · 321 citations