International Journal of Computer Applications · 2010 · 25 citations · 16 references
ComputingComputer ScienceInformation TechnologyData ManagementDatabasesInformation RetrievalSearch SystemsWeb SearchIncremental IndexingSearch Engine CrawlingSearch Engine IndexingWeb Crawling
The World Wide Web is a huge source of hyperlinked information contained in hypertext documents. Search engines use web crawlers to collect these documents from web for the purpose of storage and indexing. However, many of these documents contain dynamic information which gets changed on daily, weekly, monthly or yearly basis and hence we need to refresh the search engine side storage so that latest information is made available to the user. An incremental crawler visits the web repeatedly after a specific interval for updating its collection. In this paper to regulate the revisiting frequency a novel mechanism and a novel architecture for incremental crawler is being proposed.
16
A brief history of the internet
Barry M. Leiner, Vinton G. Cerf, David D. Clark et al. · ACM SIGCOMM Computer Communication Review · 2009 · 974 citations
Mercator: A scalable, extensible Web crawler
Allan Heydon, Marc Najork · World Wide Web · 1999 · 578 citations
Arvind Arasu, Junghoo Cho, Héctor García-Molina et al. · ACM Transactions on Internet Technology · 2001 · 506 citations