On optimisation of web crawler system on Scrapy framework

Abstract

With the continuous development of internet technology, life is accompanied by data at all times. However, network data is so complicated and confusing that it is difficult for users to find valuable information. Therefore, being able to acquire data from a vast data ocean has become an essential skill for today's business development. In this paper, a web crawler system based on the Scrapy framework is optimised to further enhance the crawler efficiency, increase the crawler speed, and break the crawler limit.