2014 · 114 citations · 17 references
ComputingComputer ScienceData MiningInformation RetrievalWeb AnalyticsGraph AlgorithmsGraph AnalyticsGraph TheoryNetwork StructureNode-link DiagramsSmall-world NetworksGraph ProcessingGraph SearchLarge-scale Web CrawlingLink AnalysisPagerankWeb CrawlingWeb Page RankingWeb Structure Mining
Knowledge about the general graph structure of the World Wide Web is important for understanding the social mechanisms that govern its growth, for designing ranking methods, for devising better crawling algorithms, and for creating accurate models of its structure. In this paper, we describe and analyse a large, publicly accessible crawl of the web that was gathered by the Common Crawl Foundation in 2012 and that contains over 3.5 billion web pages and 128.7 billion links. This crawl makes it possible to observe the evolution of the underlying structure of the World Wide Web within the last 10 years: we analyse and compare, among other features, degree distributions, connectivity, average distances, and the structure of weakly/strongly connected components.
17
The PageRank Citation Ranking : Bringing Order to the Web
Lawrence M. Page, Sergey Brin, Rajeev Motwani et al. · 1999 · 12.7K citations
Power-Law Distributions in Empirical Data
SIAM Review · 2009 · 7.6K citations · Full text
Power-law distributions in empirical data
Aaron Clauset, Cosma Rohilla Shalizi, M. E. J. Newman · Figshare · 2018 · 6.8K citations · Full text
Andrei Broder, Ravi Kumar, Farzin Maghoul et al. · Computer Networks · 2000 · 2.8K citations
Paolo Boldi, Sebastiano Vigna · 2004 · 1.2K citations