A Methodology for Sampling the World Wide Web

Abstract

Abstract The rapid growth in the number of libraries providing Web access services has created a need for reliable, timely statistics characterizing the content of Web-accessible information. The size of the Web makes it impractical to develop descriptive statistics based on an exhaustive survey. An alternative approach is to collect a representative sample of Web pages. This report describes a methodology for sampling the content of the Web through the use of randomly generated IP addresses.