Using the wayback machine to mine websites in the social sciences: A methodological resource

Sanjay Arora, Yin Li, Jan Youtie, Philip Shapira

Journal of the Association for Information Science and Technology · 2015 · 98 citations · 35 references

DOIFull text

Open access

Concepts

TL;DR

Websites provide an unobtrusive data source for studying various social science phenomena. This paper offers a methodological resource for social scientists to use the Wayback Machine to access historical web-based text. The authors review existing Wayback Machine studies and present a six‑step protocol—sampling, defining crawl boundaries, crawling, variable operationalization, integration with other data, and analysis—illustrated with a case on green goods SMEs. The method generalizes beyond green goods firms, and while both machine and human effort are needed, it yields high‑quality archived web data.

Abstract

Websites offer an unobtrusive data source for developing and analyzing information about various types of social science phenomena. In this paper, we provide a methodological resource for social scientists looking to expand their toolkit using unstructured web‐based text, and in particular, with the W ayback M achine, to access historical website data. After providing a literature review of existing research that uses the W ayback M achine, we put forward a step‐by‐step description of how the analyst can design a research project using archived websites. We draw on the example of a project that analyzes indicators of innovation activities and strategies in 300 U.S. small‐ and medium‐sized enterprises in green goods industries. We present six steps to access historical W ayback website data: (a) sampling, (b) organizing and defining the boundaries of the web crawl, (c) crawling, (d) website variable operationalization, (e) integration with other data sources, and (f) analysis. Although our examples draw on specific types of firms in green goods industries, the method can be generalized to other areas of research. In discussing the limitations and benefits of using the W ayback M achine, we note that both machine and human effort are essential to developing a high‐quality data set from archived web information.

References

35