Legality and Ethics of Web Scraping

Abstract

Automatic retrieval of data from the Web (often referred to as Web Scraping) for industry and academic research projects is becoming a common practice. A variety of tools and technologies have been developed to facilitate Web Scraping. Unfortunately, the legality and ethics of using these tools for collecting data are often overlooked. Failure to pay due attention to these aspects of Web Scraping can result in serious ethical controversies and lawsuits. This paper reviews legal literature together with the literature on ethics and privacy to identify broad areas of concern together with a list of specific questions that need to be addressed by researchers and practitioners engaged in Web Scraping. Reflecting on these questions and concerns can potentially help the researchers decrease the likelihood of ethical and legal controversies in their work.