A Survey of Web Information Extraction Systems

TLDR

The Internet hosts vast amounts of user‑formatted data that are difficult to extract, making robust, flexible IE systems essential, yet existing approaches have rarely been compared due to differing extraction tasks. The paper surveys major Web data extraction approaches and compares them across task domain, automation level, and techniques. The survey defines three criteria—site‑structure suitability, technique classification, and automation level—to evaluate IE systems. The authors conclude that these criteria provide qualitative measures for evaluating different IE approaches.

Abstract

The Internet presents a huge amount of useful information which is usually formatted for its users, which makes it difficult to extract relevant data from various sources. Therefore, the availability of robust, flexible Information Extraction (IE) systems that transform the Web pages into program-friendly structures such as a relational database will become a great necessity. Although many approaches for data extraction from Web pages have been developed, there has been limited effort to compare such tools. Unfortunately, in only a few cases can the results generated by distinct tools be directly compared since the addressed extraction tasks are different. This paper surveys the major Web data extraction approaches and compares them in three dimensions: the task domain, the automation degree, and the techniques used. The criteria of the first dimension explain why an IE system fails to handle some Web sites of particular structures. The criteria of the second dimension classify IE systems based on the techniques used. The criteria of the third dimension measure the degree of automation for IE systems. We believe these criteria provide qualitatively measures to evaluate various IE approaches.