Wrapper generation for semi-structured Internet sources

TLDR

The current explosion of information on the World Wide Web has made a wealth of semi‑structured data available online, yet accessing it still requires browsing individual pages and there is no database‑like querying based on underlying structure. The study aims to enable database‑like querying of semi‑structured WWW sources by building wrappers around these sources. The approach semi‑automatically generates wrappers by exploiting page formatting to hypothesize the underlying structure, then producing a wrapper that facilitates querying and integration. We demonstrate that our wrapper‑generation toolkit can easily build wrappers for a variety of internet sources across different domains.

Abstract

With the current explosion of information on the World Wide Web (WWW) a wealth of information on many different subjects has become available on-line. Numerous sources contain information that can be classified as semi-structured. At present, however, the only way to access the information is by browsing individual pages. We cannot query web documents in a database-like fashion based on their underlying structure. However, we can provide database-like querying for semi-structured WWW sources by building wrappers around these sources. We present an approach for semi-automatically generating such wrappers. The key idea is to exploit the formatting information in pages from the source to hypothesize the underlying structure of a page. From this structure the system generates a wrapper that facilitates querying of a source and possibly integrating it with other sources. We demonstrate the ease with which we are able to build wrappers for a number of internet sources in different domains using our implemented wrapper generation toolkit.