Archiving the web using page changes patterns

Myriam Ben Saâd, Stéphane Gançarski

2011 · 21 citations · 36 references

Concepts

TL;DR

Patterns, widely studied in data mining for anomaly detection, prediction, and system performance improvement, have not yet been applied to web archiving, which involves continuously collecting and preserving web content for future use. This paper demonstrates that patterns of page changes can serve as efficient tools for web site archiving. The authors define a pattern model of page changes, extract their temporal evolution, discover patterns, and exploit them to improve archiving, validated on the France Télévisions archive. Experimental evaluation on real web pages shows that using page‑change patterns improves archive quality and optimizes indexing and storage.

Abstract

A pattern is a model or a template used to summarize and describe the behavior (or the trend) of a data having generally some recurrent events. Patterns have received a considerable attention in recent years and were widely studied in the data mining field. Various pattern mining approaches have been proposed and used for different applications such as network monitoring, moving object tracking, financial or medical data analysis, scientific data processing, etc. In these different contexts, discovered patterns were useful to detect anomalies, to predict data behavior (or trend), or more generally, to simplify data processing or to improve system performance. However, to the best of our knowledge, patterns have never been used in the context of web archiving. Web archiving is the process of continuously collecting and preserving portions of the World Wide Web for future generations. In this paper, we show how patterns of page changes can be useful tools to efficiently archive web sites. We first define our pattern model that describes the changes of pages. Then, we present the strategy used to (i) extract the temporal evolution of page changes, to (ii) discover patterns and to (iii) exploit them to improve web archives. We choose the archive of French public TV channels « France Télévisions » as a case study in order to validate our approach. Our experimental evaluation based on real web pages shows the utility of patterns to improve archive quality and to optimize indexing or storing.

References

36