Fast and efficient log file compression

Abstract

Abstract. Contemporary information systems are replete with log files, created in multiple places (e.g., network servers, database management systems, user monitoring applications, system services and utilities) for multiple purposes (e.g., maintenance, security issues, traffic analysis, legal requirements, software debugging, customer management, user interface usability studies). Log files in complex systems may quickly grow to huge sizes. Often, they must be kept for long periods of time. For reasons of convenience and storage economy, log files should be compressed. However, most of the available log file compression tools use general-purpose algorithms (e.g., Deflate) which do not take advantage of redundancy specific for log files. In this paper a specialized log file compression scheme is described in five variants, differing in complexity and attained compression ratios. The proposed scheme introduces a log file transform whose output is much better compressible with general-purpose algorithms than original data. Using the fast Deflate algorithm, the transformed log files were, on average, 36.6 % shorter than the original files compressed with gzip (employing the same algorithm). Using the slower PPMVC algorithm, the transformed files were 62 % shorter than the original files compressed with gzip, and 41 % shorter than the original files compressed with bzip2.