Engineering the compression of massive tables: an experimental approach

Abstract

We study the problem of compressing massive tables. We devise a novel compression paradigm---training for lossless compression--- which assumes that the data exhibit dependencies that can be learned by examining a small amount of training material. We develop an experimental methodology to test the approach. Our result is a system, pzip, which outperforms gzip by factors of two in compression size and both compression and uncompression time for various tabular data. Pzip is now in production use in an AT&T network traffic data warehouse. 1 Introduction We study the problem of compressing massive tables, which arises naturally in corporate data warehouses. Our goal is to provide a working system that can be put into production use and achieve 100:1 compression, in particular, one that can compress 10s of TB of data into 100s of GB. We devise a novel compression strategy---training for lossless compression---which can leverage standard compression methods, and we demonstrate its effe...