Out‐of‐core compression and decompression of large <i>n</i>‐dimensional scalar fields

TLDR

The paper proposes a simple method for compressing very large, regularly sampled scalar fields. The method uses a new Lorenzo predictor that estimates each sample from processed neighbors, enabling out‑of‑core compression with a minimal buffer and is effective for both lossy and lossless compression. Experiments show that the predictor yields exact values for n‑dimensional polynomial fields of degree n−1, and when residuals are arithmetic‑coded the method often surpasses wavelet compression in L∞ error, scaling to higher dimensions. Categories and Subject Descriptors (ACM CCS): I.3.5 – Computer Graphics: Compression, scalar fields, out‑of‑core.

Abstract

Abstract We present a simple method for compressing very large and regularly sampled scalar fields. Our method is particularlyattractive when the entire data set does not fit in memory and when the sampling rate is high relative to thefeature size of the scalar field in all dimensions. Although we report results for and data sets, the proposedapproach may be applied to higher dimensions. The method is based on the new Lorenzo predictor, introducedhere, which estimates the value of the scalar field at each sample from the values at processed neighbors. The predictedvalues are exact when the n‐dimensional scalar field is an implicit polynomial of degree n − 1 . Surprisingly,when the residuals (differences between the actual and predicted values) are encoded using arithmetic coding,the proposed method often outperforms wavelet compression in an L ∞ sense. The proposed approach may beused both for lossy and lossless compression and is well suited for out‐of‐core compression and decompression,because a trivial implementation, which sweeps through the data set reading it once, requires maintaining only asmall buffer in core memory, whose size barely exceeds a single ( n −1)‐ dimensional slice of the data. Categories and Subject Descriptors (according to ACM CCS): I.3.5 [Computer Graphics]: Compression, scalar fields,out‐of‐core.