Online aggregation

TLDR

Traditional database aggregation is performed in batch mode, which is slow, frustrating, and has been abandoned in most other computing domains. This paper proposes an online aggregation interface that lets users observe query progress and control execution on the fly. We outline usability and performance requirements and present techniques that extend a database system to support online aggregation, including random‑order output, adjustable aggregate rates, and running confidence intervals. An initial implementation of the approach in PostgreSQL demonstrates its feasibility.

Abstract

Aggregation in traditional database systems is performed in batch mode: a query is submitted, the system processes a large volume of data over a long period of time, and, eventually, the final answer is returned. This archaic approach is frustrating to users and has been abandoned in most other areas of computing. In this paper we propose a new online aggregation interface that permits users to both observe the progress of their aggregation queries and control execution on the fly. After outlining usability and performance requirements for a system supporting online aggregation, we present a suite of techniques that extend a database system to meet these requirements. These include methods for returning the output in random order, for providing control over the relative rate at which different aggregates are computed, and for computing running confidence intervals. Finally, we report on an initial implementation of online aggregation in POSTGRES.