Encapsulation of parallelism in the Volcano query processing system

TLDR

Volcano is a new dataflow query processing system designed for research and education, and this paper introduces a novel operator model and exchange operator to parallelize all other operators. Volcano’s uniform operator interface, single‑process operator design, and translation between demand‑driven intra‑process and data‑driven inter‑process dataflow enable a new operator‑based parallelization model, with an exchange operator that encapsulates all parallelism issues. The exchange operator permits intra‑operator parallelism on partitioned datasets and both vertical and horizontal inter‑operator parallelism, simplifying and strengthening parallel database algorithm implementation, and its compatibility with commercial system interfaces suggests broader applicability.

Abstract

Volcano is a new dataflow query processing system we have developed for database systems research and education. The uniform interface between operators makes Volcano extensible by new operators. All operators are designed and coded as if they were meant for a single-process system only. When attempting to parallelize Volcano, we had to choose between two models of parallelization, called here the bracket and operator models. We describe the reasons for not choosing the bracket model, introduce the novel operator model, and provide details of Volcano's exchange operator that parallelizes all other operators. It allows intra-operator parallelism on partitioned datasets and both vertical and horizontal inter-operator parallelism. The exchange operator encapsulates all parallelism issues and therefore makes implementation of parallel database algorithms significantly easier and more robust. Included in this encapsulation is the translation between demand-driven dataflow within processes and data-driven dataflow between processes. Since the interface between Volcano operators is similar to the one used in “real,” commercial systems, the techniques described here can be used to parallelize other query processing engines.