TelegraphCQ: Continuous Dataflow Processing for an Uncertain World.

TLDR

Pervasive networks are creating a world of constant data motion, rendering traditional query processing inadequate and motivating the Telegraph project’s development of novel continuous adaptive query technologies. The paper proposes that adaptive dataflow query processors, exemplified by the next‑generation TelegraphCQ system, are required to handle large, high‑volume, highly variable continuous query streams, and outlines open issues and a research agenda. TelegraphCQ’s architecture builds on PostgreSQL, incorporating adaptive dataflow techniques to manage large, high‑volume, highly variable continuous query streams. The authors report on an ongoing implementation of TelegraphCQ that extends PostgreSQL with adaptive dataflow capabilities for continuous query processing.

Abstract

Increasingly pervasive networks are leading towards a world where data is constantly in motion. In such a world, conventional techniques for query processing, which were developed under the assumption of a far more static and predictable computational environment, will not be sufficient. Instead, query processors based on adaptive dataflow will be necessary. The Telegraph project has developed a suite of novel technologies for continuously adaptive query processing. The next generation Telegraph system, called TelegraphCQ, is focused on meeting the challenges that arise in handling large streams of continuous queries over high-volume, highly-variable data streams. In this paper, we describe the system architecture and its underlying technology, and report on our ongoing implementation effort, which leverages the PostgreSQL open source code base. We also discuss open issues and our research agenda.