Zero-Latency Data Warehousing for Heterogeneous Data Sources and Continuous Data Streams.

Abstract

In this paper, a framework for building an overall Zero-Latency Data Warehouse system (ZLDWH) is provided. Such a ZLDWH requires tasks such as data changes detection, continuous loading and updating of new data, autonomous analysis and the execution of actions to be completed in the Data Warehouse (DWH). For this purpose, we combine (1) an existing solution for the continuous data integration and (2) the known approach of Active Data Warehousing (ADWH) by introducing protocols that enable the correct collaboration between the two. The continuous data integration is realized on the one hand by asynchronous communication between multiple information sources and the DWH through a message queuing system, and on the other hand by the continuous loading and updating. To obtain a low-latency time for this cycle, an “active rule evaluation” is provided between the data feeding stage and its loading into the DWH. This is realized by applying ECA-Techniques (EventCondition-Action) to obtain a more “active” characteristic of the DWH. A second goal of this paper is to take into consideration a special class of information source namely continuous data streams. Continuous data streams are information sources in which data arrives in high-volume, rapidly, bursty, and needs to be processed continuously. Therefore novel protocols and techniques are needed for capturing, storing and evaluate analysis decision based on new coming data from such continuous data stream sources.