Survey of real-time processing systems for big data

TLDR

Real‑time processing and analytics for big data have attracted attention because traditional BI platforms with periodic updates cannot keep pace with fast‑changing business environments, and although MapReduce and Hadoop offer scalability, they lack real‑time update support, prompting the emergence of new real‑time frameworks. This paper surveys open‑source technologies that support real‑time or near‑real‑time big‑data processing, focusing on their system architectures and platforms. It reviews the architectures and platforms of these technologies to assess their real‑time capabilities.

Abstract

In recent years, real-time processing and analytics systems for big data–in the context of Business Intelligence (BI)–have received a growing attention. The traditional BI platforms that perform regular updates on daily, weekly or monthly basis are no longer adequate to satisfy the fast-changing business environments. However, due to the nature of big data, it has become a challenge to achieve the real-time capability using the traditional technologies. The recent distributed computing technology, MapReduce, provides off-the-shelf high scalability that can significantly shorten the processing time for big data; Its open-source implementation such as Hadoop has become the de-facto standard for processing big data, however, Hadoop has the limitation of supporting real-time updates. The improvements in Hadoop for the real-time capability, and the other alternative real-time frameworks have been emerging in recent years. This paper presents a survey of the open source technologies that support big data processing in a real-time/near real-time fashion, including their system architectures and platforms.