Real-time data analysis using Spark and Hadoop

Khadija Aziz, Dounia Zaidouni, Mostafa Bellafkih

2018 4th International Conference on Optimization and Applications (ICOA) · 2018 · 55 citations · 5 references

Concepts

TL;DR

Big Data is generated daily from the Internet, social networks, mobile devices, and other sources, and real‑time processing has become increasingly important, yet Hadoop MapReduce struggles to handle diverse real‑time data formats, underscoring the need for powerful tools. The study aims to implement and compare the de‑facto standard Hadoop MapReduce and Apache Spark for real‑time data processing, evaluating their architectures and performance. Experimental simulations on real‑time data streams were conducted using both frameworks, and the resulting architecture and performance metrics were compared. The analysis reveals that Hadoop suffers from significant drawbacks when applied to real‑time processing tasks.

Abstract

Big Data is at use every single day, from the adoption of Internet through social networks, mobile devices, connected objects, videos, blogs and others. Big Data real-time processing have received a growing attention especially with the expansion of data in volume and complexity. Big data is created every day, from the use of the Internet through social networks, mobile devices, connected objects, videos, blogs and others. In order to ensure a reliable and a fast real-time information processing, powerful tools are essential for the analysis and processing of Big Data. Standards MapReduce frameworks such as Hadoop MapReduce face some limitations for processing real-time data of various formats. In this paper, we highlight the implementation of the de-facto standard Hadoop MapReduce and also the implementation of the framework Apache Spark. Thereafter, we conduct experimental simulations to analyze a real-time data stream using Spark and Hadoop. To further enforce our contribution, we introduce a comparison of the two implementations in terms of architecture and performance with a discussion to feature the results of simulations. The paper discusses also the drawbacks of using Hadoop for real-time processing.

References

5