2018 4th International Conference on Optimization and Applications (ICOA) · 2018 · 55 citations · 5 references
ComputingComputer ScienceComputer SystemsBig Data AnalyticsDistributed SystemsReal-time SystemsData ParallelismDistributed Data ProcessingReal-time AnalyticsStream ProcessingBig Data ArchitectureMapreduceMapreduce AlgorithmsReal-time Data AnalysisReal-time Data ProcessingReal-time ProcessingReal-time Processing ArchitectureReal-time Stream Processing
Big Data is generated daily from the Internet, social networks, mobile devices, and other sources, and real‑time processing has become increasingly important, yet Hadoop MapReduce struggles to handle diverse real‑time data formats, underscoring the need for powerful tools. The study aims to implement and compare the de‑facto standard Hadoop MapReduce and Apache Spark for real‑time data processing, evaluating their architectures and performance. Experimental simulations on real‑time data streams were conducted using both frameworks, and the resulting architecture and performance metrics were compared. The analysis reveals that Hadoop suffers from significant drawbacks when applied to real‑time processing tasks.
Big Data is at use every single day, from the adoption of Internet through social networks, mobile devices, connected objects, videos, blogs and others. Big Data real-time processing have received a growing attention especially with the expansion of data in volume and complexity. Big data is created every day, from the use of the Internet through social networks, mobile devices, connected objects, videos, blogs and others. In order to ensure a reliable and a fast real-time information processing, powerful tools are essential for the analysis and processing of Big Data. Standards MapReduce frameworks such as Hadoop MapReduce face some limitations for processing real-time data of various formats. In this paper, we highlight the implementation of the de-facto standard Hadoop MapReduce and also the implementation of the framework Apache Spark. Thereafter, we conduct experimental simulations to analyze a real-time data stream using Spark and Hadoop. To further enforce our contribution, we introduce a comparison of the two implementations in terms of architecture and performance with a discussion to feature the results of simulations. The paper discusses also the drawbacks of using Hadoop for real-time processing.
5
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Matei Zaharia, Reynold Xin, Patrick Wendell et al. · Communications of the ACM · 2016 · 2.3K citations
Sanjay Ghemawat · Communications of the ACM · 2009 · 1.2K citations
Addressing big data problem using Hadoop and Map Reduce
Aditya Patel, Manashvi Birla, Ushma Nair · 2012 · 200 citations
Survey of real-time processing systems for big data
Xiufeng Liu, Nadeem Iftikhar, Xike Xie · 2014 · 134 citations