RECERCAT (Consorci de Serveis Universitaris de Catalunya) · 41 citations · 23 references
Open access
ComputingComputer ScienceComputer SystemsBig Data AnalyticsCloud InfrastructureParallel ComputingCompute Resource ManagementLatency OptimizationParallel Performance AnalysisPerformance Bottleneck AnalysisScalability ArchitectureData-intensive ComputingEmpirical EvaluationIn-memory AnalyticsLarge-scale Data ProcessingMemory Access LatencyMemory LatencyNuma ArchitecturesPerformance Characterization
In last decade, data analytics have rapidly progressed from traditional disk-based processing to modern in-memory processing. However, little effort has been devoted at enhancing performance at micro-architecture level. This paper characterizes the performance of in-memory data analytics using Apache Spark framework. We use a single node NUMA machine and identify the bottlenecks hampering the scalability of workloads. We also quantify the inefficiencies at micro-architecture level for various data analysis workloads. Through empirical evaluation, we show that spark workloads do not scale linearly beyond twelve threads, due to work time inflation and thread level load imbalance. Further, at the micro-architecture level, we observe memory bound latency to be the major cause of work time inflation.
23
The Hadoop Distributed File System
Konstantin V. Shvachko, Hairong Kuang, Sanjay Radia et al. · 2010 · 4.8K citations
Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik et al. · 2010 · 3.5K citations
Ankit Toshniwal, Siddarth Taneja, Amit K. Shukla et al. · 2014 · 858 citations
Michael Ferdman, Almutaz Adileh, Onur Kocberber et al. · 2012 · 814 citations