2014 · 99 citations · 13 references
ComputingComputer ScienceHigh-performance ComputingCluster ComputingInterconnection NetworksParallel ComputingBatch ProcessingCluster InterconnectsData ParallelismDistributed Data ProcessingLatency OptimizationData-intensive ComputingHigh-performance Computing NetworksHigh-performance Computing SystemsLarge-scale Data ProcessingMapreduceParallel Computing InterconnectsPlug-in ArchitectureShuffle
Apache Hadoop Map Reduce has been highly successful in processing large-scale, data-intensive batch applications on commodity clusters. However, for low-latency interactive applications and iterative computations, Apache Spark, an emerging in-memory processing framework, has been stealing the limelight. Recent studies have shown that current generation Big Data frameworks (like Hadoop) cannot efficiently leverage advanced features (e.g. RDMA) on modern clusters with high-performance networks. One of the major bottlenecks is that these middleware are traditionally written with sockets and do not deliver the best performance on modern HPC systems with RDMA-enabled high-performance interconnects. In this paper, we first assess the opportunities of bringing the benefits of RDMA into the Spark framework. We further propose a high-performance RDMA-based design for accelerating data shuffle in the Spark framework on high-performance networks. Performance evaluations show that our proposed design can achieve 79-83% performance improvement for Group By, compared with the default Spark running with IP over Infini Band (IPoIB) FDR on a 128-256 core cluster. We adopt a plug-in-based approach that can make our design to be easily integrated with newer Spark releases. To the best our knowledge, this is the first design for accelerating Spark with RDMA for Big Data processing.
13
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Spark: cluster computing with working sets
Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin et al. · 2010 · 4.2K citations
Matt Welsh, David Culler · 2001 · 847 citations
Memcached Design on High Performance RDMA Capable Interconnects
Jithin Jose, Hari Subramoni, Miao Luo et al. · 2011 · 161 citations