IEEE International Conference on High Performance Computing, Data, and Analytics · 2012 · 108 citations · 25 references
ComputingComputer ScienceComputer SystemsComputer NetworksDistributed SystemsStorage SystemsDistributed StorageFile StorageRemote Memory AccessDistributed File System PerformanceInfinibandLatency Reduction TechniquesRemote Direct Memory AccessStorageStorage Access Latency OptimizationStorage Performance Optimization
Hadoop Distributed File System (HDFS) acts as the primary storage of Hadoop and has been adopted by reputed organizations (Facebook, Yahoo! etc.) due to its portability and fault-tolerance. The existing implementation of HDFS uses Java-socket interface for communication which delivers suboptimal performance in terms of latency and throughput. For data-intensive applications, network performance becomes key component as the amount of data being stored and replicated to HDFS increases. In this paper, we present a novel design of HDFS using Remote Direct Memory Access (RDMA) over InfiniBand via JNI interfaces. Experimental results show that, for 5GB HDFS file writes, the new design reduces the communication time by 87% and 30% over 1Gigabit Ethernet (1GigE) and IP-over-InfiniBand (IPoIB), respectively, on QDR platform (32Gbps). For HBase, the Put operation performance is improved by 26% with our design. To the best of our knowledge, this is the first design of HDFS over InfiniBand networks.
25
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Space/time trade-offs in hash coding with allowable errors
Burton H. Bloom · Communications of the ACM · 1970 · 7.5K citations · Full text
Benchmarking cloud serving systems with YCSB
Brian F. Cooper, Adam Silberstein, Erwin Tam et al. · 2010 · 3.6K citations
Fay W. Chang, Sanjay Ghemawat, Wilson C. Hsieh et al. · ACM Transactions on Computer Systems · 2008 · 3.4K citations