2017 · 21 citations · 4 references
ComputingComputer ScienceComputer SystemsDatabasesDistributed SystemsParallel ComputingData ParallelismDistributed Data ProcessingDistributed Database ArchitectureDistributed StorageParallel DatabasesScalable Data StorageData Load ManagementDistributed Database ManagementKey-value StoresLarge-scale Data ProcessingParallel Data ProcessingParallel Database SystemsScalabilityScalable Data Management
Big Data has become a pervasive technology to manage the ever-increasing volumes of data. Among Big Data solutions, scalable data stores play an important role, especially, key-value data stores due to their large scalability (thousands of nodes). The typical workflow for Big Data applications include two phases. The first one is to load the data into the data store typically as part of an ETL (Extract-Transform-Load) process. The second one is the processing of the data itself. BigTable and HBase are the preferred key-value solutions based on range-partitioned data stores. However, the loading phase is inefficient and creates a single node bottleneck. In this paper, we identify and quantify this bottleneck and propose a tool for parallel massive data loading that solves satisfactorily the bottleneck enabling all the parallelism and throughput of the underlying key-value data store during the loading phase as well. The proposed solution has been implemented as a tool for parallel massive data loading over HBase, the key-value data store of the Hadoop ecosystem.
4
The Hadoop Distributed File System
Konstantin V. Shvachko, Hairong Kuang, Sanjay Radia et al. · 2010 · 4.8K citations
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani et al. · 2007 · 3.5K citations
Bigtable: a distributed storage system for structured data
Fay W. Chang, Jay B. Dean, Sanjay Ghemawat et al. · 2006 · 2K citations