Proceedings of the VLDB Endowment · 2011 · 226 citations · 15 references
ComputingComputer ScienceComputer SystemsData ManagementDatabasesStorage SystemsData PartitioningData Storage SystemsData WarehousingDistributed Data ProcessingDistributed StorageStorage ManagementColumnar StorageData-intensive ComputingLarge-scale Data ProcessingLog File ProcessingMapreduceMapreduce AlgorithmsPerformance Bottleneck DetectionRelational Database Systems
Hadoop has become an attractive platform for large-scale data analytics. In this paper, we identify a major performance bottleneck of Hadoop: its lack of ability to colocate related data on the same set of nodes. To overcome this bottleneck, we introduce CoHadoop, a lightweight extension of Hadoop that allows applications to control where data are stored. In contrast to previous approaches, CoHadoop retains the flexibility of Hadoop in that it does not require users to convert their data to a certain format (e.g., a relational database or a specific file format). Instead, applications give hints to CoHadoop that some set of files are related and may be processed jointly; CoHadoop then tries to colocate these files for improved efficiency. Our approach is designed such that the strong fault tolerance properties of Hadoop are retained. Colocation can be used to improve the efficiency of many operations, including indexing, grouping, aggregation, columnar storage, joins, and sessionization. We conducted a detailed study of joins and sessionization in the context of log processing---a common use case for Hadoop---, and propose efficient map-only algorithms that exploit colocated data partitions. In our experiments, we observed that CoHadoop outperforms both plain Hadoop and previous work. In particular, our approach not only performs better than repartition-based algorithms, but also outperforms map-only algorithms that do exploit data partitioning but not colocation. 8.
15
Christopher Olston, Benjamin Reed, Utkarsh Srivastava et al. · 2008 · 1.7K citations
Ashish Thusoo, Joydeep Sen Sarma, Namit Jain et al. · Proceedings of the VLDB Endowment · 2009 · 1.5K citations
A comparison of approaches to large-scale data analysis
Andrew Pavlo, Erik K. Paulson, Alexander Rasin et al. · 2009 · 1.1K citations
A. A. ABOUZEID, Kamil Bajda-Pawlikowski, Daniel J. Abadi et al. · Proceedings of the VLDB Endowment · 2009 · 856 citations