2012 · 24 citations · 24 references
ComputingComputer ScienceHigh-performance ComputingCluster ComputingDistributed SystemsParallel ComputingData ParallelismDistributed ComputingDistributed Data ProcessingMachine Learning For Big DataScalable Machine LearningK-means ClusteringLarge-scale Data ProcessingMapreduceMapreduce AlgorithmsParallel ClusteringParallel Data ProcessingParallel Machine LearningPerformance Characterization
Big data processing is currently becoming increasingly important in modern era due to the continuous growth of the amount of data generated by various fields such as particle physics, human genomics, earth observation, etc. However, the efficiency of processing large-scale data on modern virtual infrastructure, especially on the virtualized cloud computing infrastructure, is not clear. This paper focuses on the performance of hadoop virtual cluster and proposes a scalable hadoop virtual cluster platform vHadoop for the large-scale MapReduce-based parallel data processing. We first describe the design and implementation of vHadoop platform. Then we perform a series of experiments to investigate both the static and dynamic performance of vHadoop platform, such as the performance characterization of cross-domain hadoop virtual cluster and live migraiton of hadoop virtual cluster. After that, we use the vHadoop platform to process 6 typical parallel clustering algorithms, such as Canopy, Dirichlet, Fuzzy k-Means, k-Means, Mean Shift, MinHash, etc, on two typical datasets. Experimental results verify the efficiency of vHadoop platform to process the MapReduce-based parallel machine learning applications.
24
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Michael Armbrust, Armando Fox, Rean Griffith et al. · Communications of the ACM · 2010 · 9K citations · Full text
Xen and the art of virtualization
Paul Barham, Boris Dragovic, Keir Fraser et al. · ACM SIGOPS Operating Systems Review · 2003 · 5.9K citations
Live migration of virtual machines
Christopher J. Clark, Keir Fraser, Steven Hand et al. · 2005 · 2.6K citations
Improving MapReduce performance in heterogeneous environments
Matei Zaharia, Andy Konwinski, Anthony D. Joseph et al. · UC Berkeley · 2008 · 1.6K citations