2013 · 39 citations · 15 references
ComputingComputer ScienceComputer SystemsBig Data AnalyticsData ManagementStorage SystemsData DeduplicationData RedundancyData ReplicationData Storage SystemsStorage ManagementBig Data ArchitectureData Redundancy ReductionDeduplication Performance EvaluationHash-based DeduplicationIndexingOverhead ReductionStorage DeduplicationStorage Overhead Optimization
The demand for data storage and processing is increasing at a rapid speed in the big data era. Such a tremendous amount of data pushes the limit on storage capacity and on the storage network. A significant portion of the dataset in big data workloads is redundant. As a result, deduplication technology, which removes replicas, becomes an attractive solution to save disk space and traffic in a big data environment. However, the overhead of extra CPU computation (hash indexing) and IO latency introduced by deduplication should be considered. Therefore, the net effect of using deduplication for big data workloads needs to be examined. To this end, we characterize the redundancy of typical big data workloads to justify the need for deduplication. We analyze and characterize the performance and energy impact brought by deduplication under various big data environments. In our experiments, we identify three sources of redundancy in big data workloads: 1) deploying more nodes, 2) expanding the dataset, and 3) using replication mechanisms. We elaborate on the advantages and disadvantages of different deduplication layers, locations, and granularities. In addition, we uncover the relation between energy overhead and the degree of redundancy. Furthermore, we investigate the deduplication efficiency in an SSD environment for big data workloads.
15
Sparse indexing: large scale, inline deduplication using sampling and locality
Mark Lillibridge, Kave Eshghi, Deepavali Bhagwat et al. · 2009 · 384 citations
iDedup: latency-aware, inline data deduplication for primary storage
Kiran Srinivasan, Tim Bisson, Garth R. Goodson et al. · 2012 · 254 citations
Ricardo Koller, Raju Rangaswami · ACM Transactions on Storage · 2010 · 225 citations