2018 · 24 citations · 32 references
ComputingComputer ScienceComputer SystemsData ManagementDistributed SystemsStorage SystemsData RedundancyData Storage SystemsDistributed StorageHash TablesScalable Data StorageStorage ScalabilityData Redundancy ReductionDouble HashingHashingRedundant ComputationRedundant Data StorageScale-out StorageStorage DeduplicationStorage-aware Deduplication
Scale-out distributed storage systems can uphold balanced data growth in terms of capacity and performance on an on-demand basis. However, it is a challenge to store and manage large sets of contents being generated by the explosion of data. One of the promising solutions to mitigate big data issues is data deduplication, which removes redundant data across many nodes of the storage system. Nevertheless, it is non-trivial to apply a conventional deduplication design to the scale-out storage due to the following root causes. First, chunk-lookup for deduplication is not as scalable and extendable as the underlying storage system supports. Second, managing the metadata associated to deduplication requires a huge amount of design and implementation modifications of the existing distributed storage system. Lastly, the data processing and additional I/O traffic imposed by deduplication can significantly degrade performance of the scale-out storage. To address these challenges, we propose a new deduplication method, which is highly scalable and compatible with the existing scale-out storage. Specifically, our deduplication method employs a double hashing algorithm that leverages hashes used by the underlying scale-out storage, which addresses the limits of current fingerprint hashing. In addition, our design integrates the meta-information of file system and deduplication into a single object, and it controls the deduplication ratio at online by being aware of system demands based on post-processing. We implemented the proposed deduplication method on an open source scale-out storage. The experimental results show that our design can save more than 90% of the total amount of storage space, under the execution of diverse standard storage workloads, while offering the same or similar performance, compared to the conventional scale-out storage.
32
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani et al. · 2007 · 3.5K citations
Ceph: a scalable, high-performance distributed file system
Sage A. Weil, Scott Brandt, Ethan L. Miller et al. · eScholarship (California Digital Library) · 2006 · 1.4K citations · Full text
Sanjay Ghemawat, Howard Gobioff, Shun-Tak A. Leung · 2003 · 1.3K citations
Erasure coding in windows azure storage
Cheng Huang, Huseyin Simitci, Yikang Xu et al. · 2012 · 806 citations
Venti: A New Approach to Archival Storage
Sean Quinlan, Sean Dorward · 2002 · 728 citations