2016 · 70 citations · 29 references
ComputingComputer EngineeringComputer ScienceFault ToleranceFault-tolerant ComputingFault-tolerant SystemsCoding TheoryData RedundancyDistributed StorageDistributed Storage SystemsFault-tolerant Storage SystemsReliable Storage SystemsCode ConstructionsDistributed Storage ReliabilityEncoding AlgorithmsErasure CorrectionRegenerating CodesStorage Erasure CodingStorage Overhead OptimizationSystematic Codes
Large distributed storage systems use erasure codes to reliably store data. Compared to replication, erasure codes are capable of reducing storage overhead. However, repairing lost data in an erasure coded system requires reading from many storage devices and transferring over the network large amounts of data. Theoretically, Minimum Storage Regenerating (MSR) codes can significantly reduce this repair burden. Although several explicit MSR code constructions exist, they have not been implemented in real-world distributed storage systems. We close this gap by providing a performance analysis of Butterfly codes, systematic MSR codes with optimal repair I/O. Due to the complexity of modern distributed systems, a straightforward approach does not exist when it comes to implementing MSR codes. Instead, we show that achieving good performance requires to vertically integrate the code with multiple system layers. The encoding approach, the type of inter-node communication, the interaction between different distributed system layers, and even the programming language have a significant impact on the code repair performance. We show that with new distributed system features, and careful implementation, we can achieve the theoretically expected repair performance of MSR codes.
29
John Kubiatowicz, David Bindel, Yan Chen et al. · ACM SIGPLAN Notices · 2000 · 2K citations · Full text
Network Coding for Distributed Storage Systems
Alexandros G. Dimakis, P. Brighten Godfrey, Yunnan Wu et al. · IEEE Transactions on Information Theory · 2010 · 2K citations
Ceph: a scalable, high-performance distributed file system
Sage A. Weil, Scott Brandt, Ethan L. Miller et al. · eScholarship (California Digital Library) · 2006 · 1.4K citations · Full text
Erasure coding in windows azure storage
Cheng Huang, Huseyin Simitci, Yikang Xu et al. · 2012 · 806 citations