2014 · 87 citations · 25 references
ComputingComputer EngineeringComputer ScienceFault ToleranceFault-tolerant SystemsStorage SystemsData DurabilityDistributed StorageFault-tolerant Storage SystemsReliable Storage SystemsStorage Fault ToleranceStorage ReliabilityDistributed Storage ArchitecturesDistributed Storage RecoveryNode FailuresReplication PerformanceResilient Data ReplicationStorage Cost OptimizationStorage Recovery
Erasure coding schemes provide higher durability at lower storage cost, and thus constitute an attractive alternative to replication in distributed storage systems, in particular for storing rarely accessed "cold" data. These schemes, however, require an order of magnitude higher recovery bandwidth for maintaining a constant level of durability in the face of node failures. In this paper we propose lazy recovery, a technique to reduce recovery bandwidth demands down to the level of replicated storage. The key insight is that a careful adjustment of recovery rate substantially reduces recovery bandwidth, while keeping the impact on read performance and data durability low. We demonstrate the benefits of lazy recovery via extensive simulation using a realistic distributed storage configuration and published component failure parameters. For example, when applied to the commonly used RS(14, 10) code, lazy recovery reduces repair bandwidth by up to 76% even below replication, while increasing the amount of degraded stripes by 0.1 percentage points. Lazy recovery works well with a variety of erasure coding schemes, including the recently introduced bandwidth efficient codes, achieving up to a factor of 2 additional bandwidth savings.
25
The Hadoop Distributed File System
Konstantin V. Shvachko, Hairong Kuang, Sanjay Radia et al. · 2010 · 4.8K citations
John Kubiatowicz, David Bindel, Yan Chen et al. · ACM SIGPLAN Notices · 2000 · 2K citations · Full text
Network Coding for Distributed Storage Systems
Alexandros G. Dimakis, P. Brighten Godfrey, Yunnan Wu et al. · IEEE Transactions on Information Theory · 2010 · 2K citations
Sanjay Ghemawat, Howard Gobioff, Shun-Tak A. Leung · 2003 · 1.4K citations