A performance evaluation and examination of open-source erasure coding libraries for storage

Abstract

Over the past five years, large-scale storage installa-tions have required fault-protection beyond RAID-5, leading to a flurry of research on and development of erasure codes for multiple disk failures. Numerous open-source implementations of various coding tech-niques are available to the general public. In this pa-per, we perform a head-to-head comparison of these implementations in encoding and decoding scenarios. Our goals are to compare codes and implementations, to discern whether theory matches practice, and to demonstrate how parameter selection, especially as it concerns memory, has a significant impact on a code’s performance. Additional benefits are to give storage system designers an idea of what to expect in terms of coding performance when designing their storage sys-tems, and to identify the places where further erasure coding research can have the most impact. 1