A Data Distribution Service for Cloud and Containerized Storage Based on Information Dispersal

Abstract

Information dispersal is a fault-tolerant technique where files of size |F| are split into n redundant pieces of size |F|/k that are dispersed to different servers where k pieces suffice for recovering the original file whenever k<;n. This technique is a popular solution for service providers to withstand server failures and to improve the storage utilization. However, the coding/decoding service time produced by this technique as well as the management of pieces of heterogeneous size, that belong to different files, represent both a challenge for the deployment of this technique on clouds and clusters. This paper presents the design and development of a data distribution service for fault-tolerant cloud/cluster storage. This service includes an information dispersal client for coding/decoding files in-memory, which improves the service experience of end-users when delivering/retrieving files to/from cloud storage services. It also includes a data placement method to allocate, locate and manage redundant pieces of heterogeneous size in a uniform manner, which produces load balancing in the storage nodes. A prototype of this service was implemented in a private cloud and containerized cluster. An experimental evaluation based on synthetic traces and a case study based on satellite images revealed that the service prototype preserved a balanced load even in scenarios when managing pieces of heterogeneous size and that, when performing coding/decoding in-memory, the service experience of end-users was improved in comparison with tested traditional solutions.