On the Synchronization Bottleneck of OpenStack Swift-Like Cloud Storage Systems

Abstract

As one type of the most popular cloud storage services, OpenStack Swift and its follow-up systems replicate each object across multiple storage nodes and leverage <i>object sync protocols</i> to achieve high reliability and <i>eventual consistency</i> . The performance of object sync protocols heavily relies on two key parameters: <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> (number of replicas for each object) and <inline-formula><tex-math notation="LaTeX">$n$</tex-math></inline-formula> (number of objects hosted by each storage node). In existing tutorials and demos, the configurations are usually <inline-formula> <tex-math notation="LaTeX">$r=3$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$n&lt;1,000$</tex-math></inline-formula> by default, and the sync process seems to perform well. However, we discover in data-intensive scenarios, e.g., when <inline-formula> <tex-math notation="LaTeX">$r&gt;3$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$n\gg 1,000$</tex-math></inline-formula> , the sync process is significantly delayed and produces massive network overhead, referred to as the <i>sync bottleneck problem</i> . By reviewing the source code of OpenStack Swift, we find that its object sync protocol utilizes a fairly simple and network-intensive approach to check the consistency among replicas of objects. Hence in a sync round, the number of exchanged hash values per node is <inline-formula><tex-math notation="LaTeX">$\Theta (n\times r)$</tex-math> </inline-formula> . To tackle the problem, we propose a lightweight and practical object sync protocol, <i>LightSync</i> , which not only remarkably reduces the sync overhead, but also preserves high reliability and eventual consistency. LightSync derives this capability from three novel building blocks: 1) <i>Hashing of Hashes</i> , which aggregates all the <inline-formula><tex-math notation="LaTeX">$h$</tex-math></inline-formula> hash values of each data partition into a single but representative hash value with the Merkle tree; 2) <i>Circular Hash Checking</i> , which checks the consistency of different partition replicas by only sending the aggregated hash value to the clockwise neighbor; and 3) <i>Failed Neighbor Handling</i> , which properly detects and handles node failures with moderate overhead to effectively strengthen the robustness of LightSync. The design of LightSync offers provable guarantee on reducing the per-node network overhead from <inline-formula><tex-math notation="LaTeX">$\Theta (n\times r)$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$\Theta (\frac{n}{h})$</tex-math></inline-formula> . Furthermore, we have implemented LightSync as an open-source patch and adopted it to OpenStack Swift, thus reducing the sync delay by up to 879 <inline-formula><tex-math notation="LaTeX">$\times$</tex-math></inline-formula> and the network overhead by up to 47.5 <inline-formula><tex-math notation="LaTeX">$\times$</tex-math></inline-formula> .