Fast replication in content distribution overlays

Abstract

We present SPIDER-a system for fast replication or distribution of large content from a single source to multiple sites interconnected over Internet or via a private network. In order to exploit spatial diversity of the underlying network, SPIDER uses an overlay structure composed of dedicated transit nodes (TNs). The data transport mechanism in SPIDER leverages this overlay structure to provide a coordinated approach that minimizes the maximum time to replicate to all destination sites (the make span of content replication). In order to achieve this objective, SPIDER employs two orthogonal components: a) creation of multiple dynamic distribution trees using the transit nodes b) end-to-end reliable data transport with flow control on these trees by chaining point-to-point TCPs. We further present simulations based results to quantify benefits of tree construction algorithms in random topologies. We evaluate the real implementation of the SPIDER in Planet Lab and observe a 2-6 times speed up compared to different existing schemes.