Network coding for large scale content distribution

TLDR

Large unstructured overlay networks require nodes to make block forwarding decisions based on local information, making efficient distribution challenging. The authors propose a new network‑coding scheme for distributing large files. The scheme uses network coding so that every node can generate and transmit encoded blocks, randomizing transmission to simplify scheduling, and is evaluated against unencoded and source‑only coding in heterogeneous, dynamic, clustered networks with incentive mechanisms. Simulations show that network coding reduces file download time by 20–30 % versus server‑only coding and 2–3 × versus unencoded transmission, while also improving robustness and handling extreme node departures.

Abstract

We propose a new scheme for content distribution of large files that is based on network coding. With network coding, each node of the distribution network is able to generate and transmit encoded blocks of information. The randomization introduced by the coding process eases the scheduling of block propagation, and, thus, makes the distribution more efficient. This is particularly important in large unstructured overlay networks, where the nodes need to make block forwarding decisions based on local information only. We compare network coding to other schemes that transmit unencoded information (i.e. blocks of the original file) and, also, to schemes in which only the source is allowed to generate and transmit encoded packets. We study the performance of network coding in heterogeneous networks with dynamic node arrival and departure patterns, clustered topologies, and when incentive mechanisms to discourage free-riding are in place. We demonstrate through simulations of scenarios of practical interest that the expected file download time improves by more than 20-30% with network coding compared to coding at the server only and, by more than 2-3 times compared to sending unencoded information. Moreover, we show that network coding improves the robustness of the system and is able to smoothly handle extreme situations where the server and nodes leave the system.