Programming your network at run-time for big data applications

TLDR

Recent advances in software‑defined networking and optical switching enable end‑to‑end programmability of the network stack from physical topology to flow‑level traffic control. The paper explores tight integration of SDN controllers with optical switching to study run‑time network configuration for big data applications, aiming to jointly optimize application performance and network utilization. Using Hadoop as a case study, the authors illustrate an integrated network control architecture that includes job scheduling, topology, and routing configuration mechanisms for Hadoop jobs. Analysis shows that integrated control can improve application performance with minimal configuration overhead, indicating early promise for achieving long‑term tight network and application integration via SDN.

Abstract

Recent advances of software defined networking and optical switching technology make it possible to program the network stack all the way from physical topology to flow level traffic control. In this paper, we leverage the combination of SDN controller with optical switching to explore the tight integration of application and network control. We particularly study the run-time network configuration for big data applications to jointly optimize application performance and network utilization. We use Hadoop as an example to discuss the integrated network control architecture, job scheduling, topology and routing configuration mechanisms for Hadoop jobs. Our analysis suggests that such an integrated control has great potential to improve application performance with relatively small configuration overhead. We believe our study shows early promise of achieving the long-term goal of tight network and application integration using SDN.