Occupy the cloud

TLDR

Distributed computing remains largely inaccessible to many users because cluster management and configuration tools are complex, even for simple embarrassingly parallel jobs, despite the availability of open‑source platforms and commercial offerings. The authors propose that stateless functions provide a viable platform for these users by eliminating cluster‑management overhead and delivering elastic scalability. Using their prototype PyWren, they demonstrate that the stateless‑function model can efficiently implement various distributed computing paradigms, including bulk‑synchronous parallel (BSP). Based on recent increases in network bandwidth and the emergence of disaggregated storage, they argue that stateless functions are a natural fit for data processing in future computing environments.

Abstract

Distributed computing remains inaccessible to a large number of users, in spite of many open source platforms and extensive commercial offerings. While distributed computation frameworks have moved beyond a simple map-reduce model, many users are still left to struggle with complex cluster management and configuration tools, even for running simple embarrassingly parallel jobs. We argue that stateless functions represent a viable platform for these users, eliminating cluster management overhead, fulfilling the promise of elasticity. Furthermore, using our prototype implementation, PyWren, we show that this model is general enough to implement a number of distributed computing models, such as BSP, efficiently. Extrapolating from recent trends in network bandwidth and the advent of disaggregated storage, we suggest that stateless functions are a natural fit for data processing in future computing environments.