Broom: sweeping out garbage collection from big data systems

Abstract

Many popular systems for processing “big data ” are im-plemented in high-level programming languages with automatic memory management via garbage collection (GC). However, high object churn and large heap sizes put severe strain on the garbage collector. As a result, ap-plications underperform significantly: GC increases the runtime of typical data processing tasks by up to 40%. We propose to use region-based memory management instead of GC in distributed data processing systems. In these systems, many objects have clearly defined life-times. Hence, it is natural to allocate these objects in fate-sharing regions, obviating the need to scan a large heap. Regions can be memory-safe and could be in-ferred automatically. Our initial results show that region-based memory management reduces emulated Naiad ver-tex runtime by 34 % for typical data analytics jobs. 1