A package for OpenCL based heterogeneous computing on clusters with many GPU devices

TLDR

Heterogeneous systems offer new opportunities to boost parallel application performance on CPU‑GPU clusters, yet current GPU applications run only on local devices within their hosting nodes. This paper introduces a package that enables OpenMP, C++ and unmodified OpenCL applications to run on clusters with many GPU devices. The Many GPUs Package (MGP) implements OpenCL specifications and extends the OpenMP API to transparently access cluster‑wide CPUs and GPUs, while providing task‑dependency scheduling and buffer management to simplify programming. The authors present MGP and demonstrate its internal performance, showing efficient utilization of cluster GPUs.

Abstract

Heterogeneous systems provide new opportunities to increase the performance of parallel applications on clusters with CPU and GPU architectures. Currently, applications that utilize GPU devices run their device-executable code on local devices in their respective hosting-nodes. This paper presents a package for running OpenMP, C++ and unmodified OpenCL applications on clusters with many GPU devices. This Many GPUs Package (MGP) includes an implementation of the OpenCL specifications and extensions of the OpenMP API that allow applications on one hosting-node to transparently utilize cluster-wide devices (CPUs and/or GPUs). MGP provides means for reducing the complexity of programming and running parallel applications on clusters, including scheduling based on task dependencies and buffer management. The paper presents MGP and the performance of its internals.