Enabling Docker Containers for High-Performance and Many-Task Computing

Abstract

Docker is the most popular and user friendly platform for running and managing Linux containers. This is proven by the fact that vast majority of containerized tools are packaged as Docker images. A demanding functionality is to enable running Docker containers inside HPC job scripts for researchers to make use of the flexibility offered by containers in their real-life computational and data intensive jobs. The main two questions before implementing such functionality are: how to securely run Docker containers within cluster jobs? and how to limit the resource usage of a Docker job to the borders defined by the HPC queuing system? This paper presents Socker, a secure wrapper for running Docker containers on Slurm and similar queuing systems. Socker enforces the execution of containers within Slurm jobs as the submitting user instead of root, as well as enforcing the inclusion of containers in the cgroups assigned by the queuing system to the parent jobs. Different from other Docker supported containers-for-hpc platform, socker uses the underlaying Docker engine instead of replacing it. To eveluate socker, it has been tested for running MPI Docker jobs on Slurm. It has been also tested for Many-task computing (MTC) on interconnected clusters. Socker has proven to be secure, as well as introducing no additional overhead to the one introduced already by the Docker engine.