Containers for Software

Introduction#

Linux Containers are means to isolate software dependencies from the base Linux operating system. On Grex, we support the Singularity container system, developed by a company called SyLabs. There was a fork of the Singularity project that produced a new project called Apptainer . As of now (early 2024), Singularity-CE by Sylabs and Apptainer by Linux Foundations are using the same SIF image format and thus are largely the same with respect to their usage and main features. Grex supports Singularity-CE, while National DRI (the Alliance) HPC machines like Cedar or Graham support Apptainer.

Several different Linux container engines exist, most notably Docker which is a very popular tool in the DevOps community. Presently Docker containers cannot be directly supported on shared HPC systems like Grex. However, with help of Singularity, it is possible to run Docker images from DockerHub , as well as native Singularity images from other repositories, such as SingularityHub and SyLabsCloud .

Using Singularity on Grex#

Start with module spider singularity; it will list the current version. Due to the nature of container runtime environments, we update Singularity regularly, so the installed version is usually the latest one. Load the module (in the default Grex environment) by the following command:

module load singularity

With singularity command, one can list singularity commands and their options:

module load singularity
singularity help

A brief introduction on getting started with Singularity can be useful to get started. You will not need to install Singularity on Grex since it is already provided as a module.

To execute an application within the container, do it in the usual way for that application, but prefix the command with ‘‘singularity exec image_name.sif". For example, to run R on an R script, using a container named R-INLA.sif:

singularity exec ./R-INLA.sif R --vanilla < myscript.R

Quite often, it is useful to provide the containerized application with data residing on a shared HPC filesystem such as /home or /global/scratch. This is done via bind mounts . Normally, the container bind-mounts $HOME, /tmp and the current working directory. On Grex to bind the global Lustre filesystem the following -B option should be used:

singularity exec -B /sbb/scratch:/global/scratch ./R-INLA.sif R --vanilla < myscript.R

In case you do not want to mount anything to preserve the containers’ environment from any overlapping of data/code from say $HOME, use the --containall flag.

Some attention has to be paid to Singularity’s local cache and temporary directories. Singularity caches the container images it pulls and Docker layers under $HOME/.singularity. Containers can be large, in tens of gigabytes, and thus they can easily accumulate and exhaust the users’ storage space quota on $HOME. Thus, users might want to set the SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR variables to some place under their /global/scratch space.

For example, to change the location of SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR, one might run:

mkdir -p /global/scratch/$USER/singularity/{cache,tmp}
export SINGULARITY_CACHEDIR="/global/scratch/$USER/singularity/cache"
export SINGULARITY_TMPDIR="/global/scratch/$USER/singularity/tmp"

before building the singularity image.

Getting and building Singularity images#

The commands singularity build and singularity pull would get Singularity images from DockerHub, SingularityHub or SyLabsCloud. Images can also be built from other images, and from recipes. A recipe is a text file that specifies the base image and post-install commands to be performed on it.

Singularity with GPUs#

Use the --nv flag to singularity run/exec/shell commands. Naturally, you should be on a node that has a GPU, in an interactive job. NVIDIA provides many pre-built Docker and Singularity container images on their “GPU cloud” , together with instructions on how to pull them and to run them. These should work on Grex without much changes.

Using Apptainer from CC Stack on Grex#

The Alliance’s (formerly ComputeCanada) software stack now provides Apptainer modules in the two latest Standard Environments , StdEnv/2020 and StdEnv/2023. Most recent Apptainer versions (1.2.4 and older) do not require “suexec” and thus can be used off the CVMFS as usual. The only caveat would be to first unload any “singularity” or “apptainer” modules from other software stacks by module purge.

module purge
module load CCEnv
module load arch/avx512 
module load StdEnv/2023
module load apptainer

apptainer version
apptainer run docker://ghcr.io/apptainer/lolcow

OpenScienceGrid CVMFS#

We can run Singularity containers distributed with OSG CVMFS which is currently mounted on Grex’s CVMFS. The containers are distributed via CVMFS as unpacked directory images. So, the way to access them is to find a directory of interest and point singularity runtime to it. The directories will then be mounted and fetched automatically. The repository starts with /cvmfs/singularity.opensciencegrid.org/. Then you’d need an idea from somewhere what you are looking for in the subdirectories of the above-mentioned path. An example (accessing, that is, exploring via singularity shell command, Remoll software distributed through OSG CVMFS by jeffersonlab):

module load singularity
singularity shell /cvmfs/singularity.opensciencegrid.org/jeffersonlab/remoll\:develop

It looks like the list of what is present on the OSG CVMFS is on Github: OSG Github docker images .