Software and Applications

Software#

In HPC, “software” typically refers to scientific and engineering codes for computation, data processing, and visualization -— rather than common business IT applications like web services or email clients. Without software to run workloads, HPC systems would be of little use. Development tools and libraries used to build HPC applications are also considered to be “software”.

Fortunately, most HPC systems, including Grex, come with a curated set of pre-installed software. This section explains how to find, access, and manage HPC software, and what to do if some software item you need is missing.

How software is installed and distributed#

Linux software is often installed system-wide using package managers (e.g., apt on Debian and Ubuntu, yum or dnf on RedHat and its derivatives). These tools install binaries into standard locations like /usr/bin, making them immediately available via the system PATH. (PATH is a variable specifying where the operating systems would look for executable code).

While convenient on personal machines, this approach doesn’t scale well to HPC environments due to:

  • The need for users to have privileged (root) access, which can compromise stability and security of shared HPC systems.
  • Package managers typically support only a single, newest version of each package at a time, which hinders reproducibility.
  • Software must be consistently installed across all compute nodes.
  • Public binary packages are usually compiled for generic CPUs, not optimized for a specific HPC systems hardware.

Because of these limitations, HPC clusters usually do not rely on OS software repositories. They only include a minimal base OS install. Users don’t have access to system package managers. Instead, application software is built from source and installed on a shared filesystem accessible by all nodes. Multiple versions can coexist, and software dependencies are managed with Environment Modules .

Modules allow users to dynamically modify their environment (e.g., PATH, LD_LIBRARY_PATH, CPATH, MKLROOT, HDF5HOME) to load or unload specific software packages.

Lmod configuration on Grex#

Grex uses Lmod , a modern, Lua-based module system developed at Texas Advanced Computing Center. Lmod supports hierarchical modules, which organize software based on dependencies like CPU architecture, compilers, and MPI implementations. Using module hierarchies helps to avoid conflicts between incompatible software toolchains.

Lmod would:

  • Hide software items for which their dependencies/toolchains are not yet loaded.
  • Automatically unload conflicting modules when switching toolchains.
  • Reload required dependencies as needed when a module is switched.
  • Supports multiple software stacks through configurable module paths.

For more information about Lmod, please refer to the software stacks available on the using modules page .

There are the following Grex-specific conventions on how Grex software environment is configured:

  • There is a module hierarchy. No software modules are loaded by default!
  • There is more than one “software” stacks. A special module “environment” needs to be loaded first to switch between them.
  • CPU architecture (a module called arch ) is often a root of a module hierarchy. Most commonly used arch/avx512 .
  • For the local software stack, cuda module precees the arch module for GPU software.

On Grex, there are two software stacks, called SBEnv and CCEnv, standing for the software built on Grex locally and the software environment from the Alliance (Compute Canada), correspondingly. In practice, all the above means that a required “software stack” module must be loaded first.

In the case of CCEnv, it must be followed by a StdEnv module to pick the year/version of it. This would load a number of default modules like compilers and OpenMPI.

For SBEnv , the local environment is the only module loaded by default. This enables the “Core” modules that do not depend on CPU architecture. Usually these are commercial and/or binary packages.

Then, a module that picks architecture should be loaded : either cuda and arch (for GPU-based software) or just arch (for CPU-only software).

Note that the order of loading cuda and arch modules on SBEnv matters! cuda must always be loaded first.

In case of CCEnv, it must be followed by loading a StdEnv module to pick the year/version of it. This would load a number of default modules like compilers and OpenMPI. It still is a good practice to pick an arch module correctly for CCEnv.

Note that cuda modules of CCEnv won’t load on a non-GPU hardware. Use interactive jobs on GPU nodes to build cuda enabled software!

How to find the software with Lmod Modules#

First, a software stack must be selected and loaded with module load command.

Lmod can not find modules across different software stacks!

When a software stack module is loaded, the module spider command will find a specific software item (for example, GROMACS; note that all the module names are lower-case on Grex and on Alliance software stacks) if it exists under that stack:

module load SBEnv
module spider gromacs

It might return several versions; then usually a subsequent command with the version is used to determine dependencies required for the software. In case of GROMACS on Grex, at the time of writing, it returns four versions, with gromacs/2024.1 being the latest. Let’s find its dependencies.

module spider gromacs/2024.1

It will advise that there are two sets of dependencies, one for the CPU version and one for the GPU version. For the CPU version, we’d need to load the following modules: " arch/avx512 gcc/13.2.0 openmpi/4.1.6". Note that the first module is for CPU architecture. Most CPU nodes on Grex use arch/avx512. Then, after the dependencies are loaded module load command can be used actually to load the GROMACS environment, for the CPU version.

module load arch/avx512  gcc/13.2.0  openmpi/4.1.6
module load gromacs/2024.1

For more information about using Lmod modules, please refer to the Modules pages on this documentation.

How and when to install software in your HOME directory#

Linux (Unlike some Desktop operating systems) has a concept of user permissions separation. Regular users cannot, unless explicitly permitted, access the system’s files and files of other users.

You can almost always install software without super-user access into your /home/$USER directory. Moreover, you can manage the software with Lmod: Lmod automatically searches for module files under $HOME/modulefiles and adds the modules it discovers there into the modules tree so they can be found by module spider, loaded by module load, etc.

Most Linux software can be installed from sources using either Autoconf or CMake configuration tools. These will accept --prefix=/home/$USER/my-software/version or -DCMAKE_INSTALL_PREFIX=/home/$USER/my-software/version as arguments. These paths are used for the installation directories where the user has full access.

Software that comes as a binary archive to be unpacked can be simply unpacked into your home directory location. Then, the paths should be set for the software to be found: either by including the environment variable in $HOME/.bashrc or in $HOME/.bash_profile or by creating a specific module in $HOME/modulefiles/my-software/version following Lmod instructions for writing Modules .

There exist binary software environments like conda that manage their own tree of binary-everything. These can be used from your home directory as well, with some caution, because automatically pulling every software from a conda channel might conflict with the same software existing in the HPC environment (Python package paths, MPI libraries, etc.).

Note that as of 2024, Anaconda owners have changed their licensing policy. We do not provide any system-wide conda installations on Grex. In case users want to continue using conda, they must be sure that they have a proper Anaconda license to do so. Note also that the same applies for mamba which would use the same conda software channels.

However, if a software is really a part of the base OS (something like a graphics Desktop software, etc.), it can be hard to rebuild from sources due to many dependencies. If needed, it may be better if installed centrally or used in a container, see Containers documentation .