Running NWChem on Grex

Introduction#

NWChem is a Scalable open source solution for large scale molecular simulations. NWChem is actively developed by a consortium of developers and maintained by the EMSL located at the Pacific Northwest National Laboratory (PNNL) in Washington State. The code is distributed as open-source under the terms of the Educational Community License version 2.0 (ECL 2.0).

System specific notes#

On the Grex software stack, NWChem is using OpenMPI 3.1 with Intel compilers toolchains. To find out which versions are available, use module spider nwchem.

For a version 7.2.2, at the time of writing the following modules must be loaded:

module load arch/avx512 intel-one/2024.1 openmpi/4.1.6 
module load nwchem/7.2.2

or

module load arch/avx512  aocc/4.2.0  openmpi/4.1.6
module load nwchem/7.2.2+aocl-4.2.0-64

or

module load arch/avx512  gcc/13.2.0  openmpi/4.1.6 
module load nwchem/7.2.2+aocl-4.2.0-64

The latter two versions might be slightly faster on AMD-based partitions (such as genoa), while the former will be ffaster on Intel CPUs (skylake and largemem partitions).

The NWChem on Grex was built with the ARMCI variant MPI-PR . Thus, NWCHem needs at least One process per node reserved for data communication. To run a serial job, one needs 2 tasks per node. To run a 22-core job over two whole nodes, one should ask for 2 nodes, 12 tasks per node. Simple number of tasks specification likely won’t work because of the chance of having a single-task node allocated by SLURM; so --nodes= --ntask-per-node specification is required.

Script example for running NWChem on Grex
run-nwchem.sh
#!/bin/bash

#SBATCH --ntasks-per-node=7 --nodes=2 --cpus-per-task=1
#SBATCH --mem-per-cpu=2000mb
#SBATCH --time=0-3:00:00
#SBATCH --job-name=NWchem-dft-test

# Adjust the number of tasks, time and memory required.
# the above spec is for 12 compute tasks over two nodes.

# Load the modules:

module load arch/avx512 intel-one/2024.1 openmpi/4.1.6 
module load nwchem/7.2.2

echo "Starting run at: `date`"

which nwchem

# Uncomment/Change these in case you want to use custom basis sets

NWCHEMROOT=${MODULE_NWCHEM_PREFIX}
export NWCHEM_NWPW_LIBRARY=${NWCHEMROOT}/data/libraryps
export NWCHEM_BASIS_LIBRARY=${NWCHEMROOT}/data/libraries

# In most cases SCRATCH_DIR would  be on local nodes scratch
# While results are in the same directory

export NWCHEM_SCRATCH_DIR=$TMPDIR
export NWCHEM_PERMANENT_DIR=`pwd`

# Optional memory setting; note that this one or the one in your code
# must match the #SBATCH --mem-per-cpu times compute tasks  !

export NWCHEM_MEMORY_TOTAL=2000000000 # 24000 MB, double precision words only
export MKL_NUM_THREADS=1

srun nwchem  dft_feco5.nw > dft_feco5.$SLURM_JOBID.log

echo "Program finished with exit code $? at: `date`"

Assuming the script above is saved as run-nwchem.sh, it can be submitted with:

sbatch run-nwchem.sh

For more information, visit the page running jobs on Grex