Running NWChem on Grex

Introduction#

NWChem is a Scalable, massive parallel and open source solution for large scale molecular simulations. NWChem is actively developed by a consortium of developers and maintained by the EMSL located at the Pacific Northwest National Laboratory (PNNL) in Washington State. The code is distributed as open source under the terms of the Educational Community License version 2.0 (ECL 2.0).

System specific notes#

To find out which versions of NWChem are available, use module spider nwchem .

For a version 7.2.2, and using the local SBEnv software stack, at the time of writing the following modules must be loaded:

module load arch/avx512 intel-one/2024.1 openmpi/4.1.6 
module load nwchem/7.2.2

or

module load arch/avx512  aocc/4.2.0  openmpi/4.1.6
module load nwchem/7.2.2+aocl-4.2.0-64

or

module load arch/avx512  gcc/13.2.0  openmpi/4.1.6 
module load nwchem/7.2.2+aocl-4.2.0-64

By inspecting the dependencies, you can see that the above versions are different with respect the compilers used to build NWChem: these are IntelOneAPI compiler, GNU GCC v 13, and AMD AOCC 4.2, correspondingly. The latter two versions might be slightly faster on AMD-based partitions (such as genoa, genlm), while the former will be faster on Intel CPUs (skylake and largemem partitions).

The NWChem on Grex was built with the ARMCI variant MPI-PR . Thus, NWCHem needs at least One process per node reserved for data communication. To run a serial job, one needs 2 tasks per node. To run a 22-core job over two whole nodes, one should ask for 2 nodes, 12 tasks per node. Simple number of tasks specification likely won’t work because of the chance of having a single-task node allocated by SLURM; so --nodes= --ntask-per-node specification is required.

Script example for running NWChem on Grex
run-nwchem.sh
#!/bin/bash

#SBATCH --ntasks-per-node=7 --nodes=2 --cpus-per-task=1
#SBATCH --mem-per-cpu=2000mb
#SBATCH --time=0-3:00:00
#SBATCH --job-name=NWchem-dft-test

# Adjust the number of tasks, time and memory required.
# the above spec is for 12 compute tasks over two nodes.

# Load the modules:

module load arch/avx512 intel-one/2024.1 openmpi/4.1.6 
module load nwchem/7.2.2

echo "Starting run at: `date`"

which nwchem

# Uncomment/Change these in case you want to use custom basis sets

NWCHEMROOT=${MODULE_NWCHEM_PREFIX}
export NWCHEM_NWPW_LIBRARY=${NWCHEMROOT}/data/libraryps
export NWCHEM_BASIS_LIBRARY=${NWCHEMROOT}/data/libraries

# In most cases SCRATCH_DIR would  be on local nodes scratch
# While results are in the same directory

export NWCHEM_SCRATCH_DIR=$TMPDIR
export NWCHEM_PERMANENT_DIR=`pwd`

# Optional memory setting; note that this one or the one in your code
# must match the #SBATCH --mem-per-cpu times compute tasks  !

export NWCHEM_MEMORY_TOTAL=2000000000 # 24000 MB, double precision words only
export MKL_NUM_THREADS=1

srun nwchem  dft_feco5.nw > dft_feco5.$SLURM_JOBID.log

echo "Program finished with exit code $? at: `date`"

Assuming the script above is saved as run-nwchem.sh, it can be submitted with:

sbatch run-nwchem.sh

For more information, visit the page running jobs on Grex