Exercise for running local models on HPC Grex

Introduction#

An example running local llama.cpp AI models on Grex with GPUs, and using them with an agentic harness. Need CCDB and Grex access! Let us know your username ASAP, ask TA!

Getting locally installed llama.cpp#

The code needs a GPU node , so lets request one in an interactive job. The job takes one GPU on any of Grex GPU-enabled partitions.

salloc  --nodes=1 --gpus=1 --cpus-per-gpu=4 --mem=12000 --time=0-3:00 --partition=gpu,stamps-b,livi-b,mcordgpu-b,agpu,lgpu --reservation=ws_gpu

Check if we have GPUs.

hostname
nvidia-smi

Now we will need the Llama.cpp executables. It is either available as a module, or can be used in Apptainer. This time we will use a system wide Grex install.

module load cuda/12.9.1 arch/avx2 gcc/14.3
module load llama-cpp

which llama-cli

Running a local LLM interactively.#

We will also need models. Models are large files , usually downloaded from HuggingFace. Some of them are pre-downloaded, find them in the /home/shared/models. Let us load one of them and see if it works.

In the same interactive GPU job, lets run a model from Meta.

llama-cli -m /home/shared/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 4096

Now we have the code, the model and the GPU, and can try loading the model into LLama.cpp and issuing a simple prompt.

Does it say anything useful? Try other models?

Running local inference engine Llama.cpp as a SLURM job#

We’d want to run a persistent server job that does not quit. We will start llama-server, send the LLama.cpp into background with & and we will wait. We will also set an authentication key.

!!Do not forget to use scancel JobID when done to prevent wasting a GPU!!

Lets use VI editor and save the following job script as llama-server.job:

#!/bin/bash
#SBATCH --job-name=llama-server
#SBATCH --partition=gpu,agpu,lgpu,mcordgpu-b,livi-b,stamps-b --reservation=ws_gpu
#SBATCH --gpus=1
#SBATCH --cpus-per-task=4 --mem=12000 --time=0-3:00:00
#SBATCH --output=llama-%j.out
#SBATCH --error=llama-%j.err


#MODEL_PATH=~/models/Spark-X2.6-4B-Q6_K_S.gguf

PORT=8080
KEY="my-key"

#on Grex !!!ADD YOUR user Number here prefixed by 8!!! 8251 for PORT

#on Grex do modules below:
module load cuda/12.9.1 arch/avx2 gcc/14.3
module load llama-cpp

which llama-cli

# Run the server in model router mode (directory)

llama-server   --host 0.0.0.0   --port $PORT   -c 8192 -n 4096   -ngl 99   --jinja   --temp 0.7   -t 2  --models-dir /home/shared/models  --no-models-autoload --api-key "$KEY"   > server.log 2>&1 &

echo "Server started on port $PORT. Use SSH tunnel to access."
wait
# done

Then submit the script with sbatch llama-server.slurm command.

Connect to the model with curl to see if the server is alive. Replace the node name with what you see in sq for your GPU job. (If using different PORT, replace 8080 with the port ).

curl -H 'Authorization: Bearer my-key'   http://gXYZ:8080/health

curl -H 'Authorization: Bearer my-key'   http://gXYZ:8080/models | jq .

The model should be servable if the two above commands succeed. We also can see the models it has configured to use.

Next exercise: Connecting PI agent to LLama#

We will try to use our agentic harness now! Or at least connecting it. One lightweight, popular harness is PI .

The code needs some CPU and memory, but no GPUs , so lets request aniother interactive job. The job takes one full gpu-node.

Note should be ok on the login node on Grex! skipt the step below

salloc --partition=skylake --nodes=1 --cpus-per-task=2 --mem=4000 --time=0-3:00

Then, lets install PI.

# trusting random s**t on Internets looks like this:
curl -fsSL https://pi.dev/install.sh | sh
which pi

We need to configure it. Start pi in a terminal window Then use various command to configure the API endporint to our LLama server. find the server name using sq command and noticing where the job runs (gpu-nodeX on MC, some gXYZ=_ on Grex). Lets use PI’s command area:

  • /login llama.cpp
  • Server URL: (when asked, enter the URL as http://gpu-nodeX:8080 )
  • APi Key: (when asked, enter the model Key used in the LLama job)
  • /llama
  • Select models to “load”. Use the ones we have under /home/shared/models
  • /model
  • Select the model to be used (there are many; use the ones we just loaded)

At this point, asking the model questions and trying to perform actions should work.

try to create a SLURM job script#

Lets try to create SLURM job script for some of the applications (Gromacs?) .

Working with skills, and prompts#

You may have noticed that the agent is not very smart. It needs more guidance, and these “skills” it can self-edit and improve. It also helps to save repeateable “prompts”.

# finding where the confuguration is
ls ~/.pi/agent/
# would have some 
mkdir -p ~/.pi/agent/prompts
mkdir -p ~/.pi/agent/skills
mkdir -p ~/.pi/agent/skills/hpc-site/references

Use VI to create a prompt.

vim ~/.pi/agent/prompts/learn.md
#
vim ~/.pi/agent/skills/hpc-site/SKILL.md

We can copy our long pre-made “skills” into the references:

cp -r /home/shared/agent/skills/*.md  ~/.pi/agent/skills/hpc-site/references
#
ls  ~/.pi/agent/skills/hpc-site/references

try to create a SLURM job script#

Lets try to create SLURM job script for some of the applications (Gromacs?) .

Cancel the GPU job to free the resources#

Use sq to find your job id, and scancel to cancel it.