OpenOnDemand Portal on Status of the Grex HPC system

[Resolved] Planned Grex outage - updating SLURM and home

Wed, 02 Sep 2026 08:30:00 +0000

Sept 8, outage complete

All the /project migration jobs were completed. Grex is fully available.

Status update as of evening of Sept 4

We have to extend the ongoing outage past the planned Sept.4 ending. The tentative ETA is September 8, 2026.

We have successfully updated Grex’s SLURM scheduler to a current version to address a CVE.
We have also migrated all user /home directories to the new storage server. This should be transparent for all users.
We are still in the process of migration of /project directories to the new /project filesystem appliance.

Due to some issues we have encountered during the storage, our data migration to the new /project filesystem took longer than the expected outage window. Thus we have to extend the Grex outage to September 8. Some functionality like OpenOnDemand and Nextcloud is not available until the end of the outage.

However, we are able to open Grex for SSH access now, to groups whose /project had been migrated, so that they could log in, access their data, run their jobs and thus test the system. Access to Grex is blocked for the groups whose data are still in the process of migration. As of now, about 95% of all projects were migrated. We apologize for the delay.

A planned Grex outage

We will be performing a planned, full outage of the Grex HPC system, starting at 8:30AM on Tuesday, September 2 2026 We expect the outage to last until end of the day on Friday, September 4.

We will perform update of the SLURM controller, and NFS storage, as well as a minor Linux update on all the compute nodes. During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Unplanned Grex outage - restarting login nodes and OOD

Thu, 07 May 2026 15:00:01 +0000

We are performing a restart of login nodes of Grex, and OOD web portal. Thus access to the system will be temporarily unavailable, for about an hour. Running jobs and data/storage will not be affected. Thank you for your patience!

[Resolved] Planned Grex outage - updating storage

Mon, 08 Dec 2025 08:00:00 +0000

Planned /project storage outage resolved

The project outage is resolved.

A planned /project storage outage

We will be performing a planned outage of the Grex HPC system, starting at 8AM on Monday, December 8, 2025. We expect the outage to last one working day.

We will perform hardware maintenance on our Lustre storage controller.

During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Planned Grex outage, updating storage and patching operating system

Wed, 24 Sep 2025 08:00:00 +0000

The outage completed

Grex is operating normally. Queued jobs were not affected and are now running. Thank you for your patience! If you notice any issues or abnormalites, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A planned /project storage and Compute outage

A /project storage outage to update Lustre filesystem software is in progress on Grex! This requires a quiet filesystem so the system is drained from running jobs and access throgh login nodes and OpenOndemand is closed to users. We will also use the outage to apply various security patches to the Linux OS on Grex , as well as performing other software updates. ETA for the outage end is Friday , Sept 26. Thank you for your patience!

[Resolved] Unplanned power outage in HPCC, Grex down

Wed, 23 Apr 2025 09:10:00 +0000

Power to the HPCC Centre restored

Manitoba Hydro had restored power to Campus, and Grex is back online. All running and queued jobs were lost during the outage. We have used the opportunity to update SLURM scheduler to the current major version 24.11 . All Grex subsystems (compute, storage, login nodes and Web portal) are operational.

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A power outage happened in HPCC Centre

A power outage in Grex’s datacentre happened , with a complete loss of power at about 9:10 AM Winnipeg time. The system is down. The reason for the outage is a problem at Manitoba Hydro, our electricity provider.

https://account.hydro.mb.ca/Portal/outeroutage.aspx

We are waiting for the power to be restored. Thank you for your patience!

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

[Resolved] Planned HPCC datacentre power outage

Sun, 23 Feb 2025 07:00:00 +0000

HPCC planned power outage update

The outage started on Feb 23 is over. Grex is operational. Some of the GPU compute nodes may still be unavailable, and will be in production shortly.

HPCC planned power outage

Physical Plant had informed us that it plans to shut down power feed for transformer work, to several buildings including HPCC where Grex is located. For this reason, we have a complete Grex outage starting from 7 AM, Feb 23, 2025. The planned end of the outage is by the end of the day on Monday, Feb 24. The system is completely be unavailable to the users. Login nodes and data are not be accessible during the outage.

OpenOnDemand Portal on Status of the Grex HPC system

[Resolved] Planned Grex outage - updating SLURM and home

Sept 8, outage complete

Status update as of evening of Sept 4

A planned Grex outage

[Resolved] Unplanned Grex outage - restarting login nodes and OOD

Unplanned reboot of login nodes and OOD server

[Resolved] Planned Grex outage - updating storage

Planned /project storage outage resolved

A planned /project storage outage

[Resolved] Planned Grex outage, updating storage and patching operating system

The outage completed

A planned /project storage and Compute outage

[Resolved] Unplanned power outage in HPCC, Grex down

Power to the HPCC Centre restored

A power outage happened in HPCC Centre

[Resolved] Planned HPCC datacentre power outage

HPCC planned power outage update

HPCC planned power outage