Lustre /Project on Status of the Grex HPC system

[Resolved] Planned Grex outage - updating SLURM and home

Wed, 02 Sep 2026 08:30:00 +0000

Sept 8, outage complete

All the /project migration jobs were completed. Grex is fully available.

Status update as of evening of Sept 4

We have to extend the ongoing outage past the planned Sept.4 ending. The tentative ETA is September 8, 2026.

We have successfully updated Grex’s SLURM scheduler to a current version to address a CVE.
We have also migrated all user /home directories to the new storage server. This should be transparent for all users.
We are still in the process of migration of /project directories to the new /project filesystem appliance.

Due to some issues we have encountered during the storage, our data migration to the new /project filesystem took longer than the expected outage window. Thus we have to extend the Grex outage to September 8. Some functionality like OpenOnDemand and Nextcloud is not available until the end of the outage.

However, we are able to open Grex for SSH access now, to groups whose /project had been migrated, so that they could log in, access their data, run their jobs and thus test the system. Access to Grex is blocked for the groups whose data are still in the process of migration. As of now, about 95% of all projects were migrated. We apologize for the delay.

A planned Grex outage

We will be performing a planned, full outage of the Grex HPC system, starting at 8:30AM on Tuesday, September 2 2026 We expect the outage to last until end of the day on Friday, September 4.

We will perform update of the SLURM controller, and NFS storage, as well as a minor Linux update on all the compute nodes. During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Planned Grex outage - updating storage

Mon, 08 Dec 2025 08:00:00 +0000

Planned /project storage outage resolved

The project outage is resolved.

A planned /project storage outage

We will be performing a planned outage of the Grex HPC system, starting at 8AM on Monday, December 8, 2025. We expect the outage to last one working day.

We will perform hardware maintenance on our Lustre storage controller.

During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Planned Grex outage, updating storage and patching operating system

Wed, 24 Sep 2025 08:00:00 +0000

The outage completed

Grex is operating normally. Queued jobs were not affected and are now running. Thank you for your patience! If you notice any issues or abnormalites, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A planned /project storage and Compute outage

A /project storage outage to update Lustre filesystem software is in progress on Grex! This requires a quiet filesystem so the system is drained from running jobs and access throgh login nodes and OpenOndemand is closed to users. We will also use the outage to apply various security patches to the Linux OS on Grex , as well as performing other software updates. ETA for the outage end is Friday , Sept 26. Thank you for your patience!

[Resolved] Unplanned project storage outage in HPCC

Fri, 18 Jul 2025 07:10:00 +0000

Update 17:30 Central Time: System is available

The storage is working. If you notice any issues with the storage, please do not hesitate to contact us at support@tech.alliancecan.ca, mentioning Grex in the subject line.

Update 17:00 Central Time: System is available with conditions

The failure was traced to a hardware issue on one of the storage controllers. We have cleared the controller state and restarted the /project storage appliance. However, the storage targets are not yet properly balanced across the storage servers, so there can be some performance issues. We are working with the storage vendor to address these issues.

As of now, OpenOnDemand and /project are now available, and Grex system would accept new jobs.

A /project storage outage happened on Grex

A /project storage outage happened on Grex this morning! The /project filesystem became unavailable. This affectes running jobs and access to the OpenOnDemand portal as well. We are investigating the issue.

[Resolved] Unplanned power outage in HPCC, Grex down

Wed, 23 Apr 2025 09:10:00 +0000

Power to the HPCC Centre restored

Manitoba Hydro had restored power to Campus, and Grex is back online. All running and queued jobs were lost during the outage. We have used the opportunity to update SLURM scheduler to the current major version 24.11 . All Grex subsystems (compute, storage, login nodes and Web portal) are operational.

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A power outage happened in HPCC Centre

A power outage in Grex’s datacentre happened , with a complete loss of power at about 9:10 AM Winnipeg time. The system is down. The reason for the outage is a problem at Manitoba Hydro, our electricity provider.

https://account.hydro.mb.ca/Portal/outeroutage.aspx

We are waiting for the power to be restored. Thank you for your patience!

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

[Resolved] Planned HPCC datacentre power outage

Sun, 23 Feb 2025 07:00:00 +0000

HPCC planned power outage update

The outage started on Feb 23 is over. Grex is operational. Some of the GPU compute nodes may still be unavailable, and will be in production shortly.

HPCC planned power outage

Physical Plant had informed us that it plans to shut down power feed for transformer work, to several buildings including HPCC where Grex is located. For this reason, we have a complete Grex outage starting from 7 AM, Feb 23, 2025. The planned end of the outage is by the end of the day on Monday, Feb 24. The system is completely be unavailable to the users. Login nodes and data are not be accessible during the outage.

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Mon, 26 Aug 2024 08:00:00 +0000

Update Sept 10

The outage is over. Grex is fully online and available to users. There are many important changes made on the Grex system. Please check them out at:

https://um-grex.github.io/grex-docs/updates/

Your Grex HPC team.

Update Sept 6

Due to a delay with deployment of the new water cooling system, Grex’s outage is extended until Wednesday, Sept. 11. At this point, the cooling for new row of racks cannot be fully enabled. Thus, the partial availability of Grex continues.

SSH to Login nodes (yak.hpc.umanitoba.ca; grex.hpc.umanitoba.ca is now a yak alias)
Home and Project file systems are online.
OpenOnDemand portal (https://zebu.hpc.umanitoba.ca, Simplified Desktop) is online
Running jobs of short duration (must end before September 9, 2024) on skylake and GPU partitions would work.

Thank you for your patience!

Update Aug 30

We have completed the migration of all of the storage systems, and most of the compute servers into the new datacentre racks. However, the cooling system installation and acceptance is due next week, so the Grex system is not yet fully online.

During the long weekend, users have access to the following Grex services or systems:

SSH to Login nodes (yak.hpc.umanitoba.ca; grex.hpc.umanitoba.ca is now a yak alias)
Home and Project file systems are online.
OpenOnDemand portal (https://zebu.hpc.umanitoba.ca, Simplified Desktop) is online
Running jobs of short duration (must end before September 3, 2024) on skylake and some of the GPU partitions would work.

The following systems or services are as of now offline and unavailable: Old login nodes tatanka and bison are decommissioned and unavailable. grex.hpc.umanitoba.ca is now a yak alias. Old compute partition is decommissioned and unavailable. Most new GPU and CPU partitions are offline because the cooling system is yet to be completed in HPCC.

Update as of Aug 28

The First phase: Aug 26 - Aug 28, 2024 is done. We have migrated our storage, login and management nodes to the final location. Grex is now partially open for users with limitted services:

 - Use the login nodes and OOD portal
- Access to storage {home and project} if you need to access your data.

Please note that users can not yet submit jobs as the migration of the compute nodes is not done yet, pending completion of the new cooling systems. We may also experience intermittent interruptions with access to the storage and the login nodes as we are continue with the outage.

Outage started on Aug 26

There is a planned outage on Grex in effect now.

During this outage, Physical Plant will work on HPCC power and cooling, and the entire Grex system will be powered down. Then, the system will be migrated to our new water cooled rack infrastructure.

Users will not have access to any Grex services (compute, storage and the OOD Web portal) during the fist stage of the outage that is expected to last at least three days (until Aug 29).

We will be updating this page as the work in HPCC progresses.

Should you have any questions about the upcoming Grex outage, please do not hesitate to contact us at support@tech.alliancecan.ca ! Thank you for your patience,

Your Grex HPC team.