NFS /Home on Status of the Grex HPC system

[Resolved] Planned Grex outage - updating SLURM and home

Wed, 02 Sep 2026 08:30:00 +0000

Sept 8, outage complete

All the /project migration jobs were completed. Grex is fully available.

Status update as of evening of Sept 4

We have to extend the ongoing outage past the planned Sept.4 ending. The tentative ETA is September 8, 2026.

We have successfully updated Grex’s SLURM scheduler to a current version to address a CVE.
We have also migrated all user /home directories to the new storage server. This should be transparent for all users.
We are still in the process of migration of /project directories to the new /project filesystem appliance.

Due to some issues we have encountered during the storage, our data migration to the new /project filesystem took longer than the expected outage window. Thus we have to extend the Grex outage to September 8. Some functionality like OpenOnDemand and Nextcloud is not available until the end of the outage.

However, we are able to open Grex for SSH access now, to groups whose /project had been migrated, so that they could log in, access their data, run their jobs and thus test the system. Access to Grex is blocked for the groups whose data are still in the process of migration. As of now, about 95% of all projects were migrated. We apologize for the delay.

A planned Grex outage

We will be performing a planned, full outage of the Grex HPC system, starting at 8:30AM on Tuesday, September 2 2026 We expect the outage to last until end of the day on Friday, September 4.

We will perform update of the SLURM controller, and NFS storage, as well as a minor Linux update on all the compute nodes. During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Planned Grex outage - updating storage

Mon, 08 Dec 2025 08:00:00 +0000

Planned /project storage outage resolved

The project outage is resolved.

A planned /project storage outage

We will be performing a planned outage of the Grex HPC system, starting at 8AM on Monday, December 8, 2025. We expect the outage to last one working day.

We will perform hardware maintenance on our Lustre storage controller.

During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Planned Grex outage, updating storage and patching operating system

Wed, 24 Sep 2025 08:00:00 +0000

The outage completed

Grex is operating normally. Queued jobs were not affected and are now running. Thank you for your patience! If you notice any issues or abnormalites, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A planned /project storage and Compute outage

A /project storage outage to update Lustre filesystem software is in progress on Grex! This requires a quiet filesystem so the system is drained from running jobs and access throgh login nodes and OpenOndemand is closed to users. We will also use the outage to apply various security patches to the Linux OS on Grex , as well as performing other software updates. ETA for the outage end is Friday , Sept 26. Thank you for your patience!

[Resolved] Unplanned power outage in HPCC, Grex down

Wed, 23 Apr 2025 09:10:00 +0000

Power to the HPCC Centre restored

Manitoba Hydro had restored power to Campus, and Grex is back online. All running and queued jobs were lost during the outage. We have used the opportunity to update SLURM scheduler to the current major version 24.11 . All Grex subsystems (compute, storage, login nodes and Web portal) are operational.

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A power outage happened in HPCC Centre

A power outage in Grex’s datacentre happened , with a complete loss of power at about 9:10 AM Winnipeg time. The system is down. The reason for the outage is a problem at Manitoba Hydro, our electricity provider.

https://account.hydro.mb.ca/Portal/outeroutage.aspx

We are waiting for the power to be restored. Thank you for your patience!

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

[Resolved] Unplanned storage outage on Grex

Sun, 16 Mar 2025 10:00:00 +0000

The filesystem came back

As of now, it is working. We will investigate the cause and update the notice. If you see any problem with your data on /home , please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

Failure of the NFS /home filesystem

The NFS storage had failed around 10AM Sunday, March 16. The /home filesystem is currently unavailable. Jobs, even those that are running from the /project filesystem, may lack access to the local software stack. Thus we have placed a SLURM reservation to prefent new jobs from starting.

We are working on resolving the issue. Thank you for your patience!

[Resolved] Planned HPCC datacentre power outage

Sun, 23 Feb 2025 07:00:00 +0000

HPCC planned power outage update

The outage started on Feb 23 is over. Grex is operational. Some of the GPU compute nodes may still be unavailable, and will be in production shortly.

HPCC planned power outage

Physical Plant had informed us that it plans to shut down power feed for transformer work, to several buildings including HPCC where Grex is located. For this reason, we have a complete Grex outage starting from 7 AM, Feb 23, 2025. The planned end of the outage is by the end of the day on Monday, Feb 24. The system is completely be unavailable to the users. Login nodes and data are not be accessible during the outage.

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Tue, 16 Jul 2024 06:00:00 +0000

Update 3 on July 18

Cooling in HPCC is working, so the outage is over and Grex is operational.

Update 2 on July 17

Unfortunately, due to heat outside of the datacentre, and the work inside the datacentre, we were unable to keep the environment cool enough to run even the storage and login nodes. So Grex is fully powered down again.

Update 1 on July 17

The electrical part of the update is over. We have powered up Grex login nodes and storage, so the users can SSH in and access their data. What does not yet work is compute, because water cooling is being worked on. So no jobs will get started, and OOD Web portal on zebu is also down.

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca

Outage started on July 16

There is a planned outage on Grex in effect now. During this outage, Physical Plant will work on HPCC power and cooling, and the entire Grex system will be powered down.

Users will not have access to any Grex services (compute and storage and Web portal) during the outage. During the outage, all running jobs will be terminated.

Should you have any questions about the upcoming Grex outage, please do not hesitate to contact us at support@tech.alliancecan.ca ! Thank you for your patience,

Your Grex HPC team.

[Resolved] Planned Grex outage for Lustre FS and Major Linux update

Tue, 07 May 2024 09:00:00 +0000

Final update on May 10

The OS and Lustre update outage is over!

The OS on Grex was upgraded to Alma Linux on all CPU, GPU and login nodes {except for bison, tatanka and compute partition}. Now, Grex is available for running jobs. Please note the changes about software stack:

https://um-grex.github.io/grex-docs/updates/

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca

Update on May 9

The outage extended into May 10. The /home filesystem had been updated; update of the Linux OS and HPC software is still in progress. Sorry about the inconvenience and thank you for your patience!

Update on May 8

The outage continues. The /project filesystem appliance had been updated. We are working on updating the /home filesystem, Linux OS and HPC software stacks. The system is still closed for users.

Grex system outage on May 7 - 9 2024

There is a planned outage on Grex in effect now . We are performing a major update of the DDN Lustre storage controller. We will also do a major Linux OS update from CentOS 7 to AlmaLinux 8. We will reboot and reinstall all of Grex compute and login nodes. All running and queued jobs will be deleted. Grex login nodes , compute and storage will be unavailable during the outage window. Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Planned Grex outage for SLURM and minor OS update

Mon, 18 Dec 2023 09:00:00 +0000

Grex system outage completed on Dec 19 2023

The SLURM scheduler had been updated. Linux OS also had a minor update. The Grex system is now fully operational. If you encounter any problems, please do not hesitate to contact us at support@tech.alliancecan.ca !

Grex system outage on December 18 2023

There is a planned outage on Grex in effect. We are performing a major update of the SLURM scheduler, communication libraries, and minor Linux OS updates for security patching.

We will eboot and reinstall all of Grex compute and login nodes and to migrate the SLURM job database. Thus Jobs that are still running by the time of the outage will be lost, and Grex login nodes will be unavailable during the outage window.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Two Planned Grex outages for HPCC transformer work

Tue, 12 Sep 2023 18:00:00 +0000

Both power outages are now complete

Grex system is open to the users. Queued jobs were not affected and seems to be running now. There were no major upgrades or changes done during the outage.

Planned Grex power outages in September 2023

The Physical Plant is about to perform some electrical works on the transformer that feeds, amongst other things on campus, the HPCC data center that hosts Grex. The outage will start for 6 PM on September 12 and September 19. These outages would require a complete power shutdown in HPCC for about an hour, which means the system would be completely inaccessible to the users, and all running jobs would be terminated.

To avoid the failure of jobs, we have made two reservations to avoid any longer jobs (that cannot be finished before the beginning of the outage) from starting. For more information, run the following command from any login node: “scontrol show res”. To take advantage of the cluster before, and between, the outages, we recommend users submit short jobs that can finish by the time the upcoming outage begins.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Planned power outage Aug 31 to Sept 1

Tue, 31 Aug 2021 08:00:00 +0000

Grex is back online

Grex is back online, accepting CPU jobs. GPU nodes will take a bit more to reinstall NVIDIA updates, but will be online by end of today.

The previous jobs on the queue were lost since the outage afftected all compute nodes that are rebooted after restoring the power. Please check your data and re-submit the jobs that are not done before the outage.

Grex will be down for a campus power maintenance

Physical Plant has a planning a power outage for power breakers maintenance. This will affect Grex building, so and all the nodes and strage systems will be shut-down. The system will be unavailable for users, and all running jobs will be lost. We will be performing some work on Grex during the maintenance window as well. ETA for Grex availabinity is noon, September 1.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.