Issues on Status of the Grex HPC system

[Resolved] Planned Grex outage - updating SLURM and home

Wed, 02 Sep 2026 08:30:00 +0000

Sept 8, outage complete

All the /project migration jobs were completed. Grex is fully available.

Status update as of evening of Sept 4

We have to extend the ongoing outage past the planned Sept.4 ending. The tentative ETA is September 8, 2026.

We have successfully updated Grex’s SLURM scheduler to a current version to address a CVE.
We have also migrated all user /home directories to the new storage server. This should be transparent for all users.
We are still in the process of migration of /project directories to the new /project filesystem appliance.

Due to some issues we have encountered during the storage, our data migration to the new /project filesystem took longer than the expected outage window. Thus we have to extend the Grex outage to September 8. Some functionality like OpenOnDemand and Nextcloud is not available until the end of the outage.

However, we are able to open Grex for SSH access now, to groups whose /project had been migrated, so that they could log in, access their data, run their jobs and thus test the system. Access to Grex is blocked for the groups whose data are still in the process of migration. As of now, about 95% of all projects were migrated. We apologize for the delay.

A planned Grex outage

We will be performing a planned, full outage of the Grex HPC system, starting at 8:30AM on Tuesday, September 2 2026 We expect the outage to last until end of the day on Friday, September 4.

We will perform update of the SLURM controller, and NFS storage, as well as a minor Linux update on all the compute nodes. During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Unplanned Grex outage - restarting login nodes and OOD

Thu, 07 May 2026 15:00:01 +0000

We are performing a restart of login nodes of Grex, and OOD web portal. Thus access to the system will be temporarily unavailable, for about an hour. Running jobs and data/storage will not be affected. Thank you for your patience!

[Resolved] Unplanned Grex outage - restarting login nodes and OOD

Wed, 29 Apr 2026 17:00:00 +0000

Yak and OOD are availanle to users. The alternate login node, bison, is being worked on.

We are performing a restart of login nodes of Grex, and OOD web portal. Thus access to the system will be temporarily unavailable, for about an hour (5:10 PM to 6:00 PM). Running jobs and data/storage will not be affected. Thank you for your patience!

[Resolved] Planned Grex outage - updating storage

Mon, 08 Dec 2025 08:00:00 +0000

Planned /project storage outage resolved

The project outage is resolved.

A planned /project storage outage

We will be performing a planned outage of the Grex HPC system, starting at 8AM on Monday, December 8, 2025. We expect the outage to last one working day.

We will perform hardware maintenance on our Lustre storage controller.

During the outage, OpenOnDemand, Storage, and Login nodes will not be available, and all running jobs will be stopped. The outage will not affect the users’ data.

[Resolved] Grex login failure, login nodes and OOD.

Sun, 28 Sep 2025 06:00:00 +0000

The issue was a temporary lock-up of the /home NFS server. After restarting it, the system operates normally.

Access to Grex login nodes and OpenOnDemand are disrupted. We are investigating.

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca , mentioning Grex in the subject line.

[Resolved] Planned Grex outage, updating storage and patching operating system

Wed, 24 Sep 2025 08:00:00 +0000

The outage completed

Grex is operating normally. Queued jobs were not affected and are now running. Thank you for your patience! If you notice any issues or abnormalites, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A planned /project storage and Compute outage

A /project storage outage to update Lustre filesystem software is in progress on Grex! This requires a quiet filesystem so the system is drained from running jobs and access throgh login nodes and OpenOndemand is closed to users. We will also use the outage to apply various security patches to the Linux OS on Grex , as well as performing other software updates. ETA for the outage end is Friday , Sept 26. Thank you for your patience!

[Resolved] Unplanned project storage outage in HPCC

Fri, 18 Jul 2025 07:10:00 +0000

Update 17:30 Central Time: System is available

The storage is working. If you notice any issues with the storage, please do not hesitate to contact us at support@tech.alliancecan.ca, mentioning Grex in the subject line.

Update 17:00 Central Time: System is available with conditions

The failure was traced to a hardware issue on one of the storage controllers. We have cleared the controller state and restarted the /project storage appliance. However, the storage targets are not yet properly balanced across the storage servers, so there can be some performance issues. We are working with the storage vendor to address these issues.

As of now, OpenOnDemand and /project are now available, and Grex system would accept new jobs.

A /project storage outage happened on Grex

A /project storage outage happened on Grex this morning! The /project filesystem became unavailable. This affectes running jobs and access to the OpenOnDemand portal as well. We are investigating the issue.

[Resolved] Unplanned power outage in HPCC, Grex down

Wed, 23 Apr 2025 09:10:00 +0000

Power to the HPCC Centre restored

Manitoba Hydro had restored power to Campus, and Grex is back online. All running and queued jobs were lost during the outage. We have used the opportunity to update SLURM scheduler to the current major version 24.11 . All Grex subsystems (compute, storage, login nodes and Web portal) are operational.

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A power outage happened in HPCC Centre

A power outage in Grex’s datacentre happened , with a complete loss of power at about 9:10 AM Winnipeg time. The system is down. The reason for the outage is a problem at Manitoba Hydro, our electricity provider.

https://account.hydro.mb.ca/Portal/outeroutage.aspx

We are waiting for the power to be restored. Thank you for your patience!

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

[Resolved] Unplanned storage outage on Grex

Sun, 16 Mar 2025 10:00:00 +0000

The filesystem came back

As of now, it is working. We will investigate the cause and update the notice. If you see any problem with your data on /home , please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

Failure of the NFS /home filesystem

The NFS storage had failed around 10AM Sunday, March 16. The /home filesystem is currently unavailable. Jobs, even those that are running from the /project filesystem, may lack access to the local software stack. Thus we have placed a SLURM reservation to prefent new jobs from starting.

We are working on resolving the issue. Thank you for your patience!

[Resolved] Planned HPCC datacentre power outage

Sun, 23 Feb 2025 07:00:00 +0000

HPCC planned power outage update

The outage started on Feb 23 is over. Grex is operational. Some of the GPU compute nodes may still be unavailable, and will be in production shortly.

HPCC planned power outage

Physical Plant had informed us that it plans to shut down power feed for transformer work, to several buildings including HPCC where Grex is located. For this reason, we have a complete Grex outage starting from 7 AM, Feb 23, 2025. The planned end of the outage is by the end of the day on Monday, Feb 24. The system is completely be unavailable to the users. Login nodes and data are not be accessible during the outage.

[Resolved] Unplanned SLURM outage due to a problematic update

Tue, 21 Jan 2025 11:00:00 +0000

SLURM scheduler update done

We have finished updating SLURM scheduler on Grex. Running jobs should not have been affected, but couple of new jobs have failed to start and need to be resubmitted. Sorry about the inconvenience!

In case you experience problems with the updated schedler , please do not hesitate to contact support@tech.alliancecan.ca, mentioning Grex in the subject line.

SLURM scheduler outage

Due to a glitch during rolling SLURM scheduler update, new compute jobs are failing to start on Grex. We are working on fixing the issue. Thank you for your patience!

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Mon, 26 Aug 2024 08:00:00 +0000

Update Sept 10

The outage is over. Grex is fully online and available to users. There are many important changes made on the Grex system. Please check them out at:

https://um-grex.github.io/grex-docs/updates/

Your Grex HPC team.

Update Sept 6

Due to a delay with deployment of the new water cooling system, Grex’s outage is extended until Wednesday, Sept. 11. At this point, the cooling for new row of racks cannot be fully enabled. Thus, the partial availability of Grex continues.

SSH to Login nodes (yak.hpc.umanitoba.ca; grex.hpc.umanitoba.ca is now a yak alias)
Home and Project file systems are online.
OpenOnDemand portal (https://zebu.hpc.umanitoba.ca, Simplified Desktop) is online
Running jobs of short duration (must end before September 9, 2024) on skylake and GPU partitions would work.

Thank you for your patience!

Update Aug 30

We have completed the migration of all of the storage systems, and most of the compute servers into the new datacentre racks. However, the cooling system installation and acceptance is due next week, so the Grex system is not yet fully online.

During the long weekend, users have access to the following Grex services or systems:

SSH to Login nodes (yak.hpc.umanitoba.ca; grex.hpc.umanitoba.ca is now a yak alias)
Home and Project file systems are online.
OpenOnDemand portal (https://zebu.hpc.umanitoba.ca, Simplified Desktop) is online
Running jobs of short duration (must end before September 3, 2024) on skylake and some of the GPU partitions would work.

The following systems or services are as of now offline and unavailable: Old login nodes tatanka and bison are decommissioned and unavailable. grex.hpc.umanitoba.ca is now a yak alias. Old compute partition is decommissioned and unavailable. Most new GPU and CPU partitions are offline because the cooling system is yet to be completed in HPCC.

Update as of Aug 28

The First phase: Aug 26 - Aug 28, 2024 is done. We have migrated our storage, login and management nodes to the final location. Grex is now partially open for users with limitted services:

 - Use the login nodes and OOD portal
- Access to storage {home and project} if you need to access your data.

Please note that users can not yet submit jobs as the migration of the compute nodes is not done yet, pending completion of the new cooling systems. We may also experience intermittent interruptions with access to the storage and the login nodes as we are continue with the outage.

Outage started on Aug 26

There is a planned outage on Grex in effect now.

During this outage, Physical Plant will work on HPCC power and cooling, and the entire Grex system will be powered down. Then, the system will be migrated to our new water cooled rack infrastructure.

Users will not have access to any Grex services (compute, storage and the OOD Web portal) during the fist stage of the outage that is expected to last at least three days (until Aug 29).

We will be updating this page as the work in HPCC progresses.

Should you have any questions about the upcoming Grex outage, please do not hesitate to contact us at support@tech.alliancecan.ca ! Thank you for your patience,

Your Grex HPC team.

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Tue, 16 Jul 2024 06:00:00 +0000

Update 3 on July 18

Cooling in HPCC is working, so the outage is over and Grex is operational.

Update 2 on July 17

Unfortunately, due to heat outside of the datacentre, and the work inside the datacentre, we were unable to keep the environment cool enough to run even the storage and login nodes. So Grex is fully powered down again.

Update 1 on July 17

The electrical part of the update is over. We have powered up Grex login nodes and storage, so the users can SSH in and access their data. What does not yet work is compute, because water cooling is being worked on. So no jobs will get started, and OOD Web portal on zebu is also down.

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca

Outage started on July 16

There is a planned outage on Grex in effect now. During this outage, Physical Plant will work on HPCC power and cooling, and the entire Grex system will be powered down.

Users will not have access to any Grex services (compute and storage and Web portal) during the outage. During the outage, all running jobs will be terminated.

Should you have any questions about the upcoming Grex outage, please do not hesitate to contact us at support@tech.alliancecan.ca ! Thank you for your patience,

Your Grex HPC team.

[Resolved] Planned Grex outage for Lustre FS and Major Linux update

Tue, 07 May 2024 09:00:00 +0000

Final update on May 10

The OS and Lustre update outage is over!

The OS on Grex was upgraded to Alma Linux on all CPU, GPU and login nodes {except for bison, tatanka and compute partition}. Now, Grex is available for running jobs. Please note the changes about software stack:

https://um-grex.github.io/grex-docs/updates/

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca

Update on May 9

The outage extended into May 10. The /home filesystem had been updated; update of the Linux OS and HPC software is still in progress. Sorry about the inconvenience and thank you for your patience!

Update on May 8

The outage continues. The /project filesystem appliance had been updated. We are working on updating the /home filesystem, Linux OS and HPC software stacks. The system is still closed for users.

Grex system outage on May 7 - 9 2024

There is a planned outage on Grex in effect now . We are performing a major update of the DDN Lustre storage controller. We will also do a major Linux OS update from CentOS 7 to AlmaLinux 8. We will reboot and reinstall all of Grex compute and login nodes. All running and queued jobs will be deleted. Grex login nodes , compute and storage will be unavailable during the outage window. Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Planned Grex outage for SLURM and minor OS update

Mon, 18 Dec 2023 09:00:00 +0000

Grex system outage completed on Dec 19 2023

The SLURM scheduler had been updated. Linux OS also had a minor update. The Grex system is now fully operational. If you encounter any problems, please do not hesitate to contact us at support@tech.alliancecan.ca !

Grex system outage on December 18 2023

There is a planned outage on Grex in effect. We are performing a major update of the SLURM scheduler, communication libraries, and minor Linux OS updates for security patching.

We will eboot and reinstall all of Grex compute and login nodes and to migrate the SLURM job database. Thus Jobs that are still running by the time of the outage will be lost, and Grex login nodes will be unavailable during the outage window.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Two Planned Grex outages for HPCC transformer work

Tue, 12 Sep 2023 18:00:00 +0000

Both power outages are now complete

Grex system is open to the users. Queued jobs were not affected and seems to be running now. There were no major upgrades or changes done during the outage.

Planned Grex power outages in September 2023

The Physical Plant is about to perform some electrical works on the transformer that feeds, amongst other things on campus, the HPCC data center that hosts Grex. The outage will start for 6 PM on September 12 and September 19. These outages would require a complete power shutdown in HPCC for about an hour, which means the system would be completely inaccessible to the users, and all running jobs would be terminated.

To avoid the failure of jobs, we have made two reservations to avoid any longer jobs (that cannot be finished before the beginning of the outage) from starting. For more information, run the following command from any login node: “scontrol show res”. To take advantage of the cluster before, and between, the outages, we recommend users submit short jobs that can finish by the time the upcoming outage begins.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Unplanned power outage in HPCC datacentre

Tue, 30 May 2023 07:00:00 +0000

UPDATE: outage concluded

Grex compute and storage are back up after 1 1/2 hour of downtime. Please restart your jobs! If you notice any further malfuncions, please do not hesitate to contact us at the support@tech.alliancecan.ca , mentioning “Grex” in the subject line.

Unplanned power loss at HPCC datacentre

NOTICE: On May 30. 7AM to 8:14AM there was an unplanned power outage in HPCC that rebooted all the Grex compute nodes, storage, and management systems. Running jobs were lost. We apologize for the inconvenience. We work on restarting and testing of the system.

[Resolved] Planned Grex outage for HPCC work and storage update

Wed, 22 Feb 2023 14:30:00 +0000

UPDATE: outage concluded

During the outage of Feb 22, few changes have been made on Grex: We reinstalled all login and compute nodes with a new image to fix the errors with UCX. We have added a new storage “project” with similar structure as for Compute Canada. All the data from scratch has been moved to the project file system.

For an overview of the changes, please have a look to the documentation page:

https://um-grex.github.io/grex-docs/docs/lustre/

Planned Grex outage

The Grex storage outage is about to start. Please save your interactive work! ETA for the outage’s end is Feb 23 or Friday, Feb 24, 2023.

During the outage we are planning to reboot and reinstall all of Grex compute and login nodes, connect the new Project storage, and perform some re-cabling of the Infiniband fabric of the cluster. Jobs that are still running by the time of the outage will be lost, and Grex login nodes will be unavailable during the outage window.

Thank you for your patience! If you have questions or concerns regarding the outage, please do not hesitate to contact us at mailto:support@tech.alliancecan.ca !

[Resolved] Electric failure, loss of power to management rack

Thu, 26 Jan 2023 14:50:00 +0000

Update: datacentre change rolled back, systems operational

It appears that some of the running jobs continued running , and login nodes’ access nor storage systems were affected. Grex is now operational. In case you notice any ongoing issue, please let us know by email to support@tech.alliancecan.ca, Subject line containing Grex.

Faulty electrical work, loss of power to management rack

We have an unplanned outage on Grex due to a failed electrical work that affected its management rack at around 2:50 PM, Jan 26, 2023 .

Running jobs may be lost, and access to Grex login nodes may be degraded or unavailable.

We are working on resolving the issue. Sorry about the inconvenience it may have caused.

Upcoming cooling outage in HPCC

Thu, 01 Dec 2022 11:30:00 +0000

UPDATED NOTICE about outage on Dex 7-8 2022

(updated on Dec 1 at 3:00 PM):

Unplanned outage: Dec 7/22 at 5:30pm until Dec 8/22 at 5:00am

Physical Plant scheduled with a short notice a maintenance of water that may or may not affect Grex. The last update (as of Dec 1) from Physical Plant is that Grex may not be affected. If needed, Grex will be shut-down on Dec 7 around 4:00 PM. Otherwise, Grex will remain online and running as usual. We did not put any reservation and users can submit their jobs.

We will update the status of the maintenance as we get more information.

NOTICE: MAINTENANCE OF WATER LINES AFECTING GREX - Dec 7-8 2022

Physical Plant scheduled with a short notice a maintenance of water that will affect Grex. Therefore, **Grex will be shut-down on Dec 7 around 4:00 PM ** We will bring it back online next day around noon.

All running jobs will be terminated. A reservation will be set to prevent longer jobs from starting if they cannot finish before the outage. These jobs will show up a message “Nodes required for job are DOWN, DRAINED or RESERVED for jobs in higher priority partitions” or “ReqNodeNotAvailable”

To take advantage of the cluster for the period before the outage, please submit short jobs that should finish before the beginning of the outage. For the jobs that are pending, they should start as soon as the cluster is back online on Thursday morning.

Start Time: 3:00 PM (Winnipeg Time), Wednesday, December 7, 2022 Anticipated End Time: 1:00 PM (Winnipeg Time), Thursday, December 8, 2022

Sorry for the short notice.

[Resolved] Datacentre Plumbing works October 20, 2022

Thu, 20 Oct 2022 09:30:00 +0000

Update: work is done , no issues

Grex is fully operational. Some of the compute nodes (largemem and skylake) were restarted.

Planned HPCC work shuts down some of the compute nodes

Due to planned Datacentre work we have a partial Grex outage. Some of the racks with compute nodes were shut down, and jobs running on them are terminated.

Upcoming Datacentre Work on Oct 13.

Thu, 13 Oct 2022 08:30:00 +0000

Update: The outage was rescheduled to October 20, 2022

The outage planned to happen on Oct 13, was rescheduled to October 20.

IMPORTANT NOTICE: MAINTENANCE OF POWER AND COOLING SYSTEMS - October 13, 2022

Starting Thursday, October 13th, 2022, at 9:00 AM (Winnipeg Time), the Grex cluster will be unavailable to all users as we perform cluster maintenance. All running jobs will be terminated. A reservation will be set to prevent longer from starting if they can not finish before the outage.

The maintenance should be completed by the end of the day around 4:00 PM.

To take advantage of the cluster for the period before the outage, please submit short jobs that should finish before the beginning of the outage.

Start Time : 9 AM (Winnipeg Time), Thursday, October 13, 2022 Anticipated End Time : 4 PM (Winnipeg Time), Thursday, October 13, 2022

[Resolved] Planned Network outage of the login nodes

Fri, 09 Sep 2022 11:45:00 +0000

Update: the switch complete!

Grex’s internet connection to all login nodes is now migrated to the UManitoba network IP space. New login node names:

grex.hpc.umanitoba.ca (instead of grex.westgrid.ca; same for the ones below) bison.hpc.umanitoba.ca tatanka.hpc.umanitoba.ca yak.hpc.umanitoba.ca aurochs.hpc.umanitoba.ca

SWITCHING GREX NETWORK FROM BCNET TO UMANITOBA NETWORK. <==

On Friday, Sep 09 between 12:00 and 2:00 PM, we will switch the network from BCNET {westgrid.ca} to UManitoba network. During this process, access to Grex via old DNS names {grex, bison, tatanka} will be disrupted. We expect less disruption for yak.hpc.umanitoba.ca

The process will not affect users’ data.

OpenOnDemand interface will also be disabled and hopefully the service will be restored sometime after the outage when new SSL certificates are in place.

In the meantime, you can use yak to connect to Grex:

ssh your-user-name@yak.hpc.umanitoba.ca

Replace by your Alliance (Compute Canada) user name.

Please note that this node has avx512 architecture and if you have to use it for compiling your codes, they may not run on “compute” partition. Other than that, it should behave as any other old login node.

[Resolved] Hardware failure on one of the login nodes

Fri, 02 Sep 2022 08:30:00 +0000

Update: Bison is running again

a hardware failure on Bison

bison.westgrid.ca had a disk controller failure and is down at the moment. Other login nodes are not affected, except the “grex.westgrid.ca” which is an alias name ttah might lead to bison. A workaround is to use to the new login node yak.hpc.umanitoba.ca or tatanka.westgrid.ca for logging in and transferring files.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] CANARIE outage affects Grex external network and Legacy login nodes

Tue, 16 Aug 2022 09:00:00 +0000

Update: connectivity restored

All systems should be functioning normally, as the connection is restored.

a planned CANARIE outage disables Grex legacy network to BCNet

Thus, network connection to Grex login nodes in .westgrid.ca domain login nodes (bison and tatanka, grex.westgrid.ca ,aurochs.login.ca) is not available at the moment.

A workaround is to use to the new login node yak.hpc.umanitoba.ca for logging in and transferring files.

Most running jobs are unaffected; however, commercial licenses for software like MATLAB and ANSYS are also not reacheable due to the network unavailability.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Grex has a problem with external network and login nodes

Thu, 23 Jun 2022 19:00:00 +0000

Update Jun 24, 10AM

Finally, all Ethernet switches are powered up, and all Grex login nodes are available. Running jobs and storage were not affected during the outage. Grex should be fully operational now.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

Update Jun 24, 8AM

The reason for this partial outage is a faulty UPS that fed some of the Grex network switches. As of now, the power to most of the switches is re-routed, so jobs run normally, but only yak.hpc.umanitoba.ca works for users to connect to.

Legacy login nodes of grex.westgrid.ca are on, but external network to them is still unavailable. Please use Yak to connect for now.

We are investigating the issue. Access to Grex is not possible, but running jobs and storage seems to be largely unaffected.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Brief power outage on May 31, 2022

Tue, 31 May 2022 01:30:00 +0000

A brownout rebooted 130 compute nodes

There was a brief power interruptions in Grex datacentre. Many legacy of compute nodes got rebooted. Most running jobs were lost. Overall, the Grex keept working, network, login nodes, storage and managing VMs seems to have been unaffected.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] One of the /global/scratch filesystem servers crashed, FS runs in degraded mode

Fri, 13 May 2022 19:30:00 +0000

Lustre filesystem problems

One of the /global/scratch filesystem servers crashed, FS runs in degraded mode. We are investigating the issue. Most running jobs will continiue to run, albeit slower, and may stuck in I/O operations while Lustre servers are unresponsive.

Update:

After restarting of Lustre servers, the /global/scratch filesystem is back to normal operation.

[Resolved] Emergency SLURM update wiped all running jobs

Mon, 09 May 2022 00:00:00 +0000

SLURM security update

SLURM scheduler’s authors announced a severe security vulnerability, and dropped old SLURM versions from support at the same time. This forced us to upgrade SLURM version 19 we used to run, to the supported version 21, immediately. Unfortunately, the SLURM state got corrupted during the update, and all running jobs were lost. We apologize for the inconvenience.

The system is expected to run normally with the new SLURM. If you notice any anomalies, please contact us at support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Brief power outages on April 23, 24 , 2022

Sun, 24 Apr 2022 11:30:00 +0000

A series of brownouts

There were two brief power interruptions in Grex datacentre, one on Saturday, APril 24 and another on Sunday, April 24. Most of compute nodes got rebooted. Most running jobs were lost. Overall, the Grex keept working, network, login nodes, storage and managing VMs seems to have been unaffected.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Brief power outages on October 23, 2021

Sat, 23 Oct 2021 15:30:00 +0000

A series of brownouts

There were at least two brief power interruptions in Grex datacentre. Some, but not all of the compute and login nodes rebooted as a result. Some, but not all, running jobs were lost. Overall, the Grex keept working, network, storage and managing VMs seems to have been unaffected.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Planned power outage Aug 31 to Sept 1

Tue, 31 Aug 2021 08:00:00 +0000

Grex is back online

Grex is back online, accepting CPU jobs. GPU nodes will take a bit more to reinstall NVIDIA updates, but will be online by end of today.

The previous jobs on the queue were lost since the outage afftected all compute nodes that are rebooted after restoring the power. Please check your data and re-submit the jobs that are not done before the outage.

Grex will be down for a campus power maintenance

Physical Plant has a planning a power outage for power breakers maintenance. This will affect Grex building, so and all the nodes and strage systems will be shut-down. The system will be unavailable for users, and all running jobs will be lost. We will be performing some work on Grex during the maintenance window as well. ETA for Grex availabinity is noon, September 1.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

Grex update outage, New Status webpage

Wed, 28 Jul 2021 18:05:00 +0000

NOTICE: Grex is online for production.

After the last outage, Grex was in production in a test mode for a week since June 9, 2021. We did not get any reports about the new hardware and software. Everything worked as expected. The /home was already migrated to a new, NVME-based server.

Now, all users are encouraged to submit their jobs as usual. To see the time limit and other characteristics of all partitions, you can use our custom script “partition-list”

To submit a job, you need to specify which partition to use by adding to your job script “#SBATCH –partition=” where partition-name is one of the following: compute, skylake, largemem . Note that bigmem partition was eliminated (merged into compute).

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

New status website

Grex has now a new Status website! Visit https://grex-status.netlify.app !

[Resolved] Rebooting the nodes

Tue, 27 Jul 2021 10:00:00 +0000

Grex nodes will be rebooted and reinstalled to apply a security patch

We started draining the cluster in order to reboot all login and compute nodes to apply security patches. The current running jobs will continue and once finished, the nodes will be rebooted. The pending jobs will wait on the queue till the nodes are rebooted and will start on patched nodes. The process will not affect the data. Only the available resources are limited.

The operation may take a week or so.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Westgrid network failure

Wed, 09 Jun 2021 11:35:00 +0000

this is an example of notification. The problem was there and was resolved, the dates are arbitrary

Update : The issue with network / packet loss issue is resolved (11:35 UTC — Jul 28) .

Problem:

Login nodes of Grex are experiencing a heavy packet loss. This leads to connection slowness and intermittent failures. Sorry about the inconvenience. We are working on resloving the issue (11:35 UTC — Jun 9) .

Issues on Status of the Grex HPC system

[Resolved] Planned Grex outage - updating SLURM and home

Sept 8, outage complete

Status update as of evening of Sept 4

A planned Grex outage

[Resolved] Unplanned Grex outage - restarting login nodes and OOD

Unplanned reboot of login nodes and OOD server

[Resolved] Unplanned Grex outage - restarting login nodes and OOD

Unplanned reboot of login nodes and OOD server

Unplanned reboot of login nodes and OOD server

[Resolved] Planned Grex outage - updating storage

Planned /project storage outage resolved

A planned /project storage outage

[Resolved] Grex login failure, login nodes and OOD.

Login failure resolved

Login failure

[Resolved] Planned Grex outage, updating storage and patching operating system

The outage completed

A planned /project storage and Compute outage

[Resolved] Unplanned project storage outage in HPCC

Update 17:30 Central Time: System is available

Update 17:00 Central Time: System is available with conditions

A /project storage outage happened on Grex

[Resolved] Unplanned power outage in HPCC, Grex down

Power to the HPCC Centre restored

A power outage happened in HPCC Centre

[Resolved] Unplanned storage outage on Grex

The filesystem came back

Failure of the NFS /home filesystem

[Resolved] Planned HPCC datacentre power outage

HPCC planned power outage update

HPCC planned power outage

[Resolved] Unplanned SLURM outage due to a problematic update

SLURM scheduler update done

SLURM scheduler outage

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Update Sept 10

Update Sept 6

Update Aug 30

Update as of Aug 28

Outage started on Aug 26

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Update 3 on July 18

Update 2 on July 17

Update 1 on July 17

Outage started on July 16

[Resolved] Planned Grex outage for Lustre FS and Major Linux update

Final update on May 10

Update on May 9

Update on May 8

Grex system outage on May 7 - 9 2024

[Resolved] Planned Grex outage for SLURM and minor OS update

Grex system outage completed on Dec 19 2023

Grex system outage on December 18 2023

[Resolved] Two Planned Grex outages for HPCC transformer work

Both power outages are now complete

Planned Grex power outages in September 2023

[Resolved] Unplanned power outage in HPCC datacentre

UPDATE: outage concluded

Unplanned power loss at HPCC datacentre

[Resolved] Planned Grex outage for HPCC work and storage update

UPDATE: outage concluded

Planned Grex outage

[Resolved] Electric failure, loss of power to management rack

Update: datacentre change rolled back, systems operational

Faulty electrical work, loss of power to management rack

Upcoming cooling outage in HPCC

UPDATED NOTICE about outage on Dex 7-8 2022

NOTICE: MAINTENANCE OF WATER LINES AFECTING GREX - Dec 7-8 2022

[Resolved] Datacentre Plumbing works October 20, 2022

Update: work is done , no issues

Planned HPCC work shuts down some of the compute nodes

Upcoming Datacentre Work on Oct 13.

Update: The outage was rescheduled to October 20, 2022

IMPORTANT NOTICE: MAINTENANCE OF POWER AND COOLING SYSTEMS - October 13, 2022

[Resolved] Planned Network outage of the login nodes

Update: the switch complete!

SWITCHING GREX NETWORK FROM BCNET TO UMANITOBA NETWORK. <==

[Resolved] Hardware failure on one of the login nodes

Update: Bison is running again