Network on Status of the Grex HPC system

[Resolved] Grex login failure, login nodes and OOD.

Sun, 28 Sep 2025 06:00:00 +0000

The issue was a temporary lock-up of the /home NFS server. After restarting it, the system operates normally.

Access to Grex login nodes and OpenOnDemand are disrupted. We are investigating.

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca , mentioning Grex in the subject line.

[Resolved] Unplanned power outage in HPCC, Grex down

Wed, 23 Apr 2025 09:10:00 +0000

Power to the HPCC Centre restored

Manitoba Hydro had restored power to Campus, and Grex is back online. All running and queued jobs were lost during the outage. We have used the opportunity to update SLURM scheduler to the current major version 24.11 . All Grex subsystems (compute, storage, login nodes and Web portal) are operational.

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

A power outage happened in HPCC Centre

A power outage in Grex’s datacentre happened , with a complete loss of power at about 9:10 AM Winnipeg time. The system is down. The reason for the outage is a problem at Manitoba Hydro, our electricity provider.

https://account.hydro.mb.ca/Portal/outeroutage.aspx

We are waiting for the power to be restored. Thank you for your patience!

If you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca , mentioning Grex in the subject line.

[Resolved] Planned HPCC datacentre power outage

Sun, 23 Feb 2025 07:00:00 +0000

HPCC planned power outage update

The outage started on Feb 23 is over. Grex is operational. Some of the GPU compute nodes may still be unavailable, and will be in production shortly.

HPCC planned power outage

Physical Plant had informed us that it plans to shut down power feed for transformer work, to several buildings including HPCC where Grex is located. For this reason, we have a complete Grex outage starting from 7 AM, Feb 23, 2025. The planned end of the outage is by the end of the day on Monday, Feb 24. The system is completely be unavailable to the users. Login nodes and data are not be accessible during the outage.

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Mon, 26 Aug 2024 08:00:00 +0000

Update Sept 10

The outage is over. Grex is fully online and available to users. There are many important changes made on the Grex system. Please check them out at:

https://um-grex.github.io/grex-docs/updates/

Your Grex HPC team.

Update Sept 6

Due to a delay with deployment of the new water cooling system, Grex’s outage is extended until Wednesday, Sept. 11. At this point, the cooling for new row of racks cannot be fully enabled. Thus, the partial availability of Grex continues.

SSH to Login nodes (yak.hpc.umanitoba.ca; grex.hpc.umanitoba.ca is now a yak alias)
Home and Project file systems are online.
OpenOnDemand portal (https://zebu.hpc.umanitoba.ca, Simplified Desktop) is online
Running jobs of short duration (must end before September 9, 2024) on skylake and GPU partitions would work.

Thank you for your patience!

Update Aug 30

We have completed the migration of all of the storage systems, and most of the compute servers into the new datacentre racks. However, the cooling system installation and acceptance is due next week, so the Grex system is not yet fully online.

During the long weekend, users have access to the following Grex services or systems:

SSH to Login nodes (yak.hpc.umanitoba.ca; grex.hpc.umanitoba.ca is now a yak alias)
Home and Project file systems are online.
OpenOnDemand portal (https://zebu.hpc.umanitoba.ca, Simplified Desktop) is online
Running jobs of short duration (must end before September 3, 2024) on skylake and some of the GPU partitions would work.

The following systems or services are as of now offline and unavailable: Old login nodes tatanka and bison are decommissioned and unavailable. grex.hpc.umanitoba.ca is now a yak alias. Old compute partition is decommissioned and unavailable. Most new GPU and CPU partitions are offline because the cooling system is yet to be completed in HPCC.

Update as of Aug 28

The First phase: Aug 26 - Aug 28, 2024 is done. We have migrated our storage, login and management nodes to the final location. Grex is now partially open for users with limitted services:

 - Use the login nodes and OOD portal
- Access to storage {home and project} if you need to access your data.

Please note that users can not yet submit jobs as the migration of the compute nodes is not done yet, pending completion of the new cooling systems. We may also experience intermittent interruptions with access to the storage and the login nodes as we are continue with the outage.

Outage started on Aug 26

There is a planned outage on Grex in effect now.

During this outage, Physical Plant will work on HPCC power and cooling, and the entire Grex system will be powered down. Then, the system will be migrated to our new water cooled rack infrastructure.

Users will not have access to any Grex services (compute, storage and the OOD Web portal) during the fist stage of the outage that is expected to last at least three days (until Aug 29).

We will be updating this page as the work in HPCC progresses.

Should you have any questions about the upcoming Grex outage, please do not hesitate to contact us at support@tech.alliancecan.ca ! Thank you for your patience,

Your Grex HPC team.

[Resolved] Planned Grex outage for SLURM and minor OS update

Mon, 18 Dec 2023 09:00:00 +0000

Grex system outage completed on Dec 19 2023

The SLURM scheduler had been updated. Linux OS also had a minor update. The Grex system is now fully operational. If you encounter any problems, please do not hesitate to contact us at support@tech.alliancecan.ca !

Grex system outage on December 18 2023

There is a planned outage on Grex in effect. We are performing a major update of the SLURM scheduler, communication libraries, and minor Linux OS updates for security patching.

We will eboot and reinstall all of Grex compute and login nodes and to migrate the SLURM job database. Thus Jobs that are still running by the time of the outage will be lost, and Grex login nodes will be unavailable during the outage window.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Two Planned Grex outages for HPCC transformer work

Tue, 12 Sep 2023 18:00:00 +0000

Both power outages are now complete

Grex system is open to the users. Queued jobs were not affected and seems to be running now. There were no major upgrades or changes done during the outage.

Planned Grex power outages in September 2023

The Physical Plant is about to perform some electrical works on the transformer that feeds, amongst other things on campus, the HPCC data center that hosts Grex. The outage will start for 6 PM on September 12 and September 19. These outages would require a complete power shutdown in HPCC for about an hour, which means the system would be completely inaccessible to the users, and all running jobs would be terminated.

To avoid the failure of jobs, we have made two reservations to avoid any longer jobs (that cannot be finished before the beginning of the outage) from starting. For more information, run the following command from any login node: “scontrol show res”. To take advantage of the cluster before, and between, the outages, we recommend users submit short jobs that can finish by the time the upcoming outage begins.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Electric failure, loss of power to management rack

Thu, 26 Jan 2023 14:50:00 +0000

Update: datacentre change rolled back, systems operational

It appears that some of the running jobs continued running , and login nodes’ access nor storage systems were affected. Grex is now operational. In case you notice any ongoing issue, please let us know by email to support@tech.alliancecan.ca, Subject line containing Grex.

Faulty electrical work, loss of power to management rack

We have an unplanned outage on Grex due to a failed electrical work that affected its management rack at around 2:50 PM, Jan 26, 2023 .

Running jobs may be lost, and access to Grex login nodes may be degraded or unavailable.

We are working on resolving the issue. Sorry about the inconvenience it may have caused.

[Resolved] Planned Network outage of the login nodes

Fri, 09 Sep 2022 11:45:00 +0000

Update: the switch complete!

Grex’s internet connection to all login nodes is now migrated to the UManitoba network IP space. New login node names:

grex.hpc.umanitoba.ca (instead of grex.westgrid.ca; same for the ones below) bison.hpc.umanitoba.ca tatanka.hpc.umanitoba.ca yak.hpc.umanitoba.ca aurochs.hpc.umanitoba.ca

SWITCHING GREX NETWORK FROM BCNET TO UMANITOBA NETWORK. <==

On Friday, Sep 09 between 12:00 and 2:00 PM, we will switch the network from BCNET {westgrid.ca} to UManitoba network. During this process, access to Grex via old DNS names {grex, bison, tatanka} will be disrupted. We expect less disruption for yak.hpc.umanitoba.ca

The process will not affect users’ data.

OpenOnDemand interface will also be disabled and hopefully the service will be restored sometime after the outage when new SSL certificates are in place.

In the meantime, you can use yak to connect to Grex:

ssh your-user-name@yak.hpc.umanitoba.ca

Replace by your Alliance (Compute Canada) user name.

Please note that this node has avx512 architecture and if you have to use it for compiling your codes, they may not run on “compute” partition. Other than that, it should behave as any other old login node.

[Resolved] CANARIE outage affects Grex external network and Legacy login nodes

Tue, 16 Aug 2022 09:00:00 +0000

Update: connectivity restored

All systems should be functioning normally, as the connection is restored.

a planned CANARIE outage disables Grex legacy network to BCNet

Thus, network connection to Grex login nodes in .westgrid.ca domain login nodes (bison and tatanka, grex.westgrid.ca ,aurochs.login.ca) is not available at the moment.

A workaround is to use to the new login node yak.hpc.umanitoba.ca for logging in and transferring files.

Most running jobs are unaffected; however, commercial licenses for software like MATLAB and ANSYS are also not reacheable due to the network unavailability.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Grex has a problem with external network and login nodes

Thu, 23 Jun 2022 19:00:00 +0000

Update Jun 24, 10AM

Finally, all Ethernet switches are powered up, and all Grex login nodes are available. Running jobs and storage were not affected during the outage. Grex should be fully operational now.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

Update Jun 24, 8AM

The reason for this partial outage is a faulty UPS that fed some of the Grex network switches. As of now, the power to most of the switches is re-routed, so jobs run normally, but only yak.hpc.umanitoba.ca works for users to connect to.

Legacy login nodes of grex.westgrid.ca are on, but external network to them is still unavailable. Please use Yak to connect for now.

We are investigating the issue. Access to Grex is not possible, but running jobs and storage seems to be largely unaffected.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.

[Resolved] Westgrid network failure

Wed, 09 Jun 2021 11:35:00 +0000

this is an example of notification. The problem was there and was resolved, the dates are arbitrary

Update : The issue with network / packet loss issue is resolved (11:35 UTC — Jul 28) .

Problem:

Login nodes of Grex are experiencing a heavy packet loss. This leads to connection slowness and intermittent failures. Sorry about the inconvenience. We are working on resloving the issue (11:35 UTC — Jun 9) .

Network on Status of the Grex HPC system

[Resolved] Grex login failure, login nodes and OOD.

Login failure resolved

Login failure

[Resolved] Unplanned power outage in HPCC, Grex down

Power to the HPCC Centre restored

A power outage happened in HPCC Centre

[Resolved] Planned HPCC datacentre power outage

HPCC planned power outage update

HPCC planned power outage

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Update Sept 10

Update Sept 6

Update Aug 30

Update as of Aug 28

Outage started on Aug 26

[Resolved] Planned Grex outage for SLURM and minor OS update

Grex system outage completed on Dec 19 2023

Grex system outage on December 18 2023

[Resolved] Two Planned Grex outages for HPCC transformer work

Both power outages are now complete

Planned Grex power outages in September 2023

[Resolved] Electric failure, loss of power to management rack

Update: datacentre change rolled back, systems operational

Faulty electrical work, loss of power to management rack

[Resolved] Planned Network outage of the login nodes

Update: the switch complete!

SWITCHING GREX NETWORK FROM BCNET TO UMANITOBA NETWORK. <==

[Resolved] CANARIE outage affects Grex external network and Legacy login nodes

Update: connectivity restored

a planned CANARIE outage disables Grex legacy network to BCNet

[Resolved] Grex has a problem with external network and login nodes

Update Jun 24, 10AM

Update Jun 24, 8AM

Grex network, management VMs and login nodes are down

[Resolved] Westgrid network failure

this is an example of notification. The problem was there and was resolved, the dates are arbitrary