Lustre /Global/Scratch on Status of the Grex HPC system

[Resolved] Planned HPCC/Grex outage for electrical and cooling work.

Tue, 16 Jul 2024 06:00:00 +0000

Update 3 on July 18

Cooling in HPCC is working, so the outage is over and Grex is operational.

Update 2 on July 17

Unfortunately, due to heat outside of the datacentre, and the work inside the datacentre, we were unable to keep the environment cool enough to run even the storage and login nodes. So Grex is fully powered down again.

Update 1 on July 17

The electrical part of the update is over. We have powered up Grex login nodes and storage, so the users can SSH in and access their data. What does not yet work is compute, because water cooling is being worked on. So no jobs will get started, and OOD Web portal on zebu is also down.

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca

Outage started on July 16

There is a planned outage on Grex in effect now. During this outage, Physical Plant will work on HPCC power and cooling, and the entire Grex system will be powered down.

Users will not have access to any Grex services (compute and storage and Web portal) during the outage. During the outage, all running jobs will be terminated.

Should you have any questions about the upcoming Grex outage, please do not hesitate to contact us at support@tech.alliancecan.ca ! Thank you for your patience,

Your Grex HPC team.

[Resolved] Planned Grex outage for Lustre FS and Major Linux update

Tue, 07 May 2024 09:00:00 +0000

Final update on May 10

The OS and Lustre update outage is over!

The OS on Grex was upgraded to Alma Linux on all CPU, GPU and login nodes {except for bison, tatanka and compute partition}. Now, Grex is available for running jobs. Please note the changes about software stack:

https://um-grex.github.io/grex-docs/updates/

If you have questions or concerns, please don’t hesitate to contact us at: support@tech.alliancecan.ca

Update on May 9

The outage extended into May 10. The /home filesystem had been updated; update of the Linux OS and HPC software is still in progress. Sorry about the inconvenience and thank you for your patience!

Update on May 8

The outage continues. The /project filesystem appliance had been updated. We are working on updating the /home filesystem, Linux OS and HPC software stacks. The system is still closed for users.

Grex system outage on May 7 - 9 2024

There is a planned outage on Grex in effect now . We are performing a major update of the DDN Lustre storage controller. We will also do a major Linux OS update from CentOS 7 to AlmaLinux 8. We will reboot and reinstall all of Grex compute and login nodes. All running and queued jobs will be deleted. Grex login nodes , compute and storage will be unavailable during the outage window. Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Planned Grex outage for SLURM and minor OS update

Mon, 18 Dec 2023 09:00:00 +0000

Grex system outage completed on Dec 19 2023

The SLURM scheduler had been updated. Linux OS also had a minor update. The Grex system is now fully operational. If you encounter any problems, please do not hesitate to contact us at support@tech.alliancecan.ca !

Grex system outage on December 18 2023

There is a planned outage on Grex in effect. We are performing a major update of the SLURM scheduler, communication libraries, and minor Linux OS updates for security patching.

We will eboot and reinstall all of Grex compute and login nodes and to migrate the SLURM job database. Thus Jobs that are still running by the time of the outage will be lost, and Grex login nodes will be unavailable during the outage window.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Two Planned Grex outages for HPCC transformer work

Tue, 12 Sep 2023 18:00:00 +0000

Both power outages are now complete

Grex system is open to the users. Queued jobs were not affected and seems to be running now. There were no major upgrades or changes done during the outage.

Planned Grex power outages in September 2023

The Physical Plant is about to perform some electrical works on the transformer that feeds, amongst other things on campus, the HPCC data center that hosts Grex. The outage will start for 6 PM on September 12 and September 19. These outages would require a complete power shutdown in HPCC for about an hour, which means the system would be completely inaccessible to the users, and all running jobs would be terminated.

To avoid the failure of jobs, we have made two reservations to avoid any longer jobs (that cannot be finished before the beginning of the outage) from starting. For more information, run the following command from any login node: “scontrol show res”. To take advantage of the cluster before, and between, the outages, we recommend users submit short jobs that can finish by the time the upcoming outage begins.

Thank you for your patience! Should you have questions or concerns, please do not hesitate to contact us at support@tech.alliancecan.ca !

[Resolved] Unplanned power outage in HPCC datacentre

Tue, 30 May 2023 07:00:00 +0000

UPDATE: outage concluded

Grex compute and storage are back up after 1 1/2 hour of downtime. Please restart your jobs! If you notice any further malfuncions, please do not hesitate to contact us at the support@tech.alliancecan.ca , mentioning “Grex” in the subject line.

Unplanned power loss at HPCC datacentre

NOTICE: On May 30. 7AM to 8:14AM there was an unplanned power outage in HPCC that rebooted all the Grex compute nodes, storage, and management systems. Running jobs were lost. We apologize for the inconvenience. We work on restarting and testing of the system.

[Resolved] Planned Grex outage for HPCC work and storage update

Wed, 22 Feb 2023 14:30:00 +0000

UPDATE: outage concluded

During the outage of Feb 22, few changes have been made on Grex: We reinstalled all login and compute nodes with a new image to fix the errors with UCX. We have added a new storage “project” with similar structure as for Compute Canada. All the data from scratch has been moved to the project file system.

For an overview of the changes, please have a look to the documentation page:

https://um-grex.github.io/grex-docs/docs/lustre/

Planned Grex outage

The Grex storage outage is about to start. Please save your interactive work! ETA for the outage’s end is Feb 23 or Friday, Feb 24, 2023.

During the outage we are planning to reboot and reinstall all of Grex compute and login nodes, connect the new Project storage, and perform some re-cabling of the Infiniband fabric of the cluster. Jobs that are still running by the time of the outage will be lost, and Grex login nodes will be unavailable during the outage window.

Thank you for your patience! If you have questions or concerns regarding the outage, please do not hesitate to contact us at mailto:support@tech.alliancecan.ca !

[Resolved] Electric failure, loss of power to management rack

Thu, 26 Jan 2023 14:50:00 +0000

Update: datacentre change rolled back, systems operational

It appears that some of the running jobs continued running , and login nodes’ access nor storage systems were affected. Grex is now operational. In case you notice any ongoing issue, please let us know by email to support@tech.alliancecan.ca, Subject line containing Grex.

Faulty electrical work, loss of power to management rack

We have an unplanned outage on Grex due to a failed electrical work that affected its management rack at around 2:50 PM, Jan 26, 2023 .

Running jobs may be lost, and access to Grex login nodes may be degraded or unavailable.

We are working on resolving the issue. Sorry about the inconvenience it may have caused.

[Resolved] One of the /global/scratch filesystem servers crashed, FS runs in degraded mode

Fri, 13 May 2022 19:30:00 +0000

Lustre filesystem problems

One of the /global/scratch filesystem servers crashed, FS runs in degraded mode. We are investigating the issue. Most running jobs will continiue to run, albeit slower, and may stuck in I/O operations while Lustre servers are unresponsive.

Update:

After restarting of Lustre servers, the /global/scratch filesystem is back to normal operation.

[Resolved] Planned power outage Aug 31 to Sept 1

Tue, 31 Aug 2021 08:00:00 +0000

Grex is back online

Grex is back online, accepting CPU jobs. GPU nodes will take a bit more to reinstall NVIDIA updates, but will be online by end of today.

The previous jobs on the queue were lost since the outage afftected all compute nodes that are rebooted after restoring the power. Please check your data and re-submit the jobs that are not done before the outage.

Grex will be down for a campus power maintenance

Physical Plant has a planning a power outage for power breakers maintenance. This will affect Grex building, so and all the nodes and strage systems will be shut-down. The system will be unavailable for users, and all running jobs will be lost. We will be performing some work on Grex during the maintenance window as well. ETA for Grex availabinity is noon, September 1.

If you have questions or concerns, please don’t hesitate to contact us at: support@computecanada.ca , mentioning Grex in the subject line.