https://um-grex.github.io/status/affected/computecanada-cvmfs/Incident historygithub.com/cstateen-us2025-04-23T09:10:00+00:002025-04-23T09:10:00+00:00The MIT License (MIT) Copyright © 2025 UM-Grexhttps://um-grex.github.io/status/issues/2025-04-23-unplanned-power-outage/Wed, 23 Apr 2025 09:10:00 +0000https://um-grex.github.io/status/issues/2025-04-23-unplanned-power-outage/2025-04-23 12:40:00<h4 id="power-to-the-hpcc-centre-restored">Power to the HPCC Centre restored</h4> <p>Manitoba Hydro had restored power to Campus, and Grex is back online. All running and queued jobs were lost during the outage. We have used the opportunity to update SLURM scheduler to the current major version 24.11 . All Grex subsystems (compute, storage, login nodes and Web portal) are operational.</p> <p>If you have questions or concerns, please do not hesitate to contact us at <a href="mailto:support@tech.alliancecan.ca">support@tech.alliancecan.ca</a> , mentioning Grex in the subject line.</p><h4 id="power-to-the-hpcc-centre-restored">Power to the HPCC Centre restored</h4> <p>Manitoba Hydro had restored power to Campus, and Grex is back online. All running and queued jobs were lost during the outage. We have used the opportunity to update SLURM scheduler to the current major version 24.11 . All Grex subsystems (compute, storage, login nodes and Web portal) are operational.</p> <p>If you have questions or concerns, please do not hesitate to contact us at <a href="mailto:support@tech.alliancecan.ca">support@tech.alliancecan.ca</a> , mentioning Grex in the subject line.</p> <h4 id="a-power-outage-happened-in-hpcc-centre">A power outage happened in HPCC Centre</h4> <p>A power outage in Grex&rsquo;s datacentre happened , with a complete loss of power at about 9:10 AM Winnipeg time. The system is down. The reason for the outage is a problem at Manitoba Hydro, our electricity provider.</p> <p><a href="https://account.hydro.mb.ca/Portal/outeroutage.aspx">https://account.hydro.mb.ca/Portal/outeroutage.aspx</a></p> <p>We are waiting for the power to be restored. Thank you for your patience!</p> <p>If you have questions or concerns, please do not hesitate to contact us at <a href="mailto:support@tech.alliancecan.ca">support@tech.alliancecan.ca</a> , mentioning Grex in the subject line.</p>https://um-grex.github.io/status/issues/2025-02-23-hpcc-poweroutage/Sun, 23 Feb 2025 07:00:00 +0000https://um-grex.github.io/status/issues/2025-02-23-hpcc-poweroutage/2025-02-24 17:10:00<h4 id="hpcc-planned-power-outage-update">HPCC planned power outage update</h4> <p>The outage started on Feb 23 is over. Grex is operational. Some of the GPU compute nodes may still be unavailable, and will be in production shortly.</p> <h4 id="hpcc-planned-power-outage">HPCC planned power outage</h4> <p>Physical Plant had informed us that it plans to shut down power feed for transformer work, to several buildings including HPCC where Grex is located. For this reason, we have a complete Grex outage starting from 7 AM, Feb 23, 2025. The planned end of the outage is by the end of the day on Monday, Feb 24. The system is completely be unavailable to the users. Login nodes and data are not be accessible during the outage.</p><h4 id="hpcc-planned-power-outage-update">HPCC planned power outage update</h4> <p>The outage started on Feb 23 is over. Grex is operational. Some of the GPU compute nodes may still be unavailable, and will be in production shortly.</p> <h4 id="hpcc-planned-power-outage">HPCC planned power outage</h4> <p>Physical Plant had informed us that it plans to shut down power feed for transformer work, to several buildings including HPCC where Grex is located. For this reason, we have a complete Grex outage starting from 7 AM, Feb 23, 2025. The planned end of the outage is by the end of the day on Monday, Feb 24. The system is completely be unavailable to the users. Login nodes and data are not be accessible during the outage.</p>