{"is":"issue","title":"Unplanned project storage outage in HPCC","body":"\u003ch4 id=\"update-1730-central-time-system-is-available\"\u003eUpdate 17:30 Central Time: System is available\u003c/h4\u003e\n\u003cp\u003eThe storage is working. If you notice any issues with the storage, please do not hesitate to contact us at \u003ca href=\"mailto:support@tech.alliancecan.ca\"\u003esupport@tech.alliancecan.ca\u003c/a\u003e, mentioning Grex in the subject line.\u003c/p\u003e\n\u003ch4 id=\"update-1700-central-time-system-is-available-with-conditions\"\u003eUpdate 17:00 Central Time: System is available with conditions\u003c/h4\u003e\n\u003cp\u003eThe failure was traced to a hardware issue on one of the storage controllers.\nWe have cleared the controller state and restarted the /project storage appliance.\nHowever, the storage targets are not yet properly balanced across the storage servers, so there can be some performance issues. We are working with the storage vendor to address these issues.\u003c/p\u003e\n\u003cp\u003eAs of now, OpenOnDemand and /project are now available, and Grex system would accept new jobs.\u003c/p\u003e\n\u003ch4 id=\"a-project-storage-outage-happened-on-grex\"\u003eA /project storage outage happened on Grex\u003c/h4\u003e\n\u003cp\u003eA /project storage outage happened on Grex this morning! The /project filesystem became unavailable.\nThis affectes running jobs and access to the OpenOnDemand portal as well. We are investigating the issue.\u003c/p\u003e\n","createdAt":"2025-07-18 07:10:00 +0000 UTC","lastMod":"2025-07-18 07:10:00 +0000 UTC","permalink":"https://um-grex.github.io/status/issues/2025-07-18-storage-outage/","severity":"disrupted","resolved":true,"informational":false,"resolvedAt":"2025-07-18 17:30:00","affected":["Lustre /project"],"filename":"2025-07-18-storage-outage.md"}