General Storage Information#
As of now, the storage system of Grex consists of the following:
- The /home NFSv4/RDMA filesystem is served by a very fast NVME disk server. The total size of the filesystem is 15 TB. The quota per-user is 100 GB of space and 500K files. This quota can not be increased. For larger amount of data, users should use project.
- The /project Lustre filesystem: as of Sep 2022, an additional storage of 1 PB was added to Grex. On Jan 2024, the /project was extended by another 1 PB.
| File system | Type | Total space | Quota/User | Number of files |
|---|---|---|---|---|
| /home | NFSv4/RDMA | 15 TB | 100 GB | 500 K |
| /project | Lustre | 2 PB | - | - |
- The local node storage as defined by the environment variable $TMPDIR is recommended for temporary job data that is not needed after job completes. Grex nodes have SATA local disks of various capacities, leaving 150 Gb, 400 Gb, 800 Gb and 1700 Gb usable space per node, depending on the kind of local disk it has.
Most users would want to use /home/USERNAME for code development, source code, scripts, visualization, processed data, … etc., that do not take much space and benefits for small files I/O. For production data processing, that is, massive I/O tasks from many compute or interactive jobs, the /project FS should be used. It is often beneficial to place temporary files on the local disk space of the compute nodes, if space permits, so that the jobs do not load Lustre or NFS servers extensively.
Data retention and Backup#
Data retention policy as of now conforms to the corresponding of the Alliance (former Compute Canada) policy. While a user’s CCDB account is active and renewed, there will be no purges on /home nor /project filesystems. Should the CCDB account expire, the data will not be kept on Grex indefinitely! The data retention period for expired accounts is one (1) year. Data on login and compute nodes’ local scratch disks gets purged regularly and automatically.
A local tape backup is provided for /home and /project filesystems. Please refer to the Data Backup page for more information.
Data sharing#
Sharing of accounts login information (like passwords or SSH keys) is strictly forbidden on Grex, as well as on most of the HPC systems. There is a mechanism of data/file sharing that does not require sharing of the accounts. To access each other’s data on Grex, the UNIX groups and permissions mechanism can be used. For more information, please refer to the section Sharing Data .
Disclaimer of any responsibility for Data loss#
Every effort is made to design and maintain Grex storage systems in a way that they are a reliable storage for researcher’s data. However, we (Grex Team, or the University) make no guarantees that any data can be recovered, regardless of where they are stored, even in backed up volumes. Accidents happen, whether caused by human error, hardware or software errors, natural disasters, or any other reason. It is the user’s responsibility that the data is protected against possible risks, as appropriate to the value of the data.