There are two main reasons you may get a “disk quota exceeded” error on CARC systems.
1 — Hard quota limit
First, enter the command myquota and check your storage usage. You may have simply hit a hard quota limit, either total storage space or total number of files. In this case, try to delete, compress, consolidate, and/or archive files to free up space. Alternatively, for /project directories, you can request more space via a support ticket.
2 — Incorrect group ownership
Second, for /project directories specifically, if your storage usage is not actually near the quota limits, then the likely cause is that the group ownership of some of your project files does not match the project group ID. For example, files in the project directory /project/ttrojan_123 should have group ownership by ttrojan_123:
[ttrojan@discovery1 ~]$ ls -ld /project/ttrojan_123/ttrojan/file.txt
-rw-rw---- 1 ttrojan ttrojan_123 293 Dec 10 15:10 /project/ttrojan_123/ttrojan/file.txt
In this example, ttrojan is the user owner ID and ttrojan_123 is the group owner ID.
The group ID is used to enforce the storage quota limits for project directories. By default, new files and directories should have the correct group ID, but it is possible to override this. Typically, for this error, the group ID for some of your files is your personal group (same as your username) (e.g., ttrojan), which has a very small quota thus producing the “disk quota exceeded” error if new files are written with the personal group ID.
To check if you have files with the incorrect group ID, enter a command like the following substituting your username:
beegfs-ctl --getquota --mount=/project --gid ttrojan
To find files with the incorrect group ID, enter a command like the following substituting your project directory path and project group ID:
find /project/ttrojan_123/ttrojan \! -group ttrojan_123
The likely reason for files having the wrong group ID is using a mv, cp -a, scp -r, rsync -p, or rsync -a command that preserves file permissions from source files when moving or copying them into the project directory. Alternatively, some subdirectories within your project directory may not have the correct setgid bit that determines the default group ID for new files and directories.
To fix this issue, enter a sequence of commands like the following, substituting your project directory path and project ID:
chgrp -R ttrojan_123 /project/ttrojan_123/ttrojan
find /project/ttrojan_123/ttrojan -type d -exec chmod g+s {} \;
These commands will recursively change and set the default group ownership of files and subdirectories to match the project group. You will get an “operation not permitted” message for files you do not own, but this is fine. These commands will only change the files that you own.
It is best to run these commands only for specific subdirectories where you know you have files, because they may take awhile to run, especially for large directories.
You may also need to submit these commands for each project directory you have access to.
Please note that it may take ~15 minutes for the quota to update, after which you should be able to save new files again.
The best method for moving or copying files into a project directory is rsync -rlt. If needed, you can delete the source files after a successful copy or add the --remove-source-files option.