I typically use job arrays like this:
#SBATCH --nodes=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=12G
#SBATCH --time=3:00:00
#SBATCH --array=1-4
apptainer exec ...
...
I was under the impression that all CPUs will have access to the memory pool but the memory will not be equally divided between them. But when I allocate more memory (e.g. 16G vs 12G) I see that my memory usage (according to the rough estimate provided by Jobinfo) scales with the allocated memory size.
$ jobinfo 29979_1
...
Used CPU walltime : 03:55:06
Used CPU time : 01:56:42
CPU efficiency : 49.64%
% User (computation) : 87.70%
% System (I/O) : 12.30%
**Reserved memory : 12G**
**Max memory used : 6.41G (estimate)**
**Memory efficiency : 53.38%**
Max disk write : 7.48G
Max disk read : 14.87G
$ jobinfo 26730_4
...
Used walltime : 02:19:22
Used CPU walltime : 04:38:44
Used CPU time : 02:16:24
CPU efficiency : 48.94%
% User (computation) : 91.28%
% System (I/O) : 8.71%
**Reserved memory : 16G**
**Max memory used : 8.48G (estimate)**
**Memory efficiency : 53.01%**
Max disk write : 9.03G
Max disk read : 17.60G
Both jobs are preforming roughly the same operation, and it seems it’s not parallelized so one CPU does most of the job, but it gets roughly half of the allocated memory regardless of the total memory allocation. How can I relax this constraint?
(I had better examples (i.e. identical jobs) but apparently the whole job history from before maintenance is lost.)