Batch jobs running inside matlab not finished

Discovery Cluster
1

Dear HPC staff, I am runinng a matlab scirpt which uses the batch() function to create multiple jobs, and this matlab script is called via a slurm script which i defined with mulitple cpus, I noticed my job is not finished after the main matlab script that createds them finish and exits. I wonder is this beacuse i did not set a wait in my main matlab script? When i run the matlab script on my local it succsuflly distrbuted the jobs in background and finished them.

Also when i try to print out all jobs in my matlab cluster, i got 2000 jobs exisiting, which i am not sure if this mean the cluster is shared on all user using this patitaiton or they are my history jobs.

attached is my slurm and matlab code:

#!/bin/bash
       
#SBATCH --cpus-per-task=5
#SBATCH --mem=150GB
#SBATCH --time=48:00:00 ## maximal up to 48 hours (48:00:00)
#SBATCH --partition=main
#SBATCH --account=
#SBATCH --job-name=batch_stdp_sim
#SBATCH --output=log_%x-%j.out
#SBATCH --error=log_%x-%j.error
##SBATCH --array=1-20                 # Array range
##SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)


module purge
module load gcc/11.3.0
module load cuda/11.6.2
module load googletest/1.10.0
module load matlab/2022a
module load cmake/3.23.2
module load libpthread-stubs/0.4
module load python/3.9.12
module load swig/4.0.2
module load nvidia-hpc-sdk/21.7


matlab -nodisplay -nosplash -nodesktop -r "run('run_batch_sim.m'); exit;"

matlab code


for i_se = 1:length(rand_seeds)
 rand_seed = rand_seeds(i_se);
                
 j{trial_count} = batch('sim_function_sigmoid_xy',N_ret_var,vars,'CaptureDiary',true);
                            trial_count = trial_count +1                        
                        end

code to print jobs

% Assuming 'c' is your parcluster object
c = parcluster;

% Get the list of all properties for the parcluster object
props = properties(c);

% Loop through each property and print its name and value
fprintf('Properties of the parcluster object:\n');
for i = 1:length(props)
    propName = props{i}; % Get the name of the property
    propValue = c.(propName); % Access the property value dynamically

    % Check if the property value is an object or a container that needs special handling
    if isobject(propValue) || isstruct(propValue) || iscell(propValue)
        try
            % Attempt to convert the property value to a string for display
            % This may not work for all types, hence the try-catch block
            propValueStr = evalc('disp(propValue)');
        catch
            propValueStr = 'Complex value, cannot display directly.';
        end
    else
        % For simple properties, convert directly to string
        propValueStr = mat2str(propValue);
    end

    % Print the property name and value
    fprintf('%s: %s\n', propName, propValueStr);
end
2

Hi,

Based on the Matlab documentation, I think you are correct.

  • batch does not block MATLAB and you can continue working while computations take place. If you need to block MATLAB until the job finishes, use the wait function on the job object.
wait(job)

When you make a cluster in Matlab, only you have access to it.

1 Like
3

Thank you so much!!
With that i can delete the failed/pending jobs in my the background of my instance since they are all created by me and i won’t have seen jobs created by others.