How to wait until the job is finished?

Discovery Cluster Slurm
1

Hello,

I am running a bash file to submit several lumerical jobs. The file runs a for loop to submit jobs. I want to wait until the the job is finished. Is there a way to get job ID of the job just started and wait until it is done and then submit the next job? I tried using wait command but somehow could not extract job ID.

Currently I am just using watch command and manually terminate it using Ctrl+C aso that the loop can continue.

My bash file is something like this-

#!/bin/bash

for k in {1..100};
	 do temp="${k}";
         fname="sweep_$temp.fsp";
         #echo $fname;
	 runFDTD_discovery $fname 2:00:00 1 1 10;
	 watch -n 1 squeue --user=aghaneka
done

Thanks,
Alok

2

Hi Alok,

It sounds like you want to submit 100 jobs but don’t want to run them until the previous one has completed. Is that correct? If so, you can use Slurm’s --dependency option. You would have to modify runFDTD_discovery to include this option and you also need a way to pass the previous job ID to the next job.

Here’s an example of how you might use it

#!/bin/bash

# Get first job id

jid=$(sbatch job1.slurm | cut -d ' ' -f4)

# Remainder jobs
for k in {2..100};
    do temp="${k}"
        jid=$(sbatch --dependency=afterok:${jid} job${k}.slurm | cut -d ' ' -f4)
    done

Note that normally submitting a job returns a message similar to

Submitted batch job 778328

So the cut -d ' ' -f4 portion will return the 4th string (job id) using ’ ’ as a delimter.

1 Like
3

@csul

Thanks lot. @raudhkha modified the runFDTD_discovery to generate slurm files without submitting the job. And then using the job submittal script you suggested, we are able to queue the jobs one by one.