Hi all,
Every excited about this new system. So far it’s been working really well.
A small question I do have. In the getting started page https://carc.usc.edu/user-information/user-guides/hpc-basics/getting-started-discovery, it’s recommended to use Singularity to install & compile new apps.
However, when I do “module avail” in my discovery account, I couldn’t see a Singularity. Instead, only singularity-legacy/2.6.1 is available. Singularity help page says this:
Help:
Singularity is a container platform focused on supporting ‘Mobility of
Compute’. The software changed the installation method from AutoTools to
GoLang, so we have two separate package names to support that. The
legacy package is pre-version 3.0.0
Just wondering how can we access newer versions of Singularity or should we install it by ourselves from source?
Thanks for your work again!
1 Like
Hi, I see you opened a Jira ticket and our research staff are working on a new Singularity version to be available soon. That’ll be good for the community to take advantage of- Singularity containers are something I’ve really wanted to play around with but haven’t had time yet. We’d like to hear what kind of work you’re doing with it once you get it going- thanks.
2 Likes
Hi Sebastian,
Singularity 3.6.1 is now available on Discovery. We installed it outside of the module system, as an RPM package, so you do not need to load a module.
Of course, Singularity commands that require sudo will not be available to you. If you need to build a container from a definition file, then you will have to do this externally and then transfer it to Discovery. The easiest way to do this is using Sylabs’ cloud builder: https://cloud.sylabs.io/home
2 Likes
Hi Chris and everyone,
I am currently using GATK tools from Broad Institute and i was able to run it successfully with Singularity on Discovery. Here is how I made it work:
Define a cache folder in the scratch folder so it doesn’t clog up home1 folder:
export SINGULARITY_CACHEDIR="/scratch/…/singularity-cachedir"
cd to the folder where you want to save the image file (.sif)
Pull image from singularity containers or in my case a docker container which is also compatible
singularity pull docker://broadinstitute/gatk
#By default Singularity only mounts /home folder which is annoying because most of your files are probably elsewhere like scratch. Therefore it’s important to specify you working folder to mount later with --bind. For example:
–bind /scratch/spongebob/WES:/mount
This will make the WES folder in my scratch folder available in Singularity as /mount.
To run the image interactively:
singularity run
–bind /scratch/spongebob/WES:/mount
/scratch/…/gatk_latest.sif
Now I am inside the container and can do all sorts of stuff with that image.
To include software in the image in your slurm script:
singularity exec
–bind /scratch/spongebob/WES:/mount
/scratch/…/gatk_latest.sif
gatk CNNScoreVariants
-R /mnt/ref/Homo_sapiens_assembly19.fasta
-V /mnt/vcfs/g94982_b37_chr20_1m_15871.vcf.gz
-O /mnt/sandbox/my_1d_cnn_scored.vcf
Hope this helps! Really glad this is now available to the USC community!
Sebastian
6 Likes
Wow thanks for that great step by step, Sebastian. And thanks Derek for getting the new version installed!