Notes on SSH keys in user home directories

Some users have noticed the Discovery login node and compute resources have a slightly different SSH key set up. When you’re allocated an account and your home directory is set up, a cluster key is automatically generated in your .ssh directory:

[christay@discovery ~]$ ls ~/.ssh/cluster*
/home1/christay/.ssh/cluster /home1/christay/.ssh/cluster.pub

These are created as part of our configuration management system, which we rely on to provision hundreds of compute nodes at a time- it sets up a way for users to SSH to compute nodes allocated to them when they submit jobs on the cluster.

Don’t mess with them! :blush: They’re there to make your life easier and if you accidentally modify or overwrite them you’ll have to open a Jira ticket to get one of us to help fix it for you.

Anyway, some users have noticed that since this cluster key pair is the default, something like SSHing to Github (for instance) might produce a message like this:

[christay@discovery work]$ git clone git@github.com:chris/testing1.git
Cloning into 'testing1'...
Warning: Permanently added 'github.com,140.82.113.4' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Result: Fail!

That’s because the endpoint is trying to let you in with cluster.pub key- the default. You could always upload that key to Github, but why not just make a new one for yourself to use for this purpose? It’s easy to do (one example, call it whatever you want):

[christay@discovery .ssh]$ ssh-keygen -t rsa -b 2048 -f id_rsa_git
[christay@discovery .ssh]$ ls id_rsa_git*
id_rsa_git id_rsa_git.pub

Upload the .pub file to your account on github.com, and now you’ve got a key you can use for your personal use to interact with your Git repo. But, how do you transparently use this key instead of the default cluster key?

One way is to add something to your ~/.ssh/config – BE CAREFUL! Don’t change the part created by our friendly Warewulf node provisioning software- just add a few lines below it like this:

[christay@discovery work]$ cat ~/.ssh/config
# Added by Warewulf 2020-01-28
Host *
  IdentityFile ~/.ssh/cluster
  StrictHostKeyChecking=no
# Add below here -->
Host github.com
  HostName github.com
  IdentityFile ~/.ssh/id_rsa_git
  User <my Git username>

Now try your git clone and it will sail right through because you specified the key you generated for the github.com endpoint.

If you don’t feel like messing with the ~/.ssh/config file (maybe a good idea…), there’s another way that’s not quite so concise but gets the job done, and once you set it up it’s there whenever you need it.

Create a wrapper script for the ssh binary only for use getting to Github- something like this, where I created a file “ssh-to-git.sh” in my home directory and made it executable with ‘chmod u+x’:

[christay@discovery work]$ cat ~/ssh-to-git.sh
#!/bin/sh
ssh -i ~/.ssh/id_rsa_git -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $*

Then, whenever I want to clone from my repo or whatever, I use the “GIT_SSH” environment variable our version of git supports, right on one line:

[christay@discovery work]$ GIT_SSH=~/ssh-to-git.sh git clone [git@github.com:chris/testing1.git](mailto:git@github.com:chris/testing1.git)

Result: Success!

I looked online and there are lots of ways to specify your own SSH key when interacting with a service like Github. You can create and manage your own keys as long as you don’t touch the “cluster” and “cluster.pub” files in your .ssh directory. We hope this quick post give you some ways to streamline your research work on the Discovery cluster!

5 Likes

This is great. Thank you for the details, Chris!

Fantastic post!! Was able to connect to github by following this.

One minor thing here:

[christay@discovery work]$ cat ~/.ssh/config

Added by Warewulf 2020-01-28

Host *
IdentityFile ~/.ssh/cluster
StrictHostKeyChecking=no

Add below here →

Host github.com
HostName github.com
IdentityFile ~/.ssh/id_rsa_git
User

IdentityFile ~/.ssh/id_rsa_git should actually be IdentityFile ~/id_rsa_git

1 Like

Glad you found this useful.

That’s interesting, you make a good point- if you run ssh_keygen, you could tell it to leave the SSH keys anywhere in your home directory. I always thought ‘ssh’ would complain about permissions if they weren’t in your .ssh directory, protected by ‘drwx------’ permissions. I’ll have to try that out. Generally for me, I think it’s convenient to keep everything in ~/.ssh .

1 Like

Ah did not realize the path .ssh here.


You are right! I was creating keys directly in my home directory, which might not be a good practice.

1 Like