Unable to connect to discovery from VS code

taoanhua · April 27, 2021, 4:41am

Hello,
I am not able to connect to discovery from VS code using the remote-ssh plugin. I have tried removing .vscode-server/ and other troubleshooting tricks suggested by here, but none of them has worked.
The login process seems to get stuck on “checking some files in /.vscode-server for a running server”.

Is anyone experiencing the same issues or has encountered similar problems before?
Any and all help is greatly appreciated.

Below is the VS code connection log:

Blockquote
[20:45:43.811] Log Level: 2
[20:45:43.812] remote-ssh@0.65.4
[20:45:43.812] darwin arm64
[20:45:43.812] SSH Resolver called for “ssh-remote+discovery.usc.edu”, attempt 1
[20:45:43.812] “remote.SSH.useLocalServer”: true
[20:45:43.812] “remote.SSH.path”: undefined
[20:45:43.812] “remote.SSH.configFile”: undefined
[20:45:43.812] “remote.SSH.useFlock”: true
[20:45:43.812] “remote.SSH.lockfilesInTmp”: false
[20:45:43.813] “remote.SSH.localServerDownload”: auto
[20:45:43.813] “remote.SSH.remoteServerListenOnSocket”: false
[20:45:43.813] “remote.SSH.showLoginTerminal”: true
[20:45:43.813] “remote.SSH.defaultExtensions”: []
[20:45:43.813] “remote.SSH.loglevel”: 2
[20:45:43.813] SSH Resolver called for host: discovery.usc.edu
[20:45:43.813] Setting up SSH remote “discovery.usc.edu”
[20:45:43.814] Acquiring local install lock: /var/folders/hl/xzs6_97j57b4dg1y62r8b2dc0000gn/T/vscode-remote-ssh-discovery.usc.edu-install.lock
[20:45:43.818] Looking for existing server data file at /Users/taoanhuang/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-discovery.usc.edu-3c4e3df9e89829dce27b7b5c24508306b151f30d-0.65.4/data.json
[20:45:43.818] Using commit id “3c4e3df9e89829dce27b7b5c24508306b151f30d” and quality “stable” for server
[20:45:43.820] Install and start server if needed
[20:45:43.821] PATH: /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
[20:45:43.821] Checking ssh with “ssh -V”
[20:45:43.826] > OpenSSH_8.1p1, LibreSSL 2.7.3
[20:45:43.827] askpass server listening on /var/folders/hl/xzs6_97j57b4dg1y62r8b2dc0000gn/T/vscode-ssh-askpass-ff74eb1a5929ccf00df1cc16d97aab9174d0a698.sock
[20:45:43.827] Spawning local server with {“ipcHandlePath”:"/var/folders/hl/xzs6_97j57b4dg1y62r8b2dc0000gn/T/vscode-ssh-askpass-246c19fe15379bc340debc758c3043818f9bd15f.sock",“sshCommand”:“ssh”,“sshArgs”:["-v","-T","-D",“57429”,"-o",“ConnectTimeout=120”,“discovery.usc.edu”],“dataFilePath”:"/Users/taoanhuang/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-discovery.usc.edu-3c4e3df9e89829dce27b7b5c24508306b151f30d-0.65.4/data.json"}
[20:45:43.827] Local server env: {“DISPLAY”:“1”,“ELECTRON_RUN_AS_NODE”:“1”,“SSH_ASKPASS”:"/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/local-server/askpass.sh",“VSCODE_SSH_ASKPASS_NODE”:"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)",“VSCODE_SSH_ASKPASS_MAIN”:"/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/askpass-main.js",“VSCODE_SSH_ASKPASS_HANDLE”:"/var/folders/hl/xzs6_97j57b4dg1y62r8b2dc0000gn/T/vscode-ssh-askpass-ff74eb1a5929ccf00df1cc16d97aab9174d0a698.sock"}
[20:45:43.828] Spawned 37804
[20:45:43.897] > local-server> Spawned ssh: 37806
[20:45:43.899] stderr> OpenSSH_8.1p1, LibreSSL 2.7.3
[20:45:44.121] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:pO0FfPYahmoh/8UcgHpLKxdJe52fNuX9bCodsHI4rVE
[20:45:44.318] Got askpass request: {“request”:“Enter passphrase for key ‘/Users/taoanhuang/.ssh/id_rsa-remote-ssh2’:”}
[20:45:44.318] Detected passphrase message
[20:45:44.318] Listening for interwindow password on /var/folders/hl/xzs6_97j57b4dg1y62r8b2dc0000gn/T/vscode-ssh-askpass-b29d4b908ebfda15184c88d74e7691eb76d67298.sock
[20:45:44.318] Writing password prompt to globalState
[20:45:48.194] Got passphrase response
[20:45:48.195] Interactor gave response: ****************
[20:45:48.195] Cleaning up other-window auth server
[20:45:48.589] stderr> Authenticated with partial success.
[20:45:48.977] Got askpass request: {“request”:“Duo two-factor login for taoanhua Enter a passcode or select one of the following options: 1. Duo Push to XXX-XXX-3239 2. Phone call to XXX-XXX-3239 3. SMS passcodes to XXX-XXX-3239 (next code starts with: 2) Passcode or option (1-3):”}
[20:45:48.977] Listening for interwindow password on /var/folders/hl/xzs6_97j57b4dg1y62r8b2dc0000gn/T/vscode-ssh-askpass-73ab231aa60eb18567a96729c20099078b9743f8.sock
[20:45:48.977] Writing password prompt to globalState
[20:45:50.851] Interactor gave response: *
[20:45:50.851] Cleaning up other-window auth server
[20:45:57.617] stderr> Authenticated to discovery.usc.edu ([68.181.11.6]:22).
[20:45:57.791] > Success. Logging you in…
Last failed login: Mon Apr 26 20:45:41 PDT 2021 from 104.32.174.101 on ssh:notty
There were 2 failed login attempts since the last successful login.
[20:45:58.148] > ready: 8a2d0193227e
[20:45:58.170] > Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020
[20:45:58.170] Platform: linux
[20:45:58.211] > 8a2d0193227e: running
[20:45:58.240] > Acquiring lock on /home1/taoanhua/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/vscode-remote-lock.taoanhua.3c4e3df9e89829dce27b7b5c24508306b151f30d
[20:45:58.244] > Found existing installation at /home1/taoanhua/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d…
Checking /home1/taoanhua/.vscode-server/.3c4e3df9e89829dce27b7b5c24508306b151f30d.log and /home1/taoanhua/.vscode-server/.3c4e3df9e89829dce27b7b5c24508306b151f30d.pid for a running server
[21:05:58.382] stderr> Connection to discovery.usc.edu closed by remote host.
[21:05:58.383] stderr> Transferred: sent 18876, received 4044 bytes, in 1200.8 seconds
[21:05:58.383] stderr> Bytes per second: sent 15.7, received 3.4
[21:05:58.390] > local-server> ssh child died, shutting down
[21:05:58.397] Local server exit: 0
[21:05:58.398] Received install output: OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:pO0FfPYahmoh/8UcgHpLKxdJe52fNuX9bCodsHI4rVE
Authenticated with partial success.
Authenticated to discovery.usc.edu ([68.181.11.6]:22).
Success. Logging you in…
Last failed login: Mon Apr 26 20:45:41 PDT 2021 from 104.32.174.101 on ssh:notty
There were 2 failed login attempts since the last successful login.
ready: 8a2d0193227e
Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020
8a2d0193227e: running
Acquiring lock on /home1/taoanhua/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/vscode-remote-lock.taoanhua.3c4e3df9e89829dce27b7b5c24508306b151f30d
Found existing installation at /home1/taoanhua/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d…
Checking /home1/taoanhua/.vscode-server/.3c4e3df9e89829dce27b7b5c24508306b151f30d.log and /home1/taoanhua/.vscode-server/.3c4e3df9e89829dce27b7b5c24508306b151f30d.pid for a running server
Connection to discovery.usc.edu closed by remote host.
Transferred: sent 18876, received 4044 bytes, in 1200.8 seconds
Bytes per second: sent 15.7, received 3.4
[21:05:58.401] Failed to parse remote port from server output
[21:05:58.424] Resolver error: Error:
at Function.Create (/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:64641)
at Object.t.handleInstallOutput (/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:63284)
at Object.e [as tryInstallWithLocalServer] (/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:386419)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async /Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:294035
at async Object.t.withShowDetailsEvent (/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:405329)
at async /Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:384890
at async E (/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:381658)
at async Object.t.resolveWithLocalServer (/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:384517)
at async Object.t.resolve (/Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:1:295505)
at async /Users/taoanhuang/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.4/out/extension.js:127:110530
[21:05:58.432] ------

csul · April 27, 2021, 5:19pm

I just tried starting up my old copy of VS Code and get stuck at

Checking for *.log and *.pid for a running server

This issue seems to come and go periodically and every time it’s resolved the solution is some variant of reinstalling VS Code or the Remote SSH plugin.

This issue https://github.com/microsoft/vscode-remote-release/issues/4415 and the related ones I could find have no satisfying conclusions but it seems the suggestion is to reinstall.

csul · April 29, 2021, 10:22pm

Hi again,

It looks like my last message was totally wrong. A dutiful user wrote in with the same issue and found that during this stage

Checking for *.log and *.pid for a running server

Remote SSH runs ps ax but due to a load issue, it would hang without resolution. This should be working on discovery2.usc.edu now and we are working to get it fixed on discovery1.usc.edu. For now please use discovery2.

Let me know if you still encounter problems

taoanhua · May 2, 2021, 3:47pm

Thanks, I can connect to discovery2 now.

hprajapa · July 22, 2022, 10:27pm

I am facing this issue since a day. I have tried running the command “Kill VS Code Server on Host…” But it’s still not working. I am working on Windows, VS Code 1.69.2. I am unable to connect to discovery server (can connect from windows terminal), it keeps prompting for password. However I am able to connect to other MCL lab server(USC). Any help?

dongzeye · July 23, 2022, 12:35am

Same here… dicovery2 also doesn’t work for me and it’s stuck at

Checking for *.log and *.pid for a running server

when connecting through VSCode Remote-SSH.

There’s no problem the regular ssh connection on terminal though.

sahetya · July 26, 2022, 11:49pm

same issue here

dstrong · July 27, 2022, 12:27am

@hprajapa @dongzeye @sahetya Please try using the SSH FS extension instead:

It has similar functionality but seems to work better with CARC systems.

sahetya · July 27, 2022, 1:01am

is the configuration similar to remote-ssh?

dstrong · July 27, 2022, 1:07am

It has similar configuration options, but there may be a different set of options. You can specify hostname, SSH key, default directory, etc.

hprajapa · July 29, 2022, 1:41am

Mine did not work for like 3-4 days. Then suddenly started working. Today again met with the same issue. Tried the solution given by dstrong. With that it’s working for now.

jh_445 · September 11, 2022, 6:54pm

Hi, I’m running into the exact same issue after the downtime maintenance on Friday. Using the SSH FS extension does not seem to help either.

dongzeye · September 12, 2022, 6:01pm

I’m having the same problem. Also, there’re some issues with direct SSH connections to discovery.usc.edu (as well as discovery1/discovery2):

The connection to hpc-transfer node is fine though.

dstrong · September 12, 2022, 8:13pm

We’re trying to limit processes on the login nodes, so this is causing the issues you are seeing such as blocking logins. The Remote SSH extension spawns a lot of process threads on the login nodes and they seem to stay active even after logging out from the Remote SSH session. The SSH FS extension should be fine, but your other processes running on the login nodes may block the login for this extension.

@dongzeye We stopped your vscode-server processes on the login nodes. So you should be able to log in again. I recommend removing the Remote SSH extension (and ~/.vscode-server directory in your /home1 directory) and switching to SSH FS extension.

dongzeye · September 12, 2022, 8:50pm

Sounds good. Thanks a lot for your speedy response!

jh_445 · September 13, 2022, 5:02am

Is there a USC CARC-specific guide anywhere to using the SSH FS extension? With this extension, there are a lot of possible configurations and it’s very, very easy to dive into a deep rabbit hole trying to set it up (and therefore wasting a lot of unnecessary time). Thanks!

dstrong · September 13, 2022, 7:10pm

The SSH FS settings to modify are the following:

Host: a login or transfer node (e.g., discovery1.usc.edu)
Port: 22
Root: path to directory for project
Username: USCNetID
Password: USCNetID password (if you’ve not set up SSH keys)
Private key: path to private key (if you’ve set up SSH keys)

tianhewa · September 14, 2022, 6:29pm

Same issue here… Was unable to login after the maintenance on both SSH-FS and terminal.

hprajapa · September 25, 2022, 2:00am

@dstrong , I have started facing the issues again. Not able to connect to host using either VSCODE terminal (through SSH FS) or windows terminal. Can you please check? I am only able to edit my files, but not do any command line operations from terminal except moving from one directory to other.
On windows terminal: Terminal snippet ( am not able to post the screenshot here)
hprajapa@discovery.usc.edu’s password:
Connection to discovery.usc.edu closed by remote host.
Connection to discovery.usc.edu closed.

SSH FS (VSCode terminal):
[hprajapa@discovery1 hardik]$ salloc --time=0:30:00 --mem=50GB --partition=largemem
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: Resource temporarily unavailable
bash: [: : integer expression expected
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: Resource temporarily unavailable