I am using tmux to launch multiple terminals at backend in discovery1.usc.edu.
But I found that my tmux session would be cleaned periodically. I understand that the tmux info is stored under /scratch1/zjin8285/tmux/discovery1.usc.edu. However, I am still not very clear why this folder is cleaned by itself.
Thanks.
Are you on discovery2? You would have to be on discovery1 to access existing tmux sessions on discovery1. Also, the login nodes may be rebooted occasionally, which would clear your tmux sessions, but you could use a plugin like tmux-resurrect to persist sessions across reboots.
Thanks for your reply. I am always using discovery1 to access the tmux.
My discovery2 is only for VSCode SSH FS. I mainly use terminal to access discovery1.
I also notice that discovery1 keep running for almost 63 days without reboot at all. Actually, around every 6 hours, my tmux session will be killed automatically.
I don’t think there’s any policy that would be stopping tmux sessions automatically, but I’ll look into it. There is a process limit on the login nodes but that would only block additional processes and not stop existing ones.
This phenomenon happened again this afternoon at around 17:40.
At that time, I am using scp to transfer huge amount of files from another machine to discovery1.usc.edu under /project. And I found that there is an error message
acket_write_wait: Connection to 10.72.0.13 port 22: Broken pipe, lost connection
And at the same time, my tmux session is killed in discovery1.usc.edu. In my point of view, tmux is not related with scp lost connection at all, unless the login node is rebooted. However, I run the last -x reboot and it does not show today
wtmp begins Thu Feb 1 04:45:29 2024
So I am quite kind of confusing what happened.
I also have this issue! I am working around the drops by not keeping anything running between logging in, which has been working, but it would be convenient to be able to maintain a workspace between logins.
The broken pipe error typically means the VPN client disconnected. You could try modifying your SSH config to minimize the issue:
Also, we have dedicated data transfer nodes that I recommend using for large data transfers instead of the login nodes: hpc-transfer1.usc.edu and hpc-transfer2.usc.edu.
Thanks for your reply. I don’t think the disconnection is a VPN issue. Because I am located at USC ICT, I don’t need the USC VPN to connect to the CARC.
If I may go out on a limb, I think might be that the tmux persistent state files get wiped on logout, or the tmux process gets killed by the server due to the constraint disallowing long running processes on the discovery login nodes
I’m able to keep a tmux session going for several days on discovery1, even if a disconnect occurs or sometimes in a detached state. Ultimately I’m not able to reproduce the issue.
Try running tmux with logging and see if there are any hints in the logs the next time it happens:
tmux -vv
Not sure if this is a separate issue, but it looks like my tmux session persisted, I just can’t attach to it.
Hi all,
Double check to make sure you are using tmux provided from the module system (for example, /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/tmux-3.3a-m5kg7vt/bin/tmux). I have a couple sessions from a few days ago that I can attach to without issue.
$ tmux ls
0: 1 windows (created Thu Feb 15 08:27:48 2024)
1: 1 windows (created Tue Feb 20 08:23:36 2024)
If you use /usr/bin/tmux, session info is stored in /tmp, which gets cleared out periodically.
I am currently using /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/tmux-3.3a-m5kg7vt/bin/tmux.
And last night, my tmux session has been killed automatically. So here are the logs from tmux -vv.
I cannot appreciate it more if you could take a look. Notice that the last file is super large, roughly 2.2GB, it might take a while for the vim to load.
Sorry, I don’t see any hints in the log files.
My tmux session is killed every night. So if tmux log cannot give any hints, what else i can do to help you locate the problem?
Try stopping all your processes on discovery1, open a single tmux session with nothing running in it (just bash, not zsh), detach from it, and see if it gets killed over night. If it doesn’t get killed in that case, then I think that suggests that when it does get killed it’s caused by some process that you are running at the time or the process limits.
As you suggested, I leave a tmux session over night with eight bash windows. Amazingly, it was not killed.
I found that the problem maybe is the htop.
When I open htop inside tmux session, the session would be definitely killed automatically. But if I did not open htop, everything is fine.