New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
SolusVM node SSH hangs
Hi - I am having some trouble with SSH connections hanging on my SolusVM node (OpenVZ). I am able to SSH into the system, but my connection freezes (does not terminate, just hangs). All updates have been applied. I don't particularly have a large load on the machine. I have been monitoring VPS loads, disk, CPU, and network via SolusVM - and there is nothing unusual. No spikes of any kind. My network provider does not report any spikes in network activity as well.
It is a Dell machine - Drac reports no hardware issues. I am able to stay connected via the Drac Java application for a little bit longer. Is this likely to be network provider issue? Any suggestions would be most appreciated.
Thank you all in advance!
Comments
Does SSH hang after a while of being connected or immediately after connecting?
Thank you. Varies - sometimes within a minute or sometimes after 3-4 minutes.
If it happens while you're actively using SSH then it sounds like a network issue. If so, have you checked for packet loss?
Have you checked the logs?
Yes, nothing unusual. Another unusual thing - if I keep 'top' running - I stay connected. The connection seems to freeze the moment I step away from the console with no activity.
Checked for packet loss - no issues on that - 0% loss.
Are you using OVH? Their VAC kills idle SSH connections if there's no explicit timeout configured in the SSH client.
No - this is QuadraNET. I changed the ClientAliveInterval to 600. Will reboot and report on results. Thanks all.
Run tcpdump restricted to capture traffic from your IP or ssh only, capture to a pcap file wait for it to happen, stop tcpdump, download the file for review in wireshark.
Check the file line by line +- 15 seconds of the time you had the issue, if you see absolutely no issues at all and this does not happen to you in other ssh sessions then I think we need to know a lot more about your hardware setup to diagnose.
I have seen similar issues when an SSD disk running flashcache was failing.
Ah, I have that same issue with some ddos protected networks I've used. Enable keep alive in your SSH client and it should stop you being disconnected for inactivity when not in something like 'top'.
This is just a short excerpt of the dump file:
09:08:46.399346 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4885988, win 12327, length 0
09:08:46.411686 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4885988:4887348, ack 26656, win 32765, length 1360
09:08:46.425544 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4887348:4888708, ack 26656, win 32765, length 1360
09:08:46.425556 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4888708, win 12327, length 0
09:08:46.437909 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4888708:4890068, ack 26656, win 32765, length 1360
09:08:46.451473 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4890068:4891428, ack 26656, win 32765, length 1360
09:08:46.451485 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4891428, win 12327, length 0
09:08:46.461702 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4891428:4892788, ack 26656, win 32765, length 1360
09:08:46.473465 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4892788:4894148, ack 26656, win 32765, length 1360
09:08:46.473476 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4894148, win 12327, length 0
09:08:46.473783 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [P.], seq 26960:27040, ack 4894148, win 12327, length 80
09:08:46.486398 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4894148:4895508, ack 26656, win 32765, length 1360
09:08:46.496922 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4895508:4896868, ack 26736, win 32764, length 1360
09:08:46.496938 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4896868, win 12327, length 0
09:08:46.520289 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4896868:4898228, ack 26736, win 32764, length 1360
09:08:46.531782 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4898228:4899588, ack 26736, win 32764, length 1360
09:08:46.531793 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4899588, win 12327, length 0
09:08:46.545169 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4899588:4900948, ack 26736, win 32764, length 1360
09:08:46.553956 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4900948:4902308, ack 26736, win 32764, length 1360
09:08:46.553971 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4902308, win 12327, length 0
09:08:46.566381 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4902308:4903668, ack 26736, win 32764, length 1360
09:08:46.577303 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4903668:4905028, ack 26736, win 32764, length 1360
09:08:46.577315 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4905028, win 12327, length 0
09:08:46.589460 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4905028:4906388, ack 26736, win 32764, length 1360
09:08:46.605412 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [P.], seq 4906388:4907748, ack 26736, win 32764, length 1360
09:08:46.605424 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4907748, win 12327, length 0
09:08:46.613413 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4907748:4909108, ack 26736, win 32764, length 1360
09:08:46.625389 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [P.], seq 4909108:4910468, ack 26736, win 32764, length 1360
09:08:46.625400 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4910468, win 12327, length 0
09:08:46.625679 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [P.], seq 27040:27120, ack 4910468, win 12327, length 80
09:08:46.654300 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4910468:4911828, ack 26736, win 32764, length 1360
09:08:46.665083 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4911828:4913188, ack 26736, win 32764, length 1360
09:08:46.665097 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4913188, win 12327, length 0
09:08:46.676451 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4913188:4914548, ack 26736, win 32764, length 1360
09:08:46.687733 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], ack 26816, win 32764, length 0
09:08:46.688493 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4914548:4915908, ack 26816, win 32764, length 1360
09:08:46.688506 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4915908, win 12327, length 0
09:08:46.700361 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4915908:4917268, ack 26816, win 32764, length 1360
09:08:46.712272 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4917268:4918628, ack 26816, win 32764, length 1360
09:08:46.712286 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4918628, win 12327, length 0
09:08:46.724401 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4918628:4919988, ack 26816, win 32764, length 1360
09:08:46.736468 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4919988:4921348, ack 26816, win 32764, length 1360
09:08:46.736482 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4921348, win 12327, length 0
09:08:46.748417 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4921348:4922708, ack 26816, win 32764, length 1360
09:08:46.760460 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4922708:4924068, ack 26816, win 32764, length 1360
09:08:46.760474 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4924068, win 12327, length 0
09:08:46.772364 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4924068:4925428, ack 26816, win 32764, length 1360
09:08:46.783427 IP 14.102.69.106.61996 > mta0.loginsystemhost.net.ssh: Flags [.], seq 4925428:4926788, ack 26816, win 32764, length 1360
09:08:46.783441 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [.], ack 4926788, win 12327, length 0
09:08:46.783819 IP mta0.loginsystemhost.net.ssh > 14.102.69.106.61996: Flags [P.], seq 27120:27200, ack 4926788, win 12327, length 80
Compromised VPS?
Thanks SpartanHost - I will try that and report back.
Chetan
@SpartanHost - that did it! I get kicked out when using terminal on my Mac, but Putty with KeepAlive seems to work just fine. My client also reset the VPS for which the tcpdump is posted above.
Glad to hear it worked
Yes, me too! Thank you all so much for your assistance and quick responses. I truly appreciate it!