All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HostHatch Los Angeles storage VPS - Network issues?
I've got an open support ticket, but I'm wondering if anyone else is experiencing this, or if I'm just unlucky. I noticed an issue maybe a week ago and thought it might have been on my end. Tried tweaking the network config, etc and nothing worked. Opened a support ticket on December 5.
My storage VPS in LA has been working well over the past year, no network issues at all (really happy about that!), but recently the network has gotten a lot slower, just on the storage VPS. An NVMe VPS in the same location is fine.
Speedtest on Storage VPS, using Speedtest.net CLI
Server: Airlink Internet Inc - Los Angeles, CA (id = 33893)
ISP: HostHatch
Latency: 0.87 ms (8.50 ms jitter)
Download: 50.79 Mbps (data used: 77.0 MB )
Upload: 128.32 Mbps (data used: 206.2 MB )
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/fc548300-bbf2-4a23-a3dd-849f4152e76d
Speedtest on NVMe VPS, around the same time (~30 seconds later), in the same data center, to the same speedtest server:
Server: Airlink Internet Inc - Los Angeles, CA (id = 33893)
ISP: HostHatch
Latency: 0.56 ms (1.05 ms jitter)
Download: 909.67 Mbps (data used: 410.1 MB)
Upload: 1743.40 Mbps (data used: 1.7 GB)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/9f85855a-a940-47d8-9091-231beabdc291
It's very inconsistent though - It's fine at some times, but really bad (like 200+ms ping and <1MB/s download speed from Los Angeles to Las Vegas) at other times.
I thought it might be internet congestion, but the NVMe VPS is fine (which I assume uses the same upstream), and but earlier today I was seeing <40Mbps even over the internal network between the two VPSes, which is usually ~5-6Gbps. ¯\_(ツ)_/¯
It's not any sort of CPU throttling either... CPU usage is usually <2% on the storage VPS, except for iowait when reading from disk (which wasn't happening when I was running the speedtest).
I store backups on this system using Borgbackup over SSH... The backups used to be very fast, but now they're so slow that they hit the issue with long-running SSH connections disconnecting / timing out (blaming Psychz for that, not Hosthatch, as I've seen it on other Psychz servers too) 
Comments
We must be neighbors. I opened a ticket a week ago for the same thing because it appears to be an internal issue only affecting that server.
I setup a monitor just to make sure I wasn't crazy. It runs 4 pings every 15 seconds and logs the average response time. Here is the last 12 hours. The polling server is a Host Hatch NVMe server that has no issues with any other host except this one.
I have a another LAX storage service on a different server and it has no issues.
The first time I see this name Borgbackup, cool application. I used to tar the doc or dir, and transfer them to the backup server via SSH.
can you check your IOWait when sending file?
Here is a 12 hour window matching my graph above.
50% iowait is not uncommon. My Chicago instance hit 45% during last night's backup.
I should note that iowait has never been an issue. It’s spinning rust running backups so it doesn’t matter to me.
@cablepick Glad to hear that it's not just me.
iowait is definitely high sometimes, which is somewhat expected with regular HDDs. This was when reading a bunch of ~10MB files:
This is going back one week:
Same for me. my vm is losing connection through private lan because of the network congestion. https://i.imgur.com/c7Miw6m.png
Seems like we’re on the same node! Apologies for the formatting, I’m on a phone:
Hello,
We are emailing you because we have upcoming emergency maintenance on the host node that hosts your Storage Virtual Machine(s) in Los Angeles.
The following time window has been scheduled.
Friday, 11th December 2021
Window start: 2:00 PM PST
Window end: 3:00 PM PST
Total expected downtime: <30 minutes
Your server will be shut down and booted up after the maintenance has been completed. We do not expect any configuration or data on your server to change, but as a best practice, we always recommend that you keep the latest copy of your backups.
During this maintenance, we will replace a faulty network interface card with a new one.
If you have any questions, please do not hesitate to reach out.
Kindest Regards,
Your HostHatch team
I just got the same email. Glad to know they're fixing it, but Friday is the 10th December, not the 11th. As I write this, it's 12:05 PM PST. Did they mean Friday (i.e. today in two hours), or Saturday? 🤔
Guess its today!
Maintenance window ended 23 minutes ago but it's still down
Good example of high load average but low CPU usage... I have data on the storage VPS mounted over NFS, and a bunch of processes on my NVMe VPS are blocked waiting for the NFS server to come back online.
It's back! And the network is fixed!
/thread
Just powered on my box.. still not able to connect via novnc
There was a one and half period of downtime, the solid red period starting at 14:00, and it looked better right after but... This is the previous 24 hours:
Edit: and for a better sense of scale here is the past hour:
Oh you're right... It looks kinda broken again. Speedtests are slow and pings are fluctuating again
There was a reboot this afternoon and everything has been normal since. Hopefully its fixed.
So far no one has shown an mtr - the most basic test that should be done when you talk packet loss / high latency.
I didn't do an mtr, but I included several traceroutes in my support ticket. I just forgot to post them here. All the hops were fine except for the last one (my VPS itself).
They replied saying it should be fully fixed now, and I haven't done all the same testing yet, but it does appear to be better now after the second reboot.