New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HostHatch LA network issues after maintenance
HostHatch had some scheduled maintenance in Los Angeles yesterday/today and now I'm experiencing a lot of network issues. I can ping my servers from everywhere, but I can't connect to them from some locations at all. I can connect fine from a server in Australia, but I can't connect from my home server in the San Francisco Bay Area nor a GreenCloud VPS in San Jose. About half my backups are failing because they can't connect to my storage VPS. Cloudflare can't connect so all my sites that are fronted by Cloudflare (to block DDoS attacks) are down.
Is anyone else experiencing this? I filed a ticket but I know they take a while to reply to tickets.
Comments
Same here. All my servers are throwing connecting timeout. Disk operation and latency seem to be fine.
Private networking was broken after maintenance but the problem has resolved very soon as soon as I opened a ticket. Currently Some of vm's Ipv6 are broken. I have no connection issues.
edit) I can't connect to a hosthatch storage vm from a Greencloud VPS in San Jose.
I assume outbound connections(except ipv6) has no problem but inbound is something wrong.
It is Http Connection to Cloudflare from VMs for heartbeat.
I'm having issues with outbound connectivity too, for example I can't load anything from raw.githubusercontent.com on any of my Hosthatch LA VPSes.
I also have an LA storage VPS. Until recently, I couldn’t access the server or the looking glass from only my home wifi. After contacting support, they changed the routing from Quadranet to CDN77, which fixed it. Maybe this is related?
@hosthatch
He won't comment here unless you troll him.
They moved facilities, might just be some temporary network weirdness while stuff readjusts.
Did they announce this somewhere? The email just said it was "maintenance".
We have/had scheduled maintenances on all LA nodes from 25th to 28th February. If you see any issues with your service before or after the maintenance, please open a ticket and we'll get it resolved for you. All of our staff is on hand, so you will get a fairly quick response.
We're moving out from QuadraNet DTLA to Digital Realty LAX10, but other than a few network blips during this process, you will not see any difference (in terms of IPs, configuration, data on your servers, etc). Most of the servers are already completed.
The "temporary network weirdness" is a lower MTU. Now it's 1472 for some routes, meanwhile 1500 for others, hence the partial connectivity issue.
You have to manually set the MTU or use the TCPMSS firewall extension "to overcome criminally braindead ISPs or servers which block "ICMP Fragmentation Needed" or "ICMPv6 Packet Too Big" packets" (quoted from the manpage).
That's what Able told me in my support ticket - the MTU was misconfigured.
It seems to mostly be okay now, but some uptime monitors keep flapping every 15-20 mins. Maybe some bad routes? I'm on vacation so haven't had a chance to have a closer look yet.
--
Both of these should be addressed now. Please follow up in your ticket if you continue seeing any issues.
Yes, it seems resolved now. I'd keep the MTU lowered for a while though. I vaguely recall that the same issue occurred a few years ago after maintenance, so it might be recurring.
Edit: In a ticket dated 1st February 2023, I reported that the MTU was 1476. The MTU issue was met with silence so I'm not sure when it got fixed.
This is no longer needed as the network migration has been largely completed. Thank you for being with us for many years.