New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hosthatch Sydney: IPv6 "broken" for two weeks and counting
About two weeks ago Hosthatch announced they were investigating networking issues in Sydney. Since then their IPv6 acts somewhat strange in that location: I can reach my server fine from some locations, but in most cases it seems like their subnet is not announced at all.
Example from Hetzner's network:
1. AS24940 2a01:4f8:211:1443::2 0.0% 10 0.4 0.5 0.3 0.6 0.0
2. AS24940 core23.fsn1.hetzner.com 30.0% 10 6.4 14.7 6.4 28.9 7.6
3. AS24940 2a01:4f8:0:3::c5 0.0% 10 39.3 13.4 3.2 39.3 11.3
4. AS24940 2a01:4f8:0:3::2a9 0.0% 10 3.5 6.9 3.4 24.9 7.5
5. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
And in some cases it even looks like this:
1. AS6939 2604:a100:100:2::1 0.0% 10 0.2 0.9 0.2 6.9 2.0
2. AS6939 2604:a100:100::1 0.0% 10 0.6 0.8 0.5 1.1 0.0
3. AS6939 ve1325.core2.fmt2.he.net 0.0% 10 0.5 0.6 0.5 0.7 0.0
4. AS6939 100ge4-1.core4.fmt2.he.net 0.0% 10 0.6 2.7 0.6 21.0 6.4
5. AS6939 100ge10-1.core1.sjc1.he.net 0.0% 10 1.2 1.0 0.8 1.2 0.0
6. AS6939 e0-36.core2.sjc1.he.net 0.0% 10 1.5 2.3 1.5 7.0 1.5
7. AS6939 e0-36.core2.syd1.he.net 0.0% 10 160.8 165.9 160.4 185.2 7.5
8. AS??? 64067.syd.equinix.com 0.0% 10 155.8 155.9 155.8 156.2 0.0
9. AS??? 2400:b640:0:3::1 10.0% 10 160.9 156.8 156.0 160.9 1.5
10. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
RIPE reports 63% visibility for their subnet.
Now I'm wondering what might be going on here. Why do some networks pick up routes for their subnet and others don't? Just a misconfiguration? Bad BGP communities? What do you think?
And yes, I've opened a ticket with them and they're working on the issue. But I'm still curious what might be the cause.
Comments
I had talked about exactly this, IPv6 routing issues, which took weeks to get fixed.
Open a ticket you likely get it solved in a few weeks.
I've had lots of IPv6 issues with my storage vps in Stockholm as well
Good luck waiting for a fix guys..
That's interesting that multiple people in totally different datacenters seem to have such issues. When I tried HostHatch more than a year ago, in my case it was in London, they also had some weird IPv6 routing and IIRC no connectivity to DTAG. It took some weeks until they were able to fixed it, but it left some aftertaste and I decided not to renew.
Maybe some transit or datacenter providers are enforcing ROA now.
Or the return path is broken. The way back might be totally different and some IP space starting at the ??? might be filtered there.
Many possibilities. If I remember my experience in London correctly, there was something messed up with M247 which HostHatch seems to use in multiple locations.
I wanted to broadcast my IPv6 in Hosthatch and no one replied to my work order for over a month, just asked me to send an email to their admin email address, then I wondered if the network segment I asked to broadcast was too big, so I closed the work order again and reopened it, affirming a paragraph/48, but still no reply, but the problem was solved quickly outside of BGP.
This is an upstream issue that we're working hard to get fully resolved asap.
In some locations, we're replacing our network kits (and upgrading all nodes with 40g) and BGP announcements are on hold until we get that done.
Do you have a rough timeline?
You guys do need to work on your communication. Support rarely provides updates, even when I want to offer more money like trying to migrate to a two year plan instead of one.
It's a shame because the VPS otherwise has great performance and I have a product in mind for which I'd like to use more of your VPSes (at their regular pricing, FWIW), but I can't because I'm scared about issues like these.
It's been another week without any response to my ticket. The issue is not resolved still and in total it's about a month of downtime now:
I used to be a big fan of Hosthatch and so far I have had really good experiences with their support. And the servers I have with them are running just fine.
However, the fact that they're unable to fix (or work with their upstream to fix) an issue most likely affecting a whole location does not create any trust at all.
Still no replies to my ticket but the issue seems to have been resolved today.
Took them only 6 weeks...
IPv6 is ghetto IPv4 - @yoursunny are you triggered? :-)
The same, constantly. duckduckgo.com don't load, lol :-) and other sites
I noticed this too last week but haven't been monitoring it since then to comment if it's stable or not.
Well, it's not. Their prefix disappeared from the DFZ again
Mine was working fine, but it just broke today. My uptime monitoring shows around 4 hours of downtme for both IPv4 and IPv6 from 1:15 AM until 5:17 AM Pacific Time (UTC-7), after which IPv4 connectivity was restored but IPv6 remained down, and it's still down now (10:24 PM Pacific)
Mine been working fine but just broke yesterday. I had to open a ticket and its still being looked at...
@hosthatch Pls fix it
Time to leave the ship...
We’re working on getting it resolved with our upstreams in Sydney. If you have an open ticket, we’ll update you there once we make progress.
Are your SG location ready?
Do you have any SLA for network outages like this one? Not interested in compensation or anything like that (your sale services are already very very cheap), just wondering if you have a minimum expected level of network uptime. 😊
I guess this is an upstream outage so probably wouldn't fall under an SLA anyways?
I have an issue with my IPv6 in HostHatch HKG, I have a ticket open for more than a month, no response, made me a little bit sad
Anyway, here's the issue. If anyone has an idea if it's a configuration issue on my end, or if it's an issue with HostHatch please chime in!
The issue is that, there is heavy packet loss observed in traceroute. But it's not like classic packet loss that a percentage of packets get dropped. Instead, the traceroute looks good in the beginning, then ~20 seconds in, the packets stopped being forwarded.
Video: https://imgur.com/PVWBojO
Sorry, I missed this before. I am assuming you asked for an upgrade with free disk space, which is why it would be on pause. There should be no delays at all if you want to simply change from one to two-year billing without any changes to the server.
We have a status page at https://status.hosthatch.com/
We generally aim for 99.9% uptime - and beat that year after year, but Sydney has been a problem in the past couple of months.
We will likely stop providing native IPv6 there for now (and in some other problem locations). Most upstreams do not deal with IPv6 issues with the same priority, and the time investment that goes into troubleshooting IPv6-related issues in these locations for a -very- small number of customers who actually use it is huge.
Received via email:
@hosthatch which network provider are you using in Sydney? I've got a few VPSes in Sydney with various providers and IPv6 works fine with all of them, so I wonder if you just got unlucky with your choice of provider
An upstream provider with flaky IPv6 in 2021 is definitely a bad sign.
Which are those "others"?
No IPv6 hall of shame
Include IPv6 for no extra cost on every plan to get delisted.
If you want premium Sydney VPS with routed IPv6, try 👉 Webhosting24 Sydney NVMe VPS 👈.
They resolve IPv6 problems within hours instead of months, and it's routed IPv6 so that you don't need the NDP responder madness; quality comes with a price.
Notice: paid link above, non-aff.
Disappointing, I must be in the less than 3% of services that use it.
I also noticed after their 4 hour outage last week that the latency from my US services have jumped from ~140ms to 210ms. I use a number of other providers in Sydney and they all sit around ~140ms from LA.
FYI, email went to gmail spam folder.
Please ticket that in with a traceroute, we will get it fixed.
As for the guy plugging in his affiliate link….nice.
Wow, yeah, good catch. I just noticed the same thing too. Ping to my Sydney Hosthatch VPS is really high now - 270ms from around the San Francisco area, whereas I get 160ms to my VPS hosted with QuantumCore (who use RansomIT AS136557 as their network provider, who use ColoAU AS63956 as their upstream).
It looks like it transits from Telia (AS1299) in the US to Level3 (AS3356) in Sydney and that link has high latency.
vs the routing with QuantumCore: