All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
The IPv6 outage thread - Post-Black Friday edition 🛍️
It's a new month and therefore it's time for the monthly IPv6 outage rollup. :tada:
Black Friday and Cyber Monday are over and although @yoursunny is one of my idols, I do not yet have a numbering scheme which imposes limits on the number of active services I can have at the same time. Therefore, the inevitable has happened and I have picked up 35 VPS, two /44 IPv6 PA Subnets, a Seedbox (which sadly does not support IPv6) and had to call my credit card company twice to confirm transactions and have my card unlocked again. Some of these VPS will replace existing services, so it's not that bad (at least that's what I keep telling myself).
To my great delight, most of the VPS services were provisioned with IPv6 subnets (or single addresses), so that I only had to open two tickets to have IPv6 enabled. And I have one ticket open... but, let's get started:
🤷 OBHost – My Black Friday promo plan was provisioned on a node which apparently has no (or a wrong) IPv6 default route set. I'm apparently not the only person who stumbled upon that issue, so let's wait and see.
✅ GreenCloudVPS – Their upstream had two IPv6 outages in Vietnam which were resolved within two hours each.
HostMaze – To my surprise, they didn't have a single outage this month. Congratulations!
😞 Friendhosting – They had a routing issue between their two datacenters in Ukraine. Instead of spinning up a clean VM to confirm the issue, they went straight to asking me for my root credentials. Loyal readers of the IPv6 outage threads know that I strongly dislike that behavior. We had to go back and forth a bit until they were able to login to my servers, run some pings on their own, fiddle around with my iptables settings and forward the issue to their datacenter eventually. It was resolved on the next day.
✅ FreeRangeCloud – Short IPv6 outage in Ashburn due to upstream issues which got resolved in less than an hour after opening a ticket.
✅ WebHorizon – Short IPv6 outage in Los Angeles which got resolved within 12 hours after opening a ticket.
✅ Zappie Host – IPv6 outage in Auckland due to upstream issues which they fixed by re-routing traffic through their secondary upstream. Adam handled the situation quickly and provided me with extensive information on the issue and why their automatic failover didn't work in this case. :thumbsup:
✅ WebHorizon – Another IPv6 outage in Singapore which took a bit longer to resolve than the previous one, but still within limits.
🤷♂️ DediPath – After an outage in Las Vegas, my server came back up but IPv6 connectivity was interrupted. I sent them the output of an unsuccessful ping6 and they responded with a screenshot of PuTTY connecting to my server over IPv4 and claimed everything was fine. When I insisted on them checking IPv6 connectivity as well, the issue was resolved few hours later.
🔻 G-Core Labs – I opened a ticket about an IPv6 outage in Vladivostok and provided them with a traceroute showing that traffic gets lost after their gateway. They responded asking me to reinstall the server or check my network settings. Thanks for nothing. I had a hard time convincing them that there's an issue on their end, but they were able to fix the issue two days later.
All in all, it's been an uneventful month IPv6 outage-wise. So let's spice things up a bit. Here's the...
⚡ IPv6 outage flashback ⚡
🔻 Time4VPS – I know you like providers snooping in other people's gardens, so here's my story: I bought a decent OpenVZ 6 VPS back in August. Keep in mind that OpenVZ containers are very limited when it comes to modifying networking settings. Pretty much all settings need to be done on the host.
When I received the server, I had an IPv6 address configured on my venet interface but didn't have any IPv6 connectivity to the world outside. When I ran a traceroute, the first (or second? I don't exactly remember...) responded with "No route":
PING google.com(waw07s06-in-x0e.1e100.net (2a00:1450:401b:80e::200e)) 56 data bytes
From k2-b14-lt1.kvm.serveriai.lt (2a02:7b40:1f0e:b34a::1) icmp_seq=1 Destination unreachable: No route
I opened a ticket, but instead of checking the information I provided to them, they decided to check my .bash_history instead:
However, it seems you have adjusted the server's configuration so that this resulted in the server being unable to PING google.com via IPv6. With the default configuration, it's working correctly. We can see changes made to your VPS (configuring firewall, installing/uninstalling packets). One of these changes has resulted in the issue you contacted our regards. Our administrator has tried to reset the routing on your server to default; however, that did not help. Since the server is self-managed, further troubleshoot, and solutions need to be implemented yourself. You can also re-install the server. Have a nice day.
I was baffled. What they've seen in the history is my poor-man's Ansible doing some basic provisioning (hardening, firewall settings, installing some utility packages, configuring backup + monitoring). None of these commands touch any network settings at all.
When I recovered, I decided against asking for a refund and tried to point them in the right direction instead. But it seemed hopeless:
No issues are captured with the node where your VPS is established or our network. As mentioned, such behavior is the result of the server's configuration. To solve the issue, you can re-install the OS and use the default server's configuration.
Fair enough. I did a reinstall and surprisingly IPv6 still wasn't operational. I updated the ticket and they finally seem to have woken up:
I have passed this to our administrator to check. Your server was established on the new node, where IPv6 has not been fully activated yet. The necessary changes have been applied, and the issue is fixed now.
I couldn't resist to ask if the commands I ran on the server had any impact on IPv6 connectivity, and...
Yes, the problem was with the node where your server is located.
We checked your request with the administrators both times. We saw changes made to the server in the first case, so we assumed it was a configuration issue. This trouble could not be replicated on our test servers.
Comments
openvz 6 is EOL, such providers should be avoided.
Thats an odd reason i think
Put all people to node which has no IPv6
Now they promised to recreate my vps with IPv6 working node , i am afraid they will put it to their full node
Now i think consider to request a refund
Or give them option to exchange it to their KVM line with reduced resources
I don't really want to waiting too long
I think you should start tracking those support responses and have totally new "black" list
My 5 cents:
🔻 leapswitch:
2021-11-20 Ordered VPS within Mumbai, IN location. They offered /64 IPV6
After i received the server i've found that VPS have 7 ipv6 but there is no route (destination unreachable).
I've opened ticket and also asks them to make routed /64 subnet to my VPS. I've receiving response like "Our Network team will check about this and update you." but it lasts for 2 weeks now and their network team still checking. I've pinging support each 3 day, but answer always the same.
There was nothing like that - The IPv6 problem occurred because of upstream.
Users get VPS based on available resources and On BF/CM we already running promotion in Google Ads and other social platforms so we need to manage things accordingly.
I received an update from @OBHost earlier today and was told that their upstream still hasn't fixed the IPv6 issues.
Come on, @combahton_it - you can do better!
They are trying their best to fix the problem from 2nd December.. Maybe they are introducing some new ways to announce the IPv6 subnets..
My experience: I picked up two Racknerd boxes in Los Angeles, asked support to provision them both with IPv6. Neither had functional IPv6 after that. I was able to disable router advertisements (
net.ipv6.conf.eth0.accept_ra=0) and maybe do some other stuff which fixed it.AFAICT the router was telling my boxes they had a /64 subnet when they absolutely did not, so the boxes were setting up random addresses that didn't work. I don't know much about networking though.
They work now, so maybe if they ever break posting this comment will let me remember how I fixed them.
edit: Oh yeah, they did have IPv6 connectivity - for about a minute or two after rebooting. Then they'd get a new address which wouldn't work.
Not a configuration issue on our side. The customer is using a subnet from a different v-lan than the server he is deploying VPS from. Unfortunately, we do only have control about our network infrastructure - it's not up to us, to manage the assignment of ip-addresses or make sure the configuration is in a well managed state
Oh, boy. In that case, apologies for calling you in.
Well, my Provider forgot that it had some Subnets revoked.
IPv6 did in the sudden, suddenly.
According to the ticket reply, they where confused at first, until they noticed their own fuckup, kurwa.
One can bring Latvia out of the Soviet Union, but cannot bring Soviet Union (and its hallmark "customer service" traditions) out of Latvia.
Time4VPS is a Lithuanian company. in any case, in Lithuania, that sort of statement will automatically qualify you to receive a free punch in the face (even if you were genuinely talking about Latvia). I advise you not to make wild generalizations.
PS: I'm in no way defending the customer support for Time4VPS, which seems to be "lacking", to say the least.
Oh, I'm well aware that "Soviet" is their trigger word. But they deserve that such opinions persist, for as long as there are grounds for that. And the described customer support interaction only happens to reinforce those.
These series of posts are very interesting and not only for the IPv6 part.
They provide excellent insight into the quality of support when something goes wrong.