New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Your LowEND support ticket created ticket id #6969
What support said?
Thank you for contacted low end support, your ticket has been doubled
Logged a ticket but no response which is odd, normally you have an instant recognition of the ticket via automated email with a reply soon after
It is Sunday. Why are you posting this here?
Btw, i ate a piece of bread 7 minutes ago
1669625294
Okay.
Random, but did you choose Debian 12 for your instance by any chance? I have 2 instances on the same node in SJ. One running Ubuntu 22.04 (which I upgraded to 24.04).. and one I set up for the first time with Debian 12 recently.
I was having a bunch of packet loss issues as well, to the point that it took forever to run
apt upgradefor the first time. After a while of not being able to figure out why -- and knowing Ubuntu was running fine, but since I was planning on moving all my instances to Debian anyway, I decided to try installing Debian 11 and manually upgrading to 12. I wasn't having the same packet loss issues.. so it seems the 12 template might be misconfigured in some way. I didn't really dig too deep.You got 49% what you need 51% for? Greedy fuck
Using Ubuntu 22.04, support initially said it was just my vps on the node having issues then said it was more like network issues, connectivity is better but still some random periods of time with high packet loss
Over the last month i have been having intermittent network issues. Sometimes outages as long as 10 minutes. Did open a ticket but support insisted that it is only my VPS and there is no node issue. Hmm. Did a speedtest recently and i found the connection to southeast asia really bad...
Basic Network Info
Primary Network : IPv4
IPv6 Access : ❌ Offline
IPv4 Access : ✔ Online
ISP : tzulo, inc.
ASN : AS11878 tzulo, inc.
Host : tzulo, inc.
Location : Chicago, Illinois-IL, United States
Location (IPv4) : San Jose, California, US
Speedtest.net (Region: GLOBAL)
Location Latency Loss DL Speed UP Speed Server
ISP: tzulo
Nearest 37.66 ms 0.0% 6525.41 Mbps 2467.90 Mbps Ideatek Telcom - Wichita, KS
Kochi, IN 384.84 ms 0.0% 3131.07 Mbps 266.48 Mbps Asianet Broadband - Cochin
Bangalore, IN 251.89 ms 0.0% 2962.44 Mbps 337.62 Mbps Bharti Airtel Ltd - Bangalore
Chennai, IN 222.07 ms N/A 3625.42 Mbps 388.31 Mbps Jio - Chennai
Mumbai, IN 240.49 ms 0.0% 2989.29 Mbps 229.32 Mbps i3D.net - Mumbai
Delhi, IN 326.53 ms 1.3% 102.83 Mbps 5.78 Mbps Tata Play Fiber - New Delhi
Seattle, US 18.68 ms N/A 5185.06 Mbps 4275.77 Mbps Comcast - Seattle, WA
Los Angeles, US 8.01 ms 0.0% 5729.86 Mbps 8043.09 Mbps ReliableSite Hosting - Los Angeles, CA
Dallas, US 42.04 ms 0.0% 5011.85 Mbps 2083.57 Mbps Hivelocity - Dallas, TX
Miami, US 71.92 ms N/A 2653.56 Mbps 865.59 Mbps Dish Wireless - Miami, FL
New York, US 69.91 ms 0.0% 6185.84 Mbps 1448.28 Mbps GSL Networks - New York, NY
Toronto, CA 74.84 ms 0.0% 5348.72 Mbps 1266.68 Mbps Rogers - Toronto, ON
Mexico City, MX 63.37 ms N/A 6413.44 Mbps 1415.66 Mbps INFINITUM - Mexico City
London, UK 130.28 ms 0.0% 5447.64 Mbps 805.82 Mbps VeloxServ Communications - London
Amsterdam, NL 149.82 ms 0.0% 4960.36 Mbps 391.83 Mbps 31173 Services AB - Amsterdam
Paris, FR 147.25 ms N/A 5134.33 Mbps 360.71 Mbps Axione - Paris
Frankfurt, DE 140.66 ms 0.0% 4639.57 Mbps 519.86 Mbps Clouvider Ltd - Frankfurt am Main
Warsaw, PL 170.61 ms 0.0% 4554.65 Mbps 610.62 Mbps Play - Warszawa
Bucharest, RO 172.40 ms 0.0% 1895.07 Mbps 532.19 Mbps Vodafone Romania Fixed – Bucharest - Bucharest
Moscow, RU 184.41 ms 0.0% 1895.97 Mbps 278.07 Mbps MegaFon - Moscow
Jeddah, SA 194.30 ms 0.0% 2875.91 Mbps 552.43 Mbps Saudi Telecom Company
Dubai, AE 253.28 ms 0.0% 2983.46 Mbps 326.90 Mbps du - Dubai
Fujairah, AE 254.01 ms 0.0% 3047.39 Mbps 393.15 Mbps ETISALAT-UAE - Fujairah
Istanbul, TR 176.11 ms 0.0% 3829.10 Mbps 608.87 Mbps Turkcell - Istanbul
Tehran, IR 237.45 ms 0.0% 2241.15 Mbps 361.91 Mbps Asiatech - Tehran
Tokyo, JP 108.12 ms N/A 5022.61 Mbps 3.26 Mbps fdcservers.net - Tokyo
Shanghai, CU-CN 157.24 ms 0.0% 1440.29 Mbps 333.04 Mbps China Unicom 5G - Shanghai
Nanjing, CT-CN 140.23 ms 4.5% 1952.48 Mbps 3.84 Mbps China Telecom JiangSu 5G - Nanjing
Hong Kong, CN 155.67 ms N/A 4538.94 Mbps 597.96 Mbps STC - Hong Kong
Singapore, SG 176.25 ms 15.7% 3608.58 Mbps 1.20 Mbps i3D.net - Singapore
Jakarta, ID 225.67 ms 12.4% 2048.43 Mbps 0.64 Mbps PT Solnet Indonesia - Jakarta
Maybe just our VPS's on the node
HetrixTools shows mine was down for about four minutes from 2024-05-13 06:45:40 (UTC+00:00) to 06:49:34. Don't know what node I'm on, but my IP is in the 173.249.209.0/24 netblock.
I've also been seeing intermittent network outages over the past two months, most lasting between two and nine minutes. I've reinstalled debian 12 from a debian.org netinst image, so that rules out anything to do with their debian 12 template. mtr shows it's the last hop going offline. (I didn't have mtr checking IPv6 for this last outage, but I do now.)
It would be very interesting if we're on the same node, or if our outages are at the same time.
So.. after multiple tickets and complaints about SJC location they responded by saying this is a problem related to my VPS software stack? Obviously it was a network problem.
Fast forward a month later i get this email
Expected downtime: Less than 15 minutes when the uplink ports are moved over
Of course it was an AH-HAH moment.
So i wrote in telling them about my previous experience and asked them if this was to fix it and this is their answer ;
At least if im wrong tell me im wrong and give a solution. Hopefully this will fix the SJC intermittent network issues.
Shame it takes them more than a month to actually take it seriously.
HetrixTools saw 14 minutes of downtime from 2024-05-18 16:12:43 (UTC+00:00) to 16:26:28. I presume that was the network switch replacement that was scheduled for May 16-17.
Fingers crossed that that's the end of the intermittent network outages!
It's now been about two weeks since the network switch replacement. Has anyone noticed any intermittent network outages? I haven't. I'm hopeful this is now fixed.
And the intermittent outages appear to be back for me. Here's my recent hetrix tools downtime list:
2024-06-23 17:50:34 - 2024-06-23 18:00:34 (UTC+00:00) (10 minutes)
2024-06-23 18:36:37 - 2024-06-23 18:45:34 (UTC+00:00) (9 minutes)
2024-06-30 01:37:37 - 2024-06-30 01:45:34 (UTC+00:00) (8 minutes)
2024-06-30 01:51:37 - 2024-06-30 02:00:38 (UTC+00:00) (9 minutes)
2024-06-30 14:20:40 - 2024-06-30 14:30:37 (UTC+00:00) (10 minutes)
2024-06-30 15:06:39 - 2024-06-30 15:15:31 (UTC+00:00) (9 minutes)
2024-06-30 16:05:41 - 2024-06-30 16:15:34 (UTC+00:00) (10 minutes)
2024-07-01 01:06:40 - 2024-07-01 01:15:34 (UTC+00:00) (9 minutes)
2024-07-01 01:21:37 - 2024-07-01 01:30:34 (UTC+00:00) (9 minutes)
2024-07-01 02:51:40 - 2024-07-01 03:00:34 (UTC+00:00) (9 minutes)
2024-07-01 13:50:34 - 2024-07-01 14:00:37 (UTC+00:00) (10 minutes)
2024-07-01 14:36:36 - 2024-07-01 14:45:34 (UTC+00:00) (9 minutes)
2024-07-01 14:51:43 - 2024-07-01 15:00:34 (UTC+00:00) (9 minutes)
Does this match anyone else's downtime?
mtr shows the last live hop is 104.225.22.63. IPv6 doesn't seem to be affected at all.
I've got mtr logs on the VPS, and when IPv4 goes offline, mtr shows nothing makes it past my interface. (I'm not even seeing their router.)
More t-shooting: IPv6 doesn't go down, so I'm able to run tests when IPv4 goes offline. I had a tcpdump packet capture running before/during/after the latest outage, too. When their network goes down I'm still seeing incoming traffic (such as my ping monitors), but I can't get out. Pinging their gateway router over IPv4 from the VPS fails, yet pinging from the public internet works. The IPv6 and IPv4 gateways have the same MAC address.
If anyone else (@elmorigs, @gowrann, @stxsh ?) is still having the same outage, especially if the time correlate with mine, please open a ticket. They're convinced I'm an idiot and it's my firewall or network config, not anything to do with their network.
Thanks!
edit: if anyone is in the 173.249.209.0/24 subnet and would like to set up some mutual ping monitors, feel free to reach out. Maybe that would show if it's the interface or router that's going offline?
I have two vm in San Jose. They are on the same subnet 173.249.198.0/24. Intel server was unreachable sometimes for 5-10minute(while ipv6 reachable), the other with Ryzen has no problem. I've cancel intel vm because I don't want to open a ticket.
All settings such as Debian are same because I did my initial script. All Greencloud vm are stable except that Intel vm in San Jose.
I ended up moving my web sites to another provider so I am not monitoring it anymore
@pwned after the network switch replacement there was a few outages in June but nothing in July. I share the same frustration you have because the issue isn't on your end and when they refuse to look into it you pretty much have no choice.
Just had another of the 9 minute outages, from 2024-07-22 18:51:37 to 19:00:38 UTC. Hetrixtools shows both IPv4 tcp/80 and icmp were down.
The VPS was not able to access an external website during the outage. It couldn't ping the gateway or external hosts over IPv4. UDP from the VPS to external sites failed, too.
Several external hosts were able to send UDP over IPv4 to the VPS during the outage, which is really strange.
I've now seen these outages with their debian 11 template, their debian 12 template, and debian 12 installed from the netinst ISO.
I'm not sure what else I can do to t-shoot this any further. I'd bump the ticket, but I strongly suspect they'll just blame it on my config or my incompetence.
Just seen this thread by your mention on our feedback thread. We cannot reproduce the issue from our end, I remembered replying to a ticket suggested to boot the VPS into rescue iso (available in SolusVM Control Panel) and started from there to eliminate any issues with the OS.
no one's talking about cpu usage. is it normal?
@pwned in which node are you? I'm in SJCKVM4 and have 20s TCP checks on my server, and not a single check has failed for 7 days at least. I also have an established SSH session which hasn't gone down for like 10 days, maybe.
@zGato: I'm on EPYCSJC3. I think it's been three weeks between the last two outages, and when it does go down IPv6 doesn't seem to be affected.
@nanankcornering: No excessive CPU usage that I noticed. I normally run munin for the pretty graphs, and they box is mostly idle.
@NDTN: I've just switched to the rescue image (but I've already tested (and seen outages) with your debian 11 template, your debian 12 template, and the debian.org debian 12 netinst). That rescue is pretty old--debian stretch? But I think I got the /etc/apt/sources fixed so I could install netcat, nginx, tcpdump, etc. I find it really hard to believe this could be an OS issue and that I wouldn't see it on any of the other 30 or so debian VPSs I run. But hey, I'm game to run your rescue image a few weeks to see what happens.
Update: ran the rescue image for 55 days (!) with no network interruptions (monitored TCP, UDP, and ICMP over both IPv4 and IPv6, to/from multiple locations). So I guess I'll have to eat my words and admit that there must be something wrong with the stock debian 10, 11, and 12 that was the actual cause of the intermittent network outages. Very strange.
That said, I did a fresh Debian 12 ZFS on root install from the Debian live CD ISO, and I've not seen any of the weird network problems over the last 28 days.
Thanks for all the help on this. Too bad I never conclusively proved the problem.
EpycSJ server just got junglesec infected through IPMI interfaces
Eh? Proof? Thought they had some hardware issues.
I managed to access some node via rescue mode