All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Lost guy with network performance issues on his new VPSs (long story)
Hi everyone! I'm kinda new here, just wanna find someone to talk about some things that I'm kinda struggling right now.
I'm also quite new to buying VPS. In the past few years I have been sticking to my very first provider which is netcup, a VPS 200 G8 iv, to run my personal blog, and that only 1 VPS only. I made the decision to switch provider after discovering LET, and also come up with plans to play around with cloud stuffs since I graduated and had a job now (have income).
Last month I cancelled netcup and have purchased the following VPS:
1. Crunchbits: US, yearly 4.5G x1 - for personal blog
2. Crunchbits: US, Big Storage 2T x1 - for cloud storage, still deciding between owncloud, pydio, alist
Soon after, I realised that the network is too slow when acceding from my home, I'm only getting 10-20MBPS between my home internet and the crunchbit VPS. I don't know why their looking glass is at least 50MBPS when I tested. No matter how I test, iperf or sftp or what, the speed to my VPS is only 10-20MBPS. I felt kinda disappointed about this...
Hence, I came up with another plan: buy another small VPS in Asia (near Malaysia which I'm located) and reverse proxy to my storage VPS at crunchbits. After several days of digging, I went for Speedypage's Singapore 1G VPS. Again, the looking glass download this time around is 100MBPS (my ISP line full speed), but heck, I'm only getting 10-30MBPS when accessing the VPS... What's the issue? I can't quite understand this.......
Anyways, I proceeded to setup the reverse proxy, and the results are kinda bad as expected, only max 20MBPS when I download a file from the storage VPS at crunchbits.
Then I started to search high and low again for another VPS at asia.... finally few days back, came across the sale of Hosthatch. Bought the promotional 4GB singapore VPS. Setup reverse proxy. Ugh.... okay, that's actually pretty nice. I'm getting 40MBPS this time around. But since yesterday, I keep getting timed out issues accessing my storage VPS frontend, as well has my monitoring system keep warns me. And the speed kinda fluctuates between 10MBPS and 100MBPS (yes, I could finally reach > 70MBPS this time and I was soo happy).
Thus, I did a lot troubleshooting today and googled a lot, but no clue what's the issue. Done the following:
1. iperf3 between speedypage vps and hosthatch vps to check their internet speed - getting 6GBPS perfectly fine
2. iperf3 between my two crunchbits vps to check speed (thru public ip)- getting 1GBPS perfect
2. iperf3 between speedypage vps to crunchbits vps - upload 50, download 100mbps
3. iperf3 between hosthatch vps to crunchbits vps - upload also 50, download also 100
4. iperf3 between crunchbits vps to my laptop - don't ask... very terrible HAHA
5. iperf3 bewteen hosthatch vps to my laptop - 50, 50 mbps (my internet is 100 downoload, 50 upload)
6. iperf3 between speedypage vps to my laptop - 20, 20 mbps
Same steps above also done using SFTP and more of less the same results. And one thing to note, my reverse proxy connection between my singapore vps and crunchbits vps is connected through wireguard.
I'm kinda fedup with my "cloud stuffs hobby" plan now, will waste quite an amount of money if things didn't work out in the end (although really, the VPS prices that I bought are very great and all are good performing, I'm very satisfied. ) Does any experienced people... perhaps anyone could share some advice, kinda lost now.... I'm just interested in these kind of stuffs, and decided to spent some money to build something I could use in the long run.
I appreciate if you read all my nonsense till the end (sorry for my bad english), please don't laugh if I did something stupid lol XD (seems like setting up own cloud storage is a bad idea after all)
Comments
First of all clarify which units you mean:
MBps = megabyte per second
Mbps = megabit per second
MBPS = doesn't exist
mbps = doesn't exist
Based on the iperf results, it seems the lowest bandwidth connectivity is when you're reaching from your laptop. This can have many reason (performance of reverse proxy, internet quality, ...).
I had similar issue around a year ago, and what I eventually did was that I used and intermediary VPS (between myself and the final endpoint). This intermediary VPS had very good connectivity and via setupping a flexible remoting solution (I used KasmVNC), I was able to achieve what I needed. (yes, I know, what adding an extra hop means, but that was my only option that really worked)
BBR. BBR2
@leewp14 Open a ticket at crunchbits and I will see what we can do to help.
*Mbps, megabits per second. All mentioned in the post are Mbps. Overlooked that, sorry!
Ah! I seem to came across an article suggesting to use BRR this afternoon, forget to bookmark. I will try it out tomorrow, thanks!
Interesting. it does sounds like what I'm trying to implement at the moment. Basically my two VPS at Singapore (hosthatch and speedypage) is intended to act as intermediary, just like your case. Although Malaysia boasted about launching 5G but Malaysia is still Malaysia, connection to everywhere else in the world simply sucks. Singapore VPS also expensive, seems like not much option for me. Might have to consider HK ones....
Speaking about internet quality, I'm now started to suspecting if it is due to my ISP bad routing (peering or what is it?)? There have been cases where international sites loads very slow ornot loading at all on my ISP but perfectly fine on other ISP. It's very common situation here in Malaysia. This is what happens to me with my netcup VPS - frequent packet drop when connecting via my ISP.
I'll open ticket later morning (about to sleep now). Thanks!
*have to clarify that I'm not disappointed with crunchbits (sorry again for my bad English in the post...) but disappointed that my hobby plan is not working. I'm happy with the services and all is great.
I see a massive difference in speeds when I enable BBR. Like 25-30% increase overall. Takes 5 seconds to enable. Give it a shot!
Maybe @fluffernutter can share a magical sysctl.conf with some magic.
Could just be poor routing, but you can follow https://wiki.crowncloud.net/?How_to_enable_BBR_on_Ubuntu_20_04 to enable bbr
Might try cloudflare tunnels. Increased my speed from Hetzner Germany to US by 2x.
Thanks @fluffernutter and @Astro !
Huge thank you!
After enabling BRR, I'm able to get full speed when iperf3 form my laptop to Hoshatch Singapore, I'm very shocked
With this issue resolved, now the only remaining issue is packet loss between Hosthatch Singapore and Crunchbits US. Although some good news - with BRR enabled on both sides, the upload speed is now symmetrical to download speed, measuring at 100Mbps. TBH, that's totally fine for me already as my ISP plan is just 100/50 Mbps. As long as could reach 100Mbps, I'm more than happy.
I just tested just now and there's still tremendous amount of packet loss when pinging between the two VPS. Currently at work, will check on Speedypage VPS when free and update here. I will create a ticket for crunchbits/speedypage/hosthatch to check if the issue persists.
Are you seeing loss on the destination or just in between the routes? As long as your destination shows no loss you're fine.
On the destination, although it's just about 10 packets per 500 packets for now I think. Much more severe the day before, about 40% packet loss.
However an hour earlier I let mtr ran for about 1 hour, and do not observe any packet loss at all in between the 3 VPS (Hosthatch, Speedypage, Crunchbits). I'm not sure if the occasional packet loss could be due to network congestion, given that for some reasons the speed between US and Singapore seems to be capped at 100Mbps in a consistent manner.
As of now, I'm getting full speed download (100Mbps) from my storage VPS at Crunchbits, thru Hosthatch Singapore. I think everything is pretty much in order now after enabling BBR. Problem solved, thanks to you all!
I'll stick to Hosthatch as my reverse proxy, as the ping to my home is just ~15ms, compared to Speedypage which is ~45ms, while both performed similar when connecting to Crunchbits US at about 260ms. One interesting fact is that, ping from my home to Crunchbits US is just 200ms, but I'm not able to get past 10Mbps. I'm leaning towards blaming my ISP, as previously I did encounter not able to open Crunchbits control panel and also my Crunchbits VPS (just 2 days after purchase), and it turns out I can access both of them using my mobile phone network, with just around 100+ms ping. Ridiculous, isn't it?
Have a nice day!
@leewp14
First, @rm_ is right. Be careful because no matter how much good will to help one has, one needs data and while e.g. 40 Mb/s ('Mbps' is also OK) is crappy 40 MB/s is quite OK or even good.
Now to your story. Before digging any deeper (e.g. is 'BBR' the best choice?):
TL;DR Do not expect looking glass, iperf3, etc. numbers on your VPS!
True in most cases.
That being said, the usual culprit is the last mile to your home. Peering sucks even in parts of Europe and at peak time (evening local time) packet loss and high latency can strike and in many cases WILL strike like clockwork. Some armchair internet statisticians at some local ISP would see that the average BW consumption is C and they think Cx3 or even Cx2 should suffice and if a few hours a day there is some congestion nobody would die.
Yes, but when people come home after a long day of hard work and want to see a movie and it stutters or buffers every few seconds then they are going to be very angry and would not care that they can see 4K movies at 4 AM so a bit of "saving" in BW would end up alienating 90% of the customers.
Thanks for the comments @jsg @Maounique .
Replying to @jsg :
a) can't agree more with you. It's quite a hard decision to purchase an Asian VPS, and as an Asian, it's so hard to even buy a US/EU VPS due to the high latency and slow speed. I have spent more time to persuade myself and research on available providers, worry about if stuff works, than to spend the time enjoying setting up my VPS.
b) I wasn't actually expecting that. To be honest with you, I actually took looking glass as a reliable test that could well replicate my actual usage needs when using a VPS. Probably I'm way too new to realize the real truth. But if I'm a provider, it seems to make sense I will place looking glass on an unutilized server or non-commercial usage server/internal use server. So... stupid me that I didn't think of it.
c) Well aware of that now, thanks for the note!
@Maounique
I'm working on infra at a local IT company actually, and I do work on managing servers, VPS (VMs) for our clients (non-public facing usage), but can't be compared to an expert. Pretty much like what you said, what I wanted is going back home, able to work on my rented VPS with everything working well as per advertised, and not to worry about things that I always take care of at my work. I could have bought a server and setup at home, but I wouldn't want to take care about the extra things that I deal with (headaches) everyday at work. Because one of our major client is a huge headache that is way too stingy to fork out money for the necessary stuffs, and we do not have full direct authority as they do have their own tech team - everything has to go thru them. Their tech team is genius in a way that they used new CAT 6 cables on a pentium 4 dell PC, but lousy CAT5E on their servers, and servers with 10 year old spinning disks that has tons of ECC errors, and much more....
However about the issue of network and bad peering - do understand that it's likely an issue on my ISP end. But since now everything going thru Singapore VPS (10+ms, Singapore is at the bottom of Malaysia - neighbors), there's no more issue connecting between my US VPS to my home with Singapore VPS as reverse proxy. The remaining issue is the packet loss between my Singapore and US VPS. But as of current writing, everything works well and I'm still getting full 100Mbps download. I wouldn't bother to check mtr or whatsoever anymore as long as it works and I can get more than 60Mbps when I needed it, I'm more than happy.
(run
I still have a fleet of servers/VMs to manage, going back to work now
Nice, that means that your ISP likely peers with your SG host with a fat enough pipe and the SG host has a fat carrier to US and your VM is not in a bad neighbourhood.
The packet loss issue can be investigated with a MTR and you will know at which hop it happens. Remember that the routers along the way will occasionally drop ICMP packers so you might have some or even total packet loss indicated on some routers but if that does not propagate to the end then it is just the router dropping packets due to the configuration or low priority. Some people believe they can save some BW and router cpu cycles if they drop ICMP while also preventing some type of attacks but it is kind of BS in the end.