All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Slow single TCP connection
Hey, I have problem with 4 dedis I got from Hetzner.
All of them max out at 8-10Mbps on single TCP connection.
iperf, http1.1, http2, sftp - doesnt matter, I get 8-10Mbps maximum if its just a single TCP connection.
There is no problem when I start multiple connections on iperf or http via IDM which can open multiple connections over TCP - transfers can easily go to hundreds of Mb/s then.
I have this problem on both Debian 11 and Ubuntu 22.04
Zero impact from:
- Turning up MTU from 1500 to 9000
- Turning on BBR
- Disabling UFW
- Compiling Nginx from source with AIO (so I tried directio, aio threads and sendfile, no difference, it is always 9-10Mbps when sending big files)
- Using different "client" machine on different network
- Increased buffer size like suggested here https://www.cyberciti.biz/faq/linux-tcp-tuning/
- Rebooting
For comparison Oracle Free Tier shitty x86 1/8OCPU instance gets 25Mbps, MTU is on 9000. Its on CentOS7, maybe thats the cause?
There's no way dedi cant reach these speeds... I'm clearly missing something
Or does Hetzner limit single TCP connection speed somehow and Im just wasting time?
Server specs
2x AX41 - Ryzen 5 3600, 64GB ram, 2x 512GB NVMe etc.
2x SX162 - E5 1650V3, 128GB ram, 10x 10GB HDD etc.
CPU usage is like 1%.
Comments
If you do a speedtest downloading a file to one of these servers, does it reach the desired speed or the same happen?
Is it only over a single network route or multiple?
My point is, if you wget a speedtest file and your speed sucks the same, then would be wise to take that up with Hetzner, maybe you have some limits...forced?
Hey, you're stating that you have the following servers:
2x AX41 - Ryzen 5 3600, 64GB ram, 2x 512GB NVMe etc.
2x SX162 - E5 1650V3, 128GB ram, 10x 10GB HDD etc.
Is there something like a storage share (SMB/NFS/etc) active between those storage SX162's and ax41's? That could be a big bottleneck.
Otherwise, please give us some more information on the entire stack, maybe print the output of an entire
ps auxf
here?Irrelevant, op mentioned that testing using iperf gives the same result.
MTU 9000, as in jumbo frames? stop that, it's not gonna play well on WAN.
What's the dest server? I'm sure some of us can provide you an iperf on 10g,
but you should try speedtest-cli, since it's Hetzner - no way they are throttling 'some'
of your dests in EU.
Is this happening even with connections to other Hetzner servers?
What model of network card/chipset do you have? A low-end Realtek network adapter won't perform as well as a higher-end Intel one for example.
How high is kernel CPU usage while you're performing these tests?
I'd recommend just using iPerf, so that you can rule out disk I/O performance.
On the internet, this is going to make things worse. Internet routers don't support jumbo frames, so you'll end up with fragmented packets (which just adds overhead) and/or the connection speed quickly dropping to 0 after a few seconds.
No, they dont have shared storage active.
AX41 are EXT4 RAID1
SX162 are XFS RAID-Z2
So all of them are plenty fast
Storage isnt bottleneck and IIRC iperf3 isnt affected by storage speed.
Output will be in my next comment, because I reached max char limit.
Ok, I only tried it because on Oracle free instance I have it set like this by default and thought that maybe the issue. There's zero difference in performance tho so I just rebooted (I made non-persistent change )
Poland, two different connections in different cities. Hetzner machines are in Finland.
That is a great question!
I tried exact iperf3 benchmark, but now between two Hetzner servers (AX41 and SX162)
Hetzner -> Hetzner 950Mbps.
So no problem here.
All of them have Intel I210 Gigabit chips.
Actually, I dont know how to check kernel CPU usage, could you please tell me where I can find it?
In top I only see that iperf3 and top is eating some small percentage, nothing more. Steal, Wa, Hi, Si are 0.0. Us and Sy (User and system?) are around 1.0
I checked against serverius iperf server
hetzner -> serverius is ~744Mbps
serverius -> hetzner is ~454Mbps
hetzner -> hetzner is ~950Mbps
hetzner -> my two connections in different cities and different ISPs is ~10Mbps
ping hetzner -> hetzner is 0.5ms
ping hetzner -> serverius is 25ms
ping hetzner -> my connection is 59ms
And remember, there is no problem over multiple connections, I can easily get 1Gbps on "hetzner -> my connection" if I establish multiple ones, but every one is limited to 9-10Mbps.
Here's tracert from my connection to Hetzner
1 <1 ms <1 ms <1 ms 192.168.1.1
2 1 ms <1 ms <1 ms 192.168.0.1
3 6 ms 6 ms 7 ms 10.20.0.1
4 8 ms 6 ms 7 ms c97-241.icpnet.pl [62.21.97.241]
5 9 ms 6 ms 6 ms e91-118.icpnet.pl [46.238.91.118]
6 19 ms 8 ms 9 ms e91-109.icpnet.pl [46.238.91.109]
7 7 ms 6 ms 10 ms e123-10.icpnet.pl [46.238.123.10]
8 18 ms 21 ms 17 ms 212.162.29.165
9 56 ms 60 ms 60 ms ae1.16.edge1.Helsinki1.level3.net [4.69.161.102]
10 59 ms 60 ms 59 ms 212.133.6.2
11 61 ms 61 ms 55 ms core32.hel1.hetzner.com [213.239.224.26]
12 58 ms 58 ms 58 ms ex9k2.dc4.hel1.hetzner.com [213.239.252.214]
13 57 ms 57 ms 59 ms static.XXtargetXX.clients.your-server.de [TARGET IP]
So it goes via my ISP (ICPNET/INEA) directly to Helsinki via Level3.
Hmm...
ps auxf as requested by @FoxelVox
ps auxf
So there is no problem from Hetzner to other DCs either? Only two (home broadband?) connections in Poland?
I was going to say "try disabling Flow Control on the NICs", but from that it sounds more like your Poland ISPs are to blame, not Hetzner. Try upload tests from Hetzner to a diverse set of locations worldwide, not just one country. Running a YABS can do that.
If you face a problem with Hetzner->PL, then traceroute the same direction (from Hetzner), as it might take a different route.
That can be indeed the case, because now I checked from VPSes I have in all locations (Sofia, Amsterdam, Texas etc.) and even shitty boomer.host with 100ms+ ping gets like 100Mbps!
I created Nuremberg Cloud instance and iperf result (my connection -> Hetzner Nuremberg) is 15Mbps.
So either Hetzner or my ISP is here to blame, I will ask friends with different ISPs in Poland to test it out and I'll come back
Edit: Ok most unexpected shit ever - my phone gets 30Mbps to Hetzner, WiFi connection to same router.
I'll check if Windows is causing this shit lol
If you have so many VPSes, just check iperf3 upload from Hetzner to them all. As for why it varies with PL ISPs, both of those could be using the same peering link or same exchange (check with tracroute from Hetzner to PL), and that one might be overloaded.
Traceroute from Hetzner show the same route (Level3 -> PL Inea).
I messaged one friend that has different ISP. He got 70Mbps. Still Poland.
He has 2x less ping but his tracert looks like mess (it routes via Germany)... Who is here to blame and who can fix it? 10Mbps is unusable, cant stream video etc.
Try
netsh int tcp set global autotuninglevel=normal
(Normal is the default, was chasing similar TCP weirdness on one windows machine, not sure how it ever got turned off)
Once again, it is next to useless to look at the upload direction traceroute to Hetzner, if the issue is in download speeds from Hetzner. Up/down routes can and will route differently, even if broadly the same set of ISPs (can be different peering locations or routers).
Ask your friend for his home IPv4 and then traceroute that from your server. But if he gets 70, then perhaps he is not affected, and just keep investigation to the prior two locations.
Better yet, instead of traceroute run
mtr
, and leave it for a 1000 of packets, to see if there is any packet loss to the problem destinations.Ok I understand
Here's traceroute from Hetzner to Inea (bad speed)
and then * * * 21x
Here's traceroute from Hetzner to Komster (good speed)
and then * * * 21x
Both of these connections are behind NAT btw.
As we can see traceroute to Inea is kinda the same, but to Komster completly different.
Should I test MTR from Hetzner to local Inea connection or other way around?
So routes are here to blame?
That made my results very inconsistent, now I get 5-22Mbps, but average improved to 17Mbps from 8-10Mbps. Its still not good enough, 30-50Mbps is something I chase...
Still thank you very much, I wait when my friend with other 8-10Mbps limited connection is online so we can test if this helps him too
Yes, better from Hetzner. But in general what you can mostly achieve here is to exonerate Hetzner (as I said above with YABS or upload tests from Hetzner to all of your VPSes), and narrow the issue down to particular ISPs.
You can post Hetzner a ticket with bad speeds to your ISPs, and maybe they can try making a route change. But it is not guaranteed that they will be able to (or will want to).
Or complain to your ISPs (good luck with that).
If the servers are otherwise a good deal for you, might be an idea to explore proxying traffic from Hetzner (rerouting via VPN) through some VPS from which you do have good speeds, such as a VPS directly in Poland.
PM me address/IP will run iperf/wget from Orange PL to mess even more with your results, lmao.
PM'ed. Lets see
Why edit MTU default? It is perfect as it is. Never touch MTU.
Like I said - Oracle instance has MTU 9000 as default and I performance was way better so I tried to copy settings (incl. MTU) to my Hetzner dedi to see if something changes.
It was temporary, non-persistent changes so I rebooted my dedi and its back to default
@AXYZE Some NICs cause problems. Try disabling all hardware offload, or one by one. For instance:
ethtool -K eth0 tso off gso off
I got nice 100Mbit/s from Orange PL on his iperf in single connection, so I assume it's not really "server" limited - somewhere along the network.
Unfortunately RPi is only on 100Mbit/s card and when I run iperf3 on Windows 10 (that is gigabit NIC) results are even more worse ^.-
I have no fuckin idea what happened but problem is fixed, I did nothing, route on tracert is still the same. Now I have 300Mbps (bandwidth of my home internet) on single HTTP connection.
I think that route was somehow broken/overloaded and they just fixed it.
Damn...
Ping is still 2x higher (60ms) with my provider than with different ones in same city (30ms), I'll contact Hetzner if they can reroute it.
@MikeA , @LTniger (nice name), @JabJab , @rm_ , @darkimmortal , @Hxxx , @jmgcaguicla , @FoxelVox , @Daniel15 , @luckypenguin
Thank you all for your input and help!!!
At work, when things "self heal" (at least from our view), we'll just take it as one of those "weird Internet things" moments
If we think it's necessary or if it was highly noticeable and before the customer asks what happened, then we'll probably request for the RCA from the relevant parties/stakeholders.
@AXYZE I had the same issue, horrible performance to different networks, I have found no help at all till I changed some lines at my NGINX conf.
Normally I would assume your "listen" config looks something like this:
What most user of NGINX never come across is the "sndbuf" and "rcvbuf" config of nginx which is quit important if you're sending out lots of data.
Please try to change your config to the following:
"Fastopen" is optional and must be supported by the kernel. If you want to use if, please also make sure you have the following set at /etc/sysctl.conf:
I have fully tested this against many ISP providers, it performs well with Hetzner. If I do a https download between NetCup (downloads) and Hetzner (uploads) I get 2,5 Gbit/s on a single https connection.
If you are intressted in the rest of my sysctl.conf, have a look here:
This config is intended to put the server in any means to hits limits.
Currently, I'm running this on 8-16 Cores and 16-32GB of RAM.
Cheers.
Did you do any kind of testing to verify that these settings are really useful? I know if you google for these settings you get lots of recommendations for different settings but I feel like it's always anecdotal and rarely tested. The ones you have for vm.max_map_count, vm.swappiness and vm.vfs_cache_pressure are actually very commonly recommended if you google but having these settings just means you don't trust linux to manage the memory.
If you're running a db or something, sure set the swappiness to 1 or even 0. That will definitely help. It also makes sense to configure things like net.core.somaxconn or the open files limit depending on your use case. I feel like most other configs are just blindly increasing values just because it seems like it might increase performance (but in reality it might not).
@NoComment
Well, yes I did a lot of testing as this issue was almost driven me mad. I can definitly confirm that these settings do the job.
I highly doubt that. 9000 MTU only on internal, private NICs, not the public facing one.
But still, set as 9000. I tried to duplicate every setting from machine that works nice, thats all.
Its necro btw, problem was "fixed" 1.5months ago, it was routing problem from Finland and now I know that it sadly happens from time to time on this Hetzner location seems like network there is much worse...