All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Connectivity testing magic (or dirty tricks)
A report about an experience with (not really) surprising and rather ugly results. It's to do with network/connectivity benchmarking and in particular (but not limited to) the speedtest stuff.
First two important disclaimers: This is in no way meant to criticize @FlorinMarian or @Clouvider!
Florin created a great thread; the idea to put lots of providers' connectivity next to each other IMO is a very good one and a useful contribution to our LET community! So, thank you, Florin!
Clouvider appears to be an unpleasant provider - but this appearance is wrong and I wanted to state that clearly front up. In fact, he is among the very few with kind of a global presence who provides for both, 'speedtest' and download file testing. That is the reason why I chose him (and the fact that he known and well respected here as a decent colo/provider). Pretty much any other globally present provider wouldn't have looked any better and more than a few would in fact have looked way worse.
Softlayer for example seems to be a mess; plenty of locations don't seem to work anymore and it's hard to find information. Leaseweb is another example. They offer quite good 'download a test file' testing in quite a few locations but seem to be pretty much absent on speedtest.
I did my tests on both a normal @Hybula VPS (1 Gb/s limit) and a somewhat larger one with a 2 Gb/s limit. And I think we can all agree that Hybula is among the best in terms of connectivity (IMO even the best). So I guess my attempt at doing my tests in a fair way has a good basis (no, I did not test any provider closely linked to, let alone hosted by Hybula). As I said I wanted this to be fair.
Last thing before we go to the beef: I intentionally chose only a few locations and wherever possible with 2 targets there. Unfortunately in Brazil (the destination I chose to represent South-America) I had only one, and none in China (no surprise I guess). In Europe I have 2.5 locations ('.5' because not every one considers Russia (Moscow) as european). My reasoning was that the Netherlands is a rather small country, so I also tested another very major european hub (DE, FRA) but I also wanted a kind of far away and not too easy european location and Moscow looked like a good choice.
In North-America I chose the east coast (NYC/WDC) as well as the west coast (LA) and in Asia it's Singapore.
In the next post we look at the data and will find out the ugly truth.
Comments
First the "small" or normal Hybula VPS (and 'ST' means the 'speedtest' result):
Oopsie, do you see it? NL,AMS and DE,FRA match quite well. The difference between a true http test file download and 'speedtest' is small.
But when the targets are far away the difference becomes frankly ridiculously obvious.
So, are we really supposed to believe that a normal http download from the east coast is somewhere around 100 Mb/s but a 'speedtest' download somehow magically is about 10 times as fast?
And pretty much the same with the Singapore target ...
But wait, it gets even stranger when I test from the other, 2 Gb/s limited, Hybula VPS:
Now, when doing the same tests again even the close by targets are significantly faster with 'speedtest' than with a normal http download!
And again, with the large distances it becomes plain ridiculous. Up to 20x the speed!
It seems there is something to learn here.
I first began to notice the trickery when I had an encounter with another "hey, let's pimp up results!" attempt called 'iperf'.
The results when using 'iperf' and http with the same target, uhm, astonished me, but the difference was more modest.
But now with 'speedtest' we are confronted with what looks like obvious shitfuckery in plain sight to me.
I can of course understand that once one major provider uses those tricks all others have to choose between looking bad or also using trickery. I'm not even sure that I can be angry at them.
TL;DR
'speedtest' is - at best - a crude hint, if that, but do not trust it!
@Hybula is prem
iperf3 has UDP mode and TCP mode.
YABS chooses UDP mode.
Which one better reflects application performance shall depend on the application scenario.
HTTP/2 and (most remotes of) rclone are based on TCP; HTTP/3 and SIP telephony are based on UDP.
isnt UDP more unrealiable than TCP while TCP is slower due to all the talking it needs to do?
in my understanding udp just screams into the receiver, if it hears, good, if not, drop
while tcp technically has a better connection with the receiver.
Thank you for the feedback provided this time too!
It's very difficult for me to talk about the results because I lived to see how over 80% of the packets would go through RCS&RDS (150Mbps) if I didn't order them to choose the route from Orange (1024Mbps) with a preference of over 500%.
I believe that we have a decent connection in Romania and a good part of Europe, but what I can say is that if we had ~1Gbps dedicated with each provider, in order to have good results almost everywhere, we would have to make the compromise of paying for 900Mbps RCS&RDS as for 2.5Gbps at Orange, and the decision is damn hard to make.
The major price difference comes from the fact that Digi does not have a very capable infrastructure in my area, while at Orange we even have a dedicated line with a length of 12km and a capacity of up to 10Gbps dedicated.
you can implement reliability in higher layer (i.e Application layer). With TCP, you get it for free
One's got to bear in mind that speedtest.net uses proprietary algorithms. I believe they would also run this test simultaneously over multiple connections, which is why it is often likely going to report a higher result against different protocols tested between the same hosts. This will be very visible if your ISP runs 10G connections in a bundle and the traffic hits more saturation during a single-connection test.
Finally, as we always tell our Customers, speedtest.net is not designed to be used on servers. Our Customers are always pointed to run an iperf3 test against the same servers.
Thanks for shedding some light on speedtest's trickery.
Frankly, even iperf3 didn't look honest but, yes, it's certainly providing a more realistic guesstimate than speedtest.
At the end of the day potential customers want to know, how their stuff, typically a web server (with or without some app on top/behind) would perform. That's why I built a "boring" http test into my network testing/benchmarking. Btw. it might also be interesting to look at ping ('P') vs my "web-ping" ('WP') in my results, where (basically) P shows, well, ping time but WP shows "ping plus reaction time of the http server".
And again, kudos to you @Clouvider for being one of very few colos/providers with a (kind of) global presence who allows for, and actually seems to encourage diverse forms of testing/benchmarking.
But then, you obviously have a very solid basis for your self-confidence
Speedtest clearly states in the website that it is a multi connection test and you can switch it to single connection:
Trickery makes me think of deliberately serving a testfile of 0s over https with compression on.
Multiple connections, not so much.
What fucking trickery? They're counting bytes, and they are optimizing for it. They even tell you the app will get better results than using speedtest.net. Prove they are counting bytes wrong or stop calling shit you don't understand trickery.
Iperf3 is NOT to give you realistic real world results, it's to determine your pipe's capabilities. There's other tools to model real world traffic.
Unless you know tcp vs udp, the parallel streams and the congestion algorithms, what's the fucking point of this?
Also, I haven't looked at the code, but I highly doubt yabs uses udp for iperf3 tests since UDP needs to specify the bandwidth load and then you basically need to do a binary search to find the maximum throughout (sending way too much hurts CPU, interrupts and buffers causing a lower result than sending say ~105% of the actual pipe bandwidth).
Lastly, iperf3 is really kind of crappy and I think a lot of devs just build their own test tools after encountering so many bugs with iperf3... however, iperf3 is still used because it's free and can be used in anybody's lab as a sanity check.