All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
tcp_mtu_probing, should I touch it or no?
I run a file storage server which faces the internet and serves customers, so high throughput is one of the goals I try to achieve on my servers
The past 6 months I've been studying the Linux kernel source code and sysctl docs to learn what each tunable parameter actually does instead of blindly pasting configurations from tuning guides and just hope that it makes things perfect
Now one of the points I'm stuck at is net.ipv4.tcp_mtu_probing
I see a lot of tuning guides suggest setting that to 1 or even 2 instead of the default 0
But if that's really recommended, why doesn't Linux set it to 1 by default instead of 0? I mean 1 seems like a better moderate value to set instead of disabling it completely
Although the below points kinda hold me back from altering tcp_mtu_probing but I might be wrong and that's why I opened this topic to ask for advice:
- Packet-layer path MTU discovery (TCP MTU probing, QUIC MTU probing, etc) just mask a real underlying MTU problem which should be fixed from its root instead of hiding it
- TCP MTU probing relies on packet loss and this can falsely make congestion control algorithms work worse and reduce the congestion window even if there's no real congestion
- Certain quirky firewalls may hate the fact that my server is sending data in variable packet sizes because of the MTU probing and hence they may drop the packets completely or block the connection entirely
Do my above points make sense or am I mistaken?
Comments
Did you switch to BBR?
Yes I use BBRv1 alongside fq as the qdisc
Never had issues due to MTU probing, but had a few due to MTU mismatch.
You're starting to get into an area where you might have performance gains for some and terrible connections for others. Congestion algorithms and MTU discovery can be like playing with fire. You might be fine 99.99% of the time, but then there's a chance that you've broken it for someone else.
I'm not saying this to try and disaude you, but there's some potential for unintended consequences that can be very hard to notice.
Couple of examples:
I had a customer who enabled BBR and was getting poor performance on Slow Servers. We emailed back and forth and he found that with it disabled, things went smoothly again (able to hit near the 100Mbit/sec cap.)
Many years ago at a previous employer we had a router pair that was sending packets out of order. Super weird problem. Linux was completely unimpacted. FreeBSD and Window's TCP stacks crawled along in that scenario.
At the same employer we had an issue with an offering, I beleve it was load balancing. It only impacted Windows. Turned out to be from a load balancer pair having a high MTU configured instead of 1,500.
Unfortunately, testing for every scenario is extremely difficult! Of course right now your services could be "broken" for some and you won't even know about it. And changes that fix those might break others. I wish it wasn't so tricky, but it often is.
very interesting takes overall
for what it's worth though i'm using BBR and have done so for years with 0 issues
since it's developed by google, i assume they probably use it throughout their infra
@oddmario
I suggest to look at the source code for an authoritative - and up to date! - answer.
this is something I think about a lot and it's sadly true 😅 I often wish networking equipments globally followed strict standards so we don't have to guess what will work for most clients and what will break it for a portion of them
I'm not sure if that's the case for your BBR story, but one thing I noticed with the modern versions of BBR like BBRv2 and BBRv3 is that they're triggered by a percent of packet loss
BBRv1 completely ignores packet loss and I'm not sure whether that's a good or a bad thing, but I had a lot of trouble on certain route paths with BBRv2 and v3 during peak hours. reverting to the default BBRv1 shipped with the mainline kernel solved these problems
Most of the time I try not to tune much and just trust the kernel or the systemd defaults, but for some things like the congestion control algorithm I'm forced to tune them
For example I get really terrible throughput for high BDP clients if I don't change the CCA from CUBIC to BBR
The universal rule for performance tweaks is to measure, not guess. There are usually so many variables interacting that predictions are very unreliable. If it was always good, it would be enabled by default.
They do. The problem is that one model doesn't work best with all workloads. On one end you have satellites and lasers with terabit/s bandwidth but huge latency. And then you have low latency but low bandwidth. Then throw in various amounts of random packet loss and packets out of order.
You should be arguing with the OS makers to standardize on default settings so settings match or there's an equivalent experience.
Sometimes the defaults don't update with bandwidth increases and they do need to be tuned.
On Windows machines, I've often used speedguide.net's TCP Optimizer to get improved throughput using 10Gb nics.
One area that is under-researched and has tremendous potential, in my opinion, is dynamic tuning in the vein of
bpftune, which does have a TCP congestion control algorithm auto-tuner. But so much more could be tuned automatically, in theory.