tcp_mtu_probing, should I touch it or no?

oddmario

I run a file storage server which faces the internet and serves customers, so high throughput is one of the goals I try to achieve on my servers

The past 6 months I've been studying the Linux kernel source code and sysctl docs to learn what each tunable parameter actually does instead of blindly pasting configurations from tuning guides and just hope that it makes things perfect

Now one of the points I'm stuck at is net.ipv4.tcp_mtu_probing

I see a lot of tuning guides suggest setting that to 1 or even 2 instead of the default 0

But if that's really recommended, why doesn't Linux set it to 1 by default instead of 0? I mean 1 seems like a better moderate value to set instead of disabling it completely

Although the below points kinda hold me back from altering tcp_mtu_probing but I might be wrong and that's why I opened this topic to ask for advice:

1. Packet-layer path MTU discovery (TCP MTU probing, QUIC MTU probing, etc) just mask a real underlying MTU problem which should be fixed from its root instead of hiding it
2. TCP MTU probing relies on packet loss and this can falsely make congestion control algorithms work worse and reduce the congestion window even if there's no real congestion
3. Certain quirky firewalls may hate the fact that my server is sending data in variable packet sizes because of the MTU probing and hence they may drop the packets completely or block the connection entirely

Do my above points make sense or am I mistaken?

allthemtings

Did you switch to BBR?

oddmario

@allthemtings said:
Did you switch to BBR?

Yes I use BBRv1 alongside fq as the qdisc

tentor

Never had issues due to MTU probing, but had a few due to MTU mismatch.

slowservers

You're starting to get into an area where you might have performance gains for some and terrible connections for others. Congestion algorithms and MTU discovery can be like playing with fire. You might be fine 99.99% of the time, but then there's a chance that you've broken it for someone else.

I'm not saying this to try and disaude you, but there's some potential for unintended consequences that can be very hard to notice.

Couple of examples:

I had a customer who enabled BBR and was getting poor performance on Slow Servers. We emailed back and forth and he found that with it disabled, things went smoothly again (able to hit near the 100Mbit/sec cap.)

Many years ago at a previous employer we had a router pair that was sending packets out of order. Super weird problem. Linux was completely unimpacted. FreeBSD and Window's TCP stacks crawled along in that scenario.

At the same employer we had an issue with an offering, I beleve it was load balancing. It only impacted Windows. Turned out to be from a load balancer pair having a high MTU configured instead of 1,500.

Unfortunately, testing for every scenario is extremely difficult! Of course right now your services could be "broken" for some and you won't even know about it. And changes that fix those might break others. I wish it wasn't so tricky, but it often is.

emgh

@slowservers said: I had a customer who enabled BBR and was getting poor performance on Slow Servers. We emailed back and forth and he found that with it disabled, things went smoothly again (able to hit near the 100Mbit/sec cap.)

very interesting takes overall

for what it's worth though i'm using BBR and have done so for years with 0 issues

since it's developed by google, i assume they probably use it throughout their infra

jsg

@oddmario

I suggest to look at the source code for an authoritative - and up to date! - answer.