TCP SYN Panic Vulnerability (Affects all modern Linux kernels)

SplitIce

Patch your kernels or deploy firewall rules people.

Disclosure: https://access.redhat.com/security/vulnerabilities/tcpsack

Type: Remote Denial Of Service via Kernel Panic (BUG_ON)

Mitigation: Multiple Options.

One of the easiest:

# iptables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP
# ip6tables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP

Also easy (but with greater negative effect):

sysctl -w net.ipv4.tcp_sack=0

NOTE: We at X4B have been largely patched as side effect of our SYN mitigation (unintentional side effect). We have deployed specific mitigation rules for our infrastructure and our customers.

axzxc1236

https://news.ycombinator.com/item?id=20206756
Has links to other vendors' disclosure page.
On Netflix's advisory it also mentions another CVE, CVE-2019-5599 affects FreeBSD 12 using the RACK TCP Stack.

datanoise

Nice!

jsg

What a nice feeling to patch a vulnerability (and to wait exitedly for the next one, out of hundreds if not thousands ...)!

FHR

Note that the IPtables solution works ONLY IF net.ipv4.tcp_mtu_probing is DISABLED

Levi

A remote user can trigger this issue by setting the Maximum Segment Size(MSS) of a TCP connection to its lowest limit of 48 bytes and sending a sequence of specially crafted SACK packets. Lowest MSS leaves merely 8 bytes of data per segment, thus increasing the number of TCP segments required to send all data.

So, setting MSS size in firewall between 1 to 500 helps? What if remote attacker will set MSS to for example 50 000?

SplitIce

@LTniger that's match not set.

I'm not actually sure how 500 was calculated. It stinks of a ballpark guess, but anyway it's netflix's number. 1-500 is an abnormally low MSS anyway.

FHR

@LTniger said:

A remote user can trigger this issue by setting the Maximum Segment Size(MSS) of a TCP connection to its lowest limit of 48 bytes and sending a sequence of specially crafted SACK packets. Lowest MSS leaves merely 8 bytes of data per segment, thus increasing the number of TCP segments required to send all data.

So, setting MSS size in firewall between 1 to 500 helps? What if remote attacker will set MSS to for example 50 000?

It's not setting MSS, it's dropping all connections that would attempt to use MSS lower than 500.

Regarding why it's 500, my guess is that it was chosen based on several factors:

• It couldn't have been too large or it would start dropping a lot of legitimate connections
• The lower it is, the easier it is to cause a DOS, which the vulnerability is about.

rm_

SplitIce said: Also easy (but with greater negative effect):

sysctl -w net.ipv4.tcp_sack=0

It is understated how negative the effect can be, just now I was puzzled why my rsync downloads from a remote host go at 50-100 KB/sec with some peaks to 1-2 MB/sec, remembered this, switched back to =1, and got my usual 10.8 MB/sec back instantly.

Better deploy the iptables variant of the workaround as proposed in the 1st post.

FHR

@rm_ said:

SplitIce said: Also easy (but with greater negative effect):

sysctl -w net.ipv4.tcp_sack=0

It is understated how negative the effect can be, just now I was puzzled why my rsync downloads from a remote host go at 50-100 KB/sec with some peaks to 1-2 MB/sec, remembered this, switched back to =1, and got my usual 10.8 MB/sec back instantly.

Better deploy the iptables variant of the workaround as proposed in the 1st post.

Yeah turning off SACKs is madness.

eva2000

rm_ said: It is understated how negative the effect can be, just now I was puzzled why my rsync downloads from a remote host go at 50-100 KB/sec with some peaks to 1-2 MB/sec, remembered this, switched back to =1, and got my usual 10.8 MB/sec back instantly.

Better deploy the iptables variant of the workaround as proposed in the 1st post.

Why not update your linux kernels ? I already updated mine for CentOS

For CentOS 7

uname -r
3.10.0-957.21.3.el7.x86_64

For CentOS 6

uname -r
2.6.32-754.15.3.el6.x86_64

For ELrepo YUM Kernels

uname -r
5.1.11-1.el7.elrepo.x86_64

For Linode custom Kernels

uname -r
5.1.11-x86_64-linode127

SplitIce

@eva2000 maintenance windows & reboots suck for one on production services.

The iptables rule is basically harmless <500 MSS connections are non-standard. From our network excluding servers we have had syn floods on in the past 4 days we have had 0 hits on our rules.

eva2000

SplitIce said: maintenance windows & reboots suck for one on production services.

yeah i suspected as much

linode custom kernel update and reboot was crazy fast ~10 seconds all up !

rm_

eva2000 said: Why not update your linux kernels ? I already updated mine for CentOS

I don't want to reboot my servers just for this, so now I added to my iptables script, if the kernel is not new enough (doesn't have the newly added /proc/sys/net/ipv4/tcp_min_snd_mss), it will apply the iptables workaround.