Ubuntu 22.04 slow IPv6 forwarding

jordynegen11

Hello,

Im working on a little experiment where i'm using a ubuntu 22.04 server as my BGP router at home. Currently only have IPv6 addresses announced to it. However I run into an issue where I only get 20-300Mbit upload speed for devices that are connected to port enp3s0f1np1, while the router itself get's a decent 8-9Gbit upload.

Steps I tried:

• Update the drivers if the NIC
• Update the firmware of the NIC (Mellanox ConnectX-5)
• Replaced the NIC
• Installed everything again on a different server.
• Updated the kernel
• Downgraded the kernel
• Poked arround with the sysctl settings

What am I missing here?

Below are speedtest results but we see the same behaviour in iPerf.

Speedtest on the router (no problems here):

speedtest --server-id=35058

   Speedtest by Ookla

      Server: Clouvider Ltd - Amsterdam (id: 35058)
         ISP: 
Idle Latency:     1.86 ms   (jitter: 0.04ms, low: 1.81ms, high: 1.93ms)
    Download:  7865.62 Mbps (data used: 6.4 GB)
                  2.58 ms   (jitter: 3.39ms, low: 1.83ms, high: 210.35ms)
      Upload:  9057.08 Mbps (data used: 6.6 GB)
                  2.37 ms   (jitter: 0.45ms, low: 1.68ms, high: 4.29ms)
 Packet Loss:     0.0%

Speedtest on Server 2 that is connected to port enp3s0f1np1:

 speedtest --server-id=35058

   Speedtest by Ookla

      Server: Clouvider Ltd - Amsterdam (id: 35058)
         ISP:
 Idle Latency:     1.88 ms   (jitter: 0.02ms, low: 1.85ms, high: 1.90ms)
    Download:  7631.48 Mbps (data used: 10.4 GB)
                  2.38 ms   (jitter: 0.36ms, low: 1.65ms, high: 4.20ms)
      Upload:   278.13 Mbps (data used: 300.8 MB)
                  1.65 ms   (jitter: 0.09ms, low: 1.59ms, high: 2.24ms)
 Packet Loss:     0.0%

Netplan of the router:

network:
  renderer: networkd
  version: 2
  ethernets:
    eno6:
      dhcp4: no
    enp3s0f0np0:
      dhcp4: no
    enp3s0f1np1:
      dhcp4: no
      addresses:
      - 2001:12d:fe8::/48
  vlans:
    vlan2223:
      id: 2223
      link: enp3s0f0np0
      addresses: [117.17.18.225/29, 2a02:60:0:6::101:10/112]
      dhcp6: no
      accept-ra: no
    vlan9:
      id: 9
      link: enp3s0f0np0
      addresses: [93.239.11.221/22, 2001:8f1:23::a504:8325:1/64]
      dhcp6: no
      accept-ra: no

/etc/sysctl.conf

net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
net.ipv4.icmp_echo_ignore_all=0
net.ipv4.tcp_syn_retries=3
net.ipv4.tcp_synack_retries=3
net.core.somaxconn=65535
net.core.rmem_default=3407872
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_max_syn_backlog=8192
net.core.netdev_max_backlog=30000
net.core.netdev_max_backlog=250000
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 87380 67108864
fs.file-max=12000500
fs.nr_open=1073741816
vm.max_map_count=1048575
net.ipv4.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh3=4096
net.ipv6.neigh.default.gc_thresh1=1024
net.ipv6.neigh.default.gc_thresh2=2048
net.ipv6.neigh.default.gc_thresh3=4096
net.ipv6.route.max_size=1073741824
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_ra=0
net.ipv4.tcp_congestion_control=bbr
net.core.default_qdisc=fq
net.ipv4.ip_forward=1
net.core.netdev_budget=1000
net.core.netdev_max_backlog=1000

Bird

protocol static {
        ipv6;
        route 2001:12d:fe8::/48 via "enp3s0f1np1";
}

rm_

@jordynegen11 said: net.ipv4.tcp_congestion_control=bbr

Does the "server 2" use bbr as well?

One more thing, try disabling Flow Control on the NICs of both servers.

jordynegen11

You are amazing. Disable flow control worked!

sbcv

How can i disable flow control on Ubuntu (24.04)?