Ubuntu 22.04 slow IPv6 forwarding

Hello,

Im working on a little experiment where i'm using a ubuntu 22.04 server as my BGP router at home. Currently only have IPv6 addresses announced to it. However I run into an issue where I only get 20-300Mbit upload speed for devices that are connected to port enp3s0f1np1, while the router itself get's a decent 8-9Gbit upload.

Steps I tried:

• Update the drivers if the NIC
• Update the firmware of the NIC (Mellanox ConnectX-5)
• Replaced the NIC
• Installed everything again on a different server.
• Updated the kernel
• Downgraded the kernel
• Poked arround with the sysctl settings

What am I missing here?

Below are speedtest results but we see the same behaviour in iPerf.

Speedtest on the router (no problems here):

speedtest --server-id=35058

   Speedtest by Ookla

      Server: Clouvider Ltd - Amsterdam (id: 35058)
         ISP: 
Idle Latency:     1.86 ms   (jitter: 0.04ms, low: 1.81ms, high: 1.93ms)
    Download:  7865.62 Mbps (data used: 6.4 GB)
                  2.58 ms   (jitter: 3.39ms, low: 1.83ms, high: 210.35ms)
      Upload:  9057.08 Mbps (data used: 6.6 GB)
                  2.37 ms   (jitter: 0.45ms, low: 1.68ms, high: 4.29ms)
 Packet Loss:     0.0%

Speedtest on Server 2 that is connected to port enp3s0f1np1:

 speedtest --server-id=35058

   Speedtest by Ookla

      Server: Clouvider Ltd - Amsterdam (id: 35058)
         ISP:
 Idle Latency:     1.88 ms   (jitter: 0.02ms, low: 1.85ms, high: 1.90ms)
    Download:  7631.48 Mbps (data used: 10.4 GB)
                  2.38 ms   (jitter: 0.36ms, low: 1.65ms, high: 4.20ms)
      Upload:   278.13 Mbps (data used: 300.8 MB)
                  1.65 ms   (jitter: 0.09ms, low: 1.59ms, high: 2.24ms)
 Packet Loss:     0.0%

Netplan of the router:

network:
  renderer: networkd
  version: 2
  ethernets:
    eno6:
      dhcp4: no
    enp3s0f0np0:
      dhcp4: no
    enp3s0f1np1:
      dhcp4: no
      addresses:
      - 2001:12d:fe8::/48
  vlans:
    vlan2223:
      id: 2223
      link: enp3s0f0np0
      addresses: [117.17.18.225/29, 2a02:60:0:6::101:10/112]
      dhcp6: no
      accept-ra: no
    vlan9:
      id: 9
      link: enp3s0f0np0
      addresses: [93.239.11.221/22, 2001:8f1:23::a504:8325:1/64]
      dhcp6: no
      accept-ra: no

/etc/sysctl.conf

net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
net.ipv4.icmp_echo_ignore_all=0
net.ipv4.tcp_syn_retries=3
net.ipv4.tcp_synack_retries=3
net.core.somaxconn=65535
net.core.rmem_default=3407872
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_max_syn_backlog=8192
net.core.netdev_max_backlog=30000
net.core.netdev_max_backlog=250000
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 87380 67108864
fs.file-max=12000500
fs.nr_open=1073741816
vm.max_map_count=1048575
net.ipv4.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh3=4096
net.ipv6.neigh.default.gc_thresh1=1024
net.ipv6.neigh.default.gc_thresh2=2048
net.ipv6.neigh.default.gc_thresh3=4096
net.ipv6.route.max_size=1073741824
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_ra=0
net.ipv4.tcp_congestion_control=bbr
net.core.default_qdisc=fq
net.ipv4.ip_forward=1
net.core.netdev_budget=1000
net.core.netdev_max_backlog=1000

Bird

protocol static {
        ipv6;
        route 2001:12d:fe8::/48 via "enp3s0f1np1";
}