New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Ubuntu 22.04 slow IPv6 forwarding
jordynegen11
Member
Hello,
Im working on a little experiment where i'm using a ubuntu 22.04 server as my BGP router at home. Currently only have IPv6 addresses announced to it. However I run into an issue where I only get 20-300Mbit upload speed for devices that are connected to port enp3s0f1np1, while the router itself get's a decent 8-9Gbit upload.
Steps I tried:
- Update the drivers if the NIC
- Update the firmware of the NIC (Mellanox ConnectX-5)
- Replaced the NIC
- Installed everything again on a different server.
- Updated the kernel
- Downgraded the kernel
- Poked arround with the sysctl settings
What am I missing here?
Below are speedtest results but we see the same behaviour in iPerf.
Speedtest on the router (no problems here):
speedtest --server-id=35058
Speedtest by Ookla
Server: Clouvider Ltd - Amsterdam (id: 35058)
ISP:
Idle Latency: 1.86 ms (jitter: 0.04ms, low: 1.81ms, high: 1.93ms)
Download: 7865.62 Mbps (data used: 6.4 GB)
2.58 ms (jitter: 3.39ms, low: 1.83ms, high: 210.35ms)
Upload: 9057.08 Mbps (data used: 6.6 GB)
2.37 ms (jitter: 0.45ms, low: 1.68ms, high: 4.29ms)
Packet Loss: 0.0%
Speedtest on Server 2 that is connected to port enp3s0f1np1:
speedtest --server-id=35058
Speedtest by Ookla
Server: Clouvider Ltd - Amsterdam (id: 35058)
ISP:
Idle Latency: 1.88 ms (jitter: 0.02ms, low: 1.85ms, high: 1.90ms)
Download: 7631.48 Mbps (data used: 10.4 GB)
2.38 ms (jitter: 0.36ms, low: 1.65ms, high: 4.20ms)
Upload: 278.13 Mbps (data used: 300.8 MB)
1.65 ms (jitter: 0.09ms, low: 1.59ms, high: 2.24ms)
Packet Loss: 0.0%
Netplan of the router:
network:
renderer: networkd
version: 2
ethernets:
eno6:
dhcp4: no
enp3s0f0np0:
dhcp4: no
enp3s0f1np1:
dhcp4: no
addresses:
- 2001:12d:fe8::/48
vlans:
vlan2223:
id: 2223
link: enp3s0f0np0
addresses: [117.17.18.225/29, 2a02:60:0:6::101:10/112]
dhcp6: no
accept-ra: no
vlan9:
id: 9
link: enp3s0f0np0
addresses: [93.239.11.221/22, 2001:8f1:23::a504:8325:1/64]
dhcp6: no
accept-ra: no
/etc/sysctl.conf
net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
net.ipv4.icmp_echo_ignore_all=0
net.ipv4.tcp_syn_retries=3
net.ipv4.tcp_synack_retries=3
net.core.somaxconn=65535
net.core.rmem_default=3407872
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_max_syn_backlog=8192
net.core.netdev_max_backlog=30000
net.core.netdev_max_backlog=250000
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 87380 67108864
fs.file-max=12000500
fs.nr_open=1073741816
vm.max_map_count=1048575
net.ipv4.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh3=4096
net.ipv6.neigh.default.gc_thresh1=1024
net.ipv6.neigh.default.gc_thresh2=2048
net.ipv6.neigh.default.gc_thresh3=4096
net.ipv6.route.max_size=1073741824
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_ra=0
net.ipv4.tcp_congestion_control=bbr
net.core.default_qdisc=fq
net.ipv4.ip_forward=1
net.core.netdev_budget=1000
net.core.netdev_max_backlog=1000
Bird
protocol static {
ipv6;
route 2001:12d:fe8::/48 via "enp3s0f1np1";
}
Comments
Does the "server 2" use bbr as well?
One more thing, try disabling Flow Control on the NICs of both servers.
You are amazing. Disable flow control worked!