All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Clouvider: Slow overlay / wireguard networks between VMs
Hi all,
I have 3 Clouvider VMs in their NYC region (AMD Epyc) and I wanted to run a Kubernetes/K3S cluster on them but I am having performance issues with any kind of overlay / encapsulated (vxlan) network for the pods.
The issue also seems to affect simple Wireguard/Tailscale connections.
Peformance (using iperf3):
VM -> Clouvider LDN via YABS ~9Gbit/s
VM -> VM on same physical node: ~9 GBit/s
VM -> VM on different physical node: ~6 GBit/s
VM -wireguard-> VM: ~0.6 GBit/s
POD -> VM: ~6 GBit/s
POD -overlay-> POD: ~0.5 GBit/s
I've tried different OSs and versions including Ubuntu 22 and 24, I've tried different overlay networks (flannel, cilium, kilo) and they all end up in the same ballpark.
The VMs are otherwise idle and have 4 cores / 8GB RAM, so it is not a resource issue.
The dropoff in performance seems too high, I have a couple of low end hetzner VMs which can reach 5 Gbit/s natively, and at least 1 Gbit/s on wireguard.
Do any of you run an overlay or wireguard network on Clouvider VMs and if so what kind of performance do you get?
Comments
Try playing with sysctl. Brr usually helps. Everything related to udp should be fine tuned for maximum perf. Also, personally I do not recomend, but fidle with MTU.
Sounds like MTU issue. Tunnel does not negotiate MTU unlike TCP connections
I have confirmed with Clouvider support that it should have an mtu of 1500
The tailscale and flannel interfaces seem to have appropriate mtus to allow for overhead.
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 100
3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
Thanks for the reply, I've added BRR on both nodes and confirmed that applied with restart for good measure.
Unfortunately, it hasn't made any difference to the performance. I will take a look at potential udp rated tweaks for sysctl.
I second the suspicion that MTU is at fault. Bad MTU can fuck you royaly... I'd play around even if those might look correct. You can use ping with fragmentation disabled to figure out what MTU an interface (most interestingly ens3) will really take. The mere fact that it's set to 1500 says pretty much nothing in regards to 1500 actually working as something along the path might push it down.
I'm learning new things but I didn't get much joy with this.
ens3
I was able to ping upto 1472 with no packet loss or fragmentation, which should be effective mtu of 1500
ping -c 10 -M do -s 1472 -I ens3
any higher and i get the error
ping: local error: message too long, mtu=1500
I tried changing MTU upwards on both vms that i'm testing, but it timed out for higher than 1472.
Also, I can get high speeds between VMs on this interface.
tailscale0
I was able to ping without fragmentation up to the mtu - overhead, so 1252. I experimented with increasing the mtu, and this did increase the speed in iperf from ~500Mbits to ~700Mbits but with a lot of retransmissions.
flannel.1
I was again able to ping without fragmentation up to the mtu - overhead (1422). I experimented with reducing the mtu and this slowed the speed in iperf slightly.
Tailscale mtu 1280
root@nyc0:~# iperf3 -c 100.98.193.41 -t 10
Connecting to host 100.98.193.41, port 5201
[ 5] local 100.106.76.40 port 59080 connected to 100.98.193.41 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 55.2 MBytes 463 Mbits/sec 1 812 KBytes
[ 5] 1.00-2.00 sec 53.8 MBytes 451 Mbits/sec 0 863 KBytes
[ 5] 2.00-3.00 sec 52.5 MBytes 440 Mbits/sec 442 581 KBytes
[ 5] 3.00-4.00 sec 55.0 MBytes 461 Mbits/sec 1176 655 KBytes
[ 5] 4.00-5.00 sec 53.8 MBytes 451 Mbits/sec 78 751 KBytes
[ 5] 5.00-6.00 sec 53.8 MBytes 451 Mbits/sec 101 670 KBytes
[ 5] 6.00-7.00 sec 55.0 MBytes 461 Mbits/sec 47 626 KBytes
[ 5] 7.00-8.00 sec 52.5 MBytes 440 Mbits/sec 0 520 KBytes
[ 5] 8.00-9.00 sec 55.0 MBytes 461 Mbits/sec 0 621 KBytes
[ 5] 9.00-10.00 sec 53.8 MBytes 451 Mbits/sec 18 534 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 540 MBytes 453 Mbits/sec 1863 sender
[ 5] 0.00-10.05 sec 536 MBytes 448 Mbits/sec receiver
Tailscale mtu 1330
root@nyc0:~# sudo ip link set dev tailscale0 mtu 1330
root@nyc0:~# iperf3 -c 100.98.193.41
Connecting to host 100.98.193.41, port 5201
[ 5] local 100.106.76.40 port 51810 connected to 100.98.193.41 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 87.0 MBytes 729 Mbits/sec 2227 569 KBytes
[ 5] 1.00-2.00 sec 83.8 MBytes 703 Mbits/sec 1517 524 KBytes
[ 5] 2.00-3.00 sec 83.8 MBytes 703 Mbits/sec 2171 784 KBytes
[ 5] 3.00-4.00 sec 85.0 MBytes 713 Mbits/sec 1402 643 KBytes
[ 5] 4.00-5.00 sec 85.0 MBytes 713 Mbits/sec 2282 414 KBytes
[ 5] 5.00-6.00 sec 83.8 MBytes 703 Mbits/sec 2278 508 KBytes
[ 5] 6.00-7.00 sec 82.5 MBytes 692 Mbits/sec 1774 437 KBytes
[ 5] 7.00-8.00 sec 86.2 MBytes 724 Mbits/sec 962 502 KBytes
[ 5] 8.00-9.00 sec 82.5 MBytes 692 Mbits/sec 1715 482 KBytes
[ 5] 9.00-10.00 sec 83.8 MBytes 703 Mbits/sec 1724 452 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 843 MBytes 707 Mbits/sec 18052 sender
[ 5] 0.00-10.04 sec 840 MBytes 701 Mbits/sec receiver
Apologies for all the text here, but I've noticed i do get almost the opposite results with udp with native vs tailscale. Is that normal?
TCP: node to node over ens3
* root@nyc0:~# iperf3 -c nyc1
* Connecting to host nyc1, port 5201
* [ 5] local nyc0 port 53324 connected to nyc1 port 5201
* [ ID] Interval Transfer Bitrate Retr Cwnd
* [ 5] 0.00-1.00 sec 696 MBytes 5.84 Gbits/sec 42 1.33 MBytes
* [ 5] 1.00-2.00 sec 712 MBytes 5.98 Gbits/sec 0 662 KBytes
* [ 5] 2.00-3.00 sec 742 MBytes 6.23 Gbits/sec 0 701 KBytes
* [ 5] 3.00-4.00 sec 658 MBytes 5.52 Gbits/sec 0 656 KBytes
* [ 5] 4.00-5.00 sec 784 MBytes 6.57 Gbits/sec 0 716 KBytes
* [ 5] 5.00-6.00 sec 766 MBytes 6.43 Gbits/sec 24 831 KBytes
* [ 5] 6.00-7.00 sec 779 MBytes 6.53 Gbits/sec 13 662 KBytes
* [ 5] 7.00-8.00 sec 729 MBytes 6.11 Gbits/sec 0 645 KBytes
* [ 5] 8.00-9.00 sec 678 MBytes 5.68 Gbits/sec 55 634 KBytes
* [ 5] 9.00-10.00 sec 759 MBytes 6.36 Gbits/sec 11 738 KBytes
* - - - - - - - - - - - - - - - - - - - - - - - - -
* [ ID] Interval Transfer Bitrate Retr
* [ 5] 0.00-10.00 sec 7.13 GBytes 6.13 Gbits/sec 145 sender
* [ 5] 0.00-10.04 sec 7.12 GBytes 6.09 Gbits/sec receiver
UDP: node to node over ens3
root@nyc0:~# iperf3 -c nyc1 --udp -b 4000M
Connecting to host nyc1, port 5201
[ 5] local nyc0 port 47601 connected to nyc1 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 72.2 MBytes 606 Mbits/sec 52300
[ 5] 1.00-2.00 sec 71.8 MBytes 602 Mbits/sec 51995
[ 5] 2.00-3.00 sec 73.0 MBytes 613 Mbits/sec 52894
[ 5] 3.00-4.00 sec 73.4 MBytes 616 Mbits/sec 53136
[ 5] 4.00-5.00 sec 73.5 MBytes 616 Mbits/sec 53205
[ 5] 5.00-6.00 sec 73.9 MBytes 620 Mbits/sec 53520
[ 5] 6.00-7.00 sec 73.7 MBytes 618 Mbits/sec 53365
[ 5] 7.00-8.00 sec 74.0 MBytes 621 Mbits/sec 53597
[ 5] 8.00-9.00 sec 71.8 MBytes 602 Mbits/sec 51989
[ 5] 9.00-10.00 sec 72.8 MBytes 611 Mbits/sec 52740
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 730 MBytes 612 Mbits/sec 0.000 ms 0/528741 (0%) sender
[ 5] 0.00-10.04 sec 705 MBytes 588 Mbits/sec 0.019 ms 18266/528476 (3.5%) receiver
TCP: node to node over tailscale
root@nyc0:~# iperf3 -c nyc1-tailscale
Connecting to host nyc1-tailscale, port 5201
[ 5] local 100.106.76.40 port 50490 connected to 100.98.193.41 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 58.7 MBytes 492 Mbits/sec 0 684 KBytes
[ 5] 1.00-2.00 sec 51.2 MBytes 430 Mbits/sec 0 748 KBytes
[ 5] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 133 638 KBytes
[ 5] 3.00-4.00 sec 61.2 MBytes 514 Mbits/sec 7 839 KBytes
[ 5] 4.00-5.00 sec 62.5 MBytes 524 Mbits/sec 0 650 KBytes
[ 5] 5.00-6.00 sec 63.8 MBytes 535 Mbits/sec 0 696 KBytes
[ 5] 6.00-7.00 sec 57.5 MBytes 482 Mbits/sec 0 513 KBytes
[ 5] 7.00-8.00 sec 52.5 MBytes 440 Mbits/sec 0 607 KBytes
[ 5] 8.00-9.00 sec 55.0 MBytes 461 Mbits/sec 89 1.06 MBytes
[ 5] 9.00-10.00 sec 62.5 MBytes 524 Mbits/sec 0 820 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 581 MBytes 488 Mbits/sec 229 sender
[ 5] 0.00-10.05 sec 576 MBytes 481 Mbits/sec receiver
UDP: node to node over tailscale
root@nyc0:~# iperf3 -c nyc1-tailscale --udp -b 4000M
Connecting to host nyc1-tailscale, port 5201
[ 5] local 100.106.76.40 port 41465 connected to 100.98.193.41 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 214 MBytes 1.79 Gbits/sec 182693
[ 5] 1.00-2.00 sec 213 MBytes 1.78 Gbits/sec 181664
[ 5] 2.00-3.00 sec 205 MBytes 1.72 Gbits/sec 175374
[ 5] 3.00-4.00 sec 199 MBytes 1.67 Gbits/sec 170151
[ 5] 4.00-5.00 sec 201 MBytes 1.69 Gbits/sec 171835
[ 5] 5.00-6.00 sec 203 MBytes 1.71 Gbits/sec 173728
[ 5] 6.00-7.00 sec 202 MBytes 1.70 Gbits/sec 172766
[ 5] 7.00-8.00 sec 195 MBytes 1.64 Gbits/sec 166667
[ 5] 8.00-9.00 sec 198 MBytes 1.66 Gbits/sec 168766
[ 5] 9.00-10.00 sec 211 MBytes 1.77 Gbits/sec 179813
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.99 GBytes 1.71 Gbits/sec 0.000 ms 0/1743457 (0%) sender
[ 5] 0.00-10.26 sec 589 MBytes 482 Mbits/sec 0.010 ms 1229126/1732241 (71%) receiver
Actually, speeds are the same within tailscale tunnel