All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Struggling with OpenVPN performance only with 1 server (fastest and nearest one)
Hello
As my name suggests, I'm running a small VPN service for Apple devices. I'm using Debian 12 and OpenVPN 2.6.11 with DCO enabled. I use the exact same configurations on all of my servers which I've thoroughly optimized over the months (MTU etc.). I'm running several instances with different ports (UDP 1194, UDP 80, TCP 443). On the TCP instance I additionally have configured XOR obfuscation. I also use TLS crypt v2 on all instances.
My home connection in Switzerland has about 320 mbit/s down and 100 mbit/s upload. On most of my VPN servers I reach almost my full speed (around 310 mbit/s over UDP and 290 mbit/s over TCP). But with just this one provider (Infomaniak) I'm struggling. Ironically it's the nearest (located in Switzerland) and the fastest one (10 gbit/s, high CPU performance). YABS shows almost double of the network speeds compared to my other servers. Also using speedtest directly on the server I get about 7-8 gbit/s.
The problem is, while using TCP I also reach around 290 mbit/s, but using UDP only about 160-170 mbit/s and for the hell out of it I can't figure out why. I tried everything (different MTU values server and client side, played with mssfix, played with all the other network variables like rmem wmem etc.) but no difference. The performance problem is only when connecting to external servers through the VPN like speedtest or downloading files. If I download directly something from the server speed is fine, also if I run iperf3 through my the tunnel speed is also fine. Again, this is only through UDP instances, TCP is perfectly fine like on my other servers.
Any hints? Is there anything special with 10 gbit/s servers? How to resolve the problem?
Thanks
Comments
that's fast enough anyway.
@TarZZ92 not for me lol
Yeah, fast enough
maybe they filter incoming udp traffic
How’s your cpu usage during speedtest? That might be the bottleneck
Pretty much this. Pushing 100's of mbits through OpenVPN needs quite a bit of CPU power. I find it odd that UDP would need more of it than TCP but who knows? If you are expecting multiple gbits i hope you have some serious CPU available and if it's somehow single core bound you'll probably run into problems one way or another.
Something messing with UDP traffic would be another option. I'm not really sure how you'd test a high volume UDP stream (of seemingly random data - with modern filters you can't be sure that everything UDP will behave the same) for loss/latency but there's probably some kind of ping tool that would do it. In theory it might already just be enough that packets arrive (at your server) badly out of order to impact performance though, which would be even harder to test for.
Ports used might play a role too. If you are for example using port 53 for (UDP) OpenVPN traffic chances are you'll experience weird stuff.
@zakkuuno
@totally_not_banned
CPU usage is less than 10%, as written I use DCO which does offload a lot and uses multitasking, that‘s why OpenVPN can be that fast.
If running iperf through the tunnel with UDP I get my 310 mbit/s without hesitation, just external servers have a performance impact. On TCP both is fine.
Might point towards external UDP being filtered in some way. Like i said this might be affected even by stuff like packets arriving out of order.
@totally_not_banned I see, but where should the filtering happen? Between me and my server can‘t be, otherwise iperf (using TCP, through the UDP tunnel) would also struggle, but it is perfectly fine. Also when using speedtest on my server (which uses TCP anyway), blazing fast. But same speedtest server from my client through the UDP tunnel has the impact. Makes no sense to me.
At the provider of the server.
No, filters don't necessarily treat packets equally just because they use the same protocol.
Edit: OK, i see what you mean. Well at least TCP might change stuff in relation to packet ordering.
Well, something is slowing down OpenVPN's UDP traffic but not (some?) other UDP traffic.
@totally_not_banned
But what can distinguishe between iperf through UDP OpenVPN and speedtest to external server through UDP OpenVPN? In both cases it‘s the same encrypted UDP OpenVPN tunnel, same port, same protocol, everything.
EDIT: I‘m gonna play with iptables today, maybe there‘s something wrong or to optimize. Still doesn‘t explain why it‘s fine on my other (much slower) servers.
Yeah, i see. I misunderstood what you meant (sorry, i should read more carefully). Well, like i said in my edit TCP might have an impact on how packet ordering and related windows are handled. There's also the part where TCP figures out the optimal packetsize on it's own (yeah, you played around with MTU but maybe TCP somehow figures out a better value?). Beyond that i also don't really see what's the difference there.
Edit: I don't know if iperf cares for ordering with UDP. If it does it might be interesting to test with it disabled. You could also run tcpdump to check what happens around MSS, window size and other variable TCP parameters.
Edit2: I think there was another guy a couple weeks ago that had a pretty similar problem but i don't think the thread led to any solution.
Just out of curiosity, why OpenVPN? Wireguard's performance is way better.
@yoyek Easier to implement, integrated obfuscation options, TCP supported, and with DCO performance is on par with WireGuard.
Yeah, while i might be biased due to working with OpenVPN for years i also feel that it's way more straight forward than Wireguard. Sure, there's more configuration and all that but Wireguard just seems like some kind of behind-your-back-blackbox to me.
I have a bit of hope that NetBSD's Wireguard implementation fares a bit better in this regard but i didn't get to try it yet. In any case the interaction they had with the Wireguard author is quite remarkable. Not that it changes anything about the software but if it would this exchange alone would basically kill Wireguard for me.
It's really not that bad! Sure, the initial setup is a bit tricky, but after that it's really easy and clear.
Setting up another wg server is matter of few minutes for me now without any external scripts.
Not really my impression. While setup is extremely easy i pretty much miss the clear part. With OpenVPN almost everything is explicitly configurable. Wireguard just does what it likes best.
Well, i'd never use scripts to setup VPNs in general. That's actually kinda how Wireguard feels to me though: Something that would appeal to people whose main concern is to get things running as fast as possible.
I'm just trying to setup WireGuard now to check if it has same or different behaviour, but I didn't succeed yet lol...
Got WireGuard up and running, exactly the same behaviour. So at least now I know that it's not OpenVPN related.
do you try contacting your provider?
@budi1413 Not yet, I suspect they would blame it to OpenVPN configuration and complexity etc. But I just found out that if I use iperf3 in UDP mode directly on the server (testing to a public server), I also get the performance issue... I was only using iperf3 in TCP mode to test. That perfectly replicates the problem, gonna open a ticket now.
This is what it looks like:
root@vpn-ch1:~# iperf3 --udp -b0 -f m -c ping.online.net -p 5208
Connecting to host ping.online.net, port 5208
[ 5] local xxx.xxx.xxx.xxx port 57791 connected to 51.158.1.21 port 5208
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 31.1 MBytes 261 Mbits/sec 22550
[ 5] 1.00-2.00 sec 33.2 MBytes 278 Mbits/sec 24020
[ 5] 2.00-3.00 sec 30.0 MBytes 252 Mbits/sec 21760
[ 5] 3.00-4.00 sec 33.1 MBytes 278 Mbits/sec 24000
[ 5] 4.00-5.00 sec 31.0 MBytes 260 Mbits/sec 22430
[ 5] 5.00-6.00 sec 31.5 MBytes 264 Mbits/sec 22820
[ 5] 6.00-7.00 sec 30.4 MBytes 255 Mbits/sec 22000
[ 5] 7.00-8.00 sec 29.7 MBytes 249 Mbits/sec 21530
[ 5] 8.00-9.00 sec 29.5 MBytes 247 Mbits/sec 21330
[ 5] 9.00-10.00 sec 32.3 MBytes 271 Mbits/sec 23400
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 312 MBytes 262 Mbits/sec 0.000 ms 0/225840 (0%) sender
[ 5] 0.00-10.05 sec 312 MBytes 260 Mbits/sec 0.133 ms 0/225792 (0%) receiver
iperf Done.
root@vpn-ch1:~# iperf3 -b0 -f m -c ping.online.net -p 5208
Connecting to host ping.online.net, port 5208
[ 5] local xxx.xxx.xxx.xxx port 52182 connected to 51.158.1.21 port 5208
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 131 MBytes 1096 Mbits/sec 763 3.32 MBytes
[ 5] 1.00-2.00 sec 159 MBytes 1332 Mbits/sec 0 3.48 MBytes
[ 5] 2.00-3.00 sec 158 MBytes 1321 Mbits/sec 0 3.61 MBytes
[ 5] 3.00-4.00 sec 159 MBytes 1332 Mbits/sec 0 3.72 MBytes
[ 5] 4.00-5.00 sec 161 MBytes 1353 Mbits/sec 0 3.75 MBytes
[ 5] 5.00-6.00 sec 158 MBytes 1321 Mbits/sec 0 3.75 MBytes
[ 5] 6.00-7.00 sec 159 MBytes 1332 Mbits/sec 0 3.75 MBytes
[ 5] 7.00-8.00 sec 141 MBytes 1185 Mbits/sec 935 2.71 MBytes
[ 5] 8.00-9.00 sec 146 MBytes 1227 Mbits/sec 0 2.85 MBytes
[ 5] 9.00-10.00 sec 146 MBytes 1227 Mbits/sec 0 2.96 MBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.48 GBytes 1272 Mbits/sec 1698 sender
[ 5] 0.00-10.04 sec 1.48 GBytes 1268 Mbits/sec receiver
iperf Done.