Need some help with openvpn tunnel that stalls

hacktek

Folks,

I'm running a no auth, no cipher OpenVPN server with 2 tap interfaces and using the linux bonding driver for redundancy purposes. Everything worked great when I was on a 10 Mbps link but I've upgraded it to 25 Mbps each and now I'm having issues. They appear to me like mtu problems but I can't seem to find a setting that works (tried tun-mtu/fragment/mssfix to no avail). Here's how performance looks over the tunnel with tun-mtu 1460 and mssfix 1400:

root@artemis:~# iperf -c 10.10.0.2

Client connecting to 10.10.0.2, TCP port 5001

TCP window size: 45.0 KByte (default)

[ 3] local 10.10.0.1 port 44316 connected with 10.10.0.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-11.7 sec 14.1 MBytes 10.1 Mbits/sec

Ew.

Now, let's set client and server to:

tun-mtu 6000
fragment 0
mssfix 0

And also equalize the bond interface's mtu and do some other optimizations:

ifconfig bond0 mtu 6000
ifconfig bond0 txqueuelen 10000
echo 3000 > /proc/sys/net/core/netdev_max_backlog

root@artemis:~# iperf -c 10.10.0.2

Client connecting to 10.10.0.2, TCP port 5001

TCP window size: 165 KByte (default)

[ 3] local 10.10.0.1 port 44317 connected with 10.10.0.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 29.8 MBytes 24.8 Mbits/sec

Nice!

But then I go out to the internet and the above clearly breaks:

root@raspberrypi:~# ./speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Velocihost (x.x.x.x)...
Selecting best server based on latency...
Hosted by AT&T (Miami, FL) [4.23 km]: 84.123 ms
Testing download speed........................................
Download: 13.03 Mbit/s

A throughput graph looks like a rollercoaster, whenever it's reaching speed it stalls and falls flat only to go up again. Same thing happens when I don't tweak mtu.

Ideas?

tommy

I still didn't get it, what broken? what's not working?

where's your openvpn server?

hacktek

I am unable to max out the connection going out to the internet through the tunnel, it tends to hit around 10-15 Mbps before stalling.

Look at pass 9 download how it started stalling and slowly dying:

http://s.speedof.me/result/2016/06/28/160628054048-8372.png

Another example with passes 8 and 9 stalling:

http://s.speedof.me/result/2016/06/28/160628054212-797.png

Server's in Miami.

hacktek

Nothing?

hacktek

Wanted to follow up on this and in case it helps someone. I took a tcpdump from the client, bond interfaces (both sides), tap interfaces (both sides), ethernet interfaces (both sides). I saw retransmissions start to creep up at the bond and tap interfaces. This pointed to either an issue with the bond driver or just hitting some sort of KVM performance wall. I took the bond device out of the picture and configured openvpn point-to-point directly using both tap and tun interfaces and the performance didn't change so that appeared to point the finger at KVM. I then tried direct openvpn tun tunnels in an OpenVZ VPS and got absolutely perfect performance with no mtu tweaks. This further seems to confirm that KVM is the problem.

Here's an iperf over the tunnel interface from the VPS:

[ 3] 0.0-10.1 sec 29.1 MBytes 24.3 Mbits/sec

And here's a speedtest going out to the internet from the pi:

Hosted by Comcast (Atlanta, GA) [1122.80 km]: 85.357 ms Testing download speed........................................ Download: 23.53 Mbit/s Testing upload speed.................................................. Upload: 2.45 Mbit/s

MikePT

Do you really need OpenVPN? I've experienced the same.

Disable encryption, if needed, compression, etc.

hacktek

Any other suggestions other than openvpn? I do run it with no encryption to make it faster. I tried l2tp/ipsec and l2tp/xauth at some point but the encryption kills performance. It's also way more difficult to set up and maintain.

MikeA

@hacktek said:
Any other suggestions other than openvpn? I do run it with no encryption to make it faster. I tried l2tp/ipsec and l2tp/xauth at some point but the encryption kills performance. It's also way more difficult to set up and maintain.

I've had lots of clients who have used or currently use Softether, they claim the performance is much better than OpenVPN. There's also a comparison of features/speed limits on the Softether website.

Never used it myself though, probably will soon.

hacktek

@MikeA said:

@hacktek said:
Any other suggestions other than openvpn? I do run it with no encryption to make it faster. I tried l2tp/ipsec and l2tp/xauth at some point but the encryption kills performance. It's also way more difficult to set up and maintain.

I've had lots of clients who have used or currently use Softether, they claim the performance is much better than OpenVPN. There's also a comparison of features/speed limits on the Softether website.

Never used it myself though, probably will soon.

Oh nice. I'll check it out.

hacktek

I want to re-follow up on this. It doesn't seem like KVM should be having this perf issue, the provider has confirmed this. I even tried a different distro with a different linux kernel, no dice. If I run UDP over the tunnel, I can see pretty consistent performance:

`root@raspberrypi:~# vnstat -i tun1 -l
Monitoring tun1... (press CTRL-C to stop)

rx: 22.91 Mbit/s 1957 p/s tx: 4 kbit/s`

But tcp acts all weird. If I measure throughput now I'm hitting some sort of wall at ~13 Mbps:

[ 3] 0.0-10.1 sec 16.2 MBytes 13.5 Mbits/sec

If I measure bandwidth with 10 connections I see an inordinate amount of retransmissions when throughput exceeds 15 Mbps.

I already messed with send and receive buffers and nothing has changed.

For shts and giggles, here's the same test from a different provider with both TCP and UDP working fine:

TCP: [ 3] 0.0-10.2 sec 29.8 MBytes 24.6 Mbits/sec
UDP: [ 3] 0.0-10.0 sec 29.8 MBytes 25.0 Mbits/sec

The above is using the exact same openvpn setup (used a script to set up servers and clients).

I've never been so frustrated in my life.