Fixing 70% transfer speed slowdown over VPN compared as compared to SSH? (Tested OpenVPN, IPSec/L2TP, OpenSSH, SoftEther)

Question

I'm currently configuring a site-to-site VPN between two VPSes that requires both outgoing and incoming connections. It will be used by a high bandwidth application so I require the maximum amount of speed that I can get over the connection.

Using iperf3, I'm getting about 600Mbit/s over the connection reliably, with ~30ms ping.

Over OpenSSH SCP, I get about 260Mbit/s, which I'm happy with given the additional encryption.

I have been trying various kinds of configurations of VPNs, mostly with OpenVPN. I have tried sndbuf/rcvbuf changes, no encryption, no compression, but I still only generally get 20Mbit with UDP, 40Mbit with TLS on port 443.

I have also set up IPSec/L2TP, SoftEther (though I got only like 500Kbit/s with that), and OpenSSH built-in tun adapter. None of these have been able to give me an iperf speed above 40Mbit/s.

I have been closely watching the HDD and CPU of each node, and neither have been saturated. One server is significantly less powerful, but only ever hits ~30% CPU usage during the test.

I am kind of at a loss. I need something that can achieve the speeds above 200Mbit/s (which I know for sure is possible), and just needs to route from one virtual interface to another. Theoretically, this is what SoftEther is for. Should I continue to try to fix SoftEther to get any kind of actual speeds?

Any suggestions for more aspects to test/debug/configure to try to get a solid tunnel interface up and running? Is there another piece of software that will help route the incoming connections so I can use a normal proxy-like tunnel? Thanks!

Answer 1

Given the link posted in the comments, I decided to take another look at this openvpn article: https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

I was able to achieve ~150Mbit/s average using a couple of the settings from this article. Here are the steps I took to configure my OpenVPN to achieve this.

These are the steps I tried in order:

• Default Settings (w/ AES-128): 14.7Mbit/s
• Enable mssfix 0, tun-mtu 6000, fragment 0: 16.3Mbit/s
• Setting 'cipher none': 17.7Mbit/s
• Back to AES-128, tun-mtu 9000: 22.0Mbit/s
• tun-mtu 18000: 27.2Mbit/s
• tun-mtu 36000: 37.2Mbit/s
• tun-mtu 60000: 44.9Mbit/s
• Setting 'cipher BF-CBC' (not much difference): 44.0Mbit/s
• sndbuf 393216, rcvbuf 393216 on server and client: 67.1Mbit/s
• Increased size of Linux UDP Recv buffers: 102Mbit/s
• iperf3 [...] -P 10 to enable 10 parallel connections: SUM: 135Mbit/s - 170Mbit/s

Here is a final iperf3 output:

64 bytes from 10.8.0.1: icmp_seq=1 ttl=64 time=37.2 ms

$ iperf3 -c 10.8.0.1 -p 5201
Connecting to host 10.8.0.1, port 5201
[  4] local 10.8.0.2 port 59230 connected to 10.8.0.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  10.3 MBytes  86.5 Mbits/sec    2    632 KBytes
[  4]   1.00-2.00   sec  12.0 MBytes   101 Mbits/sec    7    569 KBytes
[  4]   2.00-3.00   sec  11.4 MBytes  95.8 Mbits/sec    7    443 KBytes
[  4]   3.00-4.00   sec  10.9 MBytes  91.2 Mbits/sec    5    443 KBytes
[  4]   4.00-5.00   sec  11.7 MBytes  98.4 Mbits/sec    2    759 KBytes
[  4]   5.00-6.00   sec  13.0 MBytes   109 Mbits/sec    6    822 KBytes
[  4]   6.00-7.00   sec  13.5 MBytes   113 Mbits/sec    5    696 KBytes
[  4]   7.00-8.00   sec  13.9 MBytes   117 Mbits/sec    6    696 KBytes
[  4]   8.00-9.00   sec  11.8 MBytes  98.9 Mbits/sec    5    696 KBytes
[  4]   9.00-10.00  sec  12.5 MBytes   105 Mbits/sec    4    696 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   121 MBytes   102 Mbits/sec   49             sender
[  4]   0.00-10.00  sec   120 MBytes   101 Mbits/sec                  receiver

Compared to non-VPN direct connection:

$ iperf3 -c [...] -p 5201
Connecting to host [...], port 5201
[  4] local [...] port 52172 connected to [...] port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  51.9 MBytes   435 Mbits/sec    0   3.03 MBytes
[  4]   1.00-2.00   sec  73.3 MBytes   615 Mbits/sec    0   3.03 MBytes
[  4]   2.00-3.00   sec  73.3 MBytes   615 Mbits/sec    0   3.03 MBytes
[  4]   3.00-4.00   sec  72.7 MBytes   610 Mbits/sec    0   3.03 MBytes
[  4]   4.00-5.00   sec  72.1 MBytes   605 Mbits/sec    0   3.03 MBytes
[  4]   5.00-6.00   sec  73.7 MBytes   619 Mbits/sec    0   3.03 MBytes
[  4]   6.00-7.00   sec  75.0 MBytes   629 Mbits/sec    0   3.03 MBytes
[  4]   7.00-8.00   sec  72.5 MBytes   608 Mbits/sec    0   3.03 MBytes
[  4]   8.00-9.00   sec  74.9 MBytes   628 Mbits/sec    0   3.03 MBytes
[  4]   9.00-10.00  sec  72.6 MBytes   609 Mbits/sec    0   3.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   712 MBytes   597 Mbits/sec    0             sender
[  4]   0.00-10.00  sec   710 MBytes   596 Mbits/sec                  receiver

What I learned

• My network benefited significantly from the increased tun-mtu.
• UDP Congestion is a huge problem. Increasing the Linux UDP recvbuf significantly improved UDP performance. The retransmitted packets in iperf show that congestion is still an issue. Any suggestions to improve are appreciated.
• Parallel iperf3 connections helped boost speed even further.
• Ciphers didn't really affect performance much as long as the CPU core stayed below 100%.
• It's difficult to configure a VPN over the open internet that achieves gigabit speeds.
• This is still 1/6th of the performance of the plain network.