30

I am experiencing extremely slow OpenVPN transfer rates between two servers. For this question, I'll call the servers Server A and Server B.

Both Server A and Server B are running CentOS 6.6. Both are located in datacenters with a 100Mbit line and data transfers between the two servers outside of OpenVPN run close to ~88Mbps.

However, when I attempt to transfer any files over the OpenVPN connection I've established between Server A and Server B, I get throughput right around 6.5Mbps.

Test results from iperf:

[  4] local 10.0.0.1 port 5001 connected with 10.0.0.2 port 49184
[  4]  0.0-10.0 sec  7.38 MBytes  6.19 Mbits/sec
[  4]  0.0-10.5 sec  7.75 MBytes  6.21 Mbits/sec
[  5] local 10.0.0.1 port 5001 connected with 10.0.0.2 port 49185
[  5]  0.0-10.0 sec  7.40 MBytes  6.21 Mbits/sec
[  5]  0.0-10.4 sec  7.75 MBytes  6.26 Mbits/sec

Aside from these OpenVPN iperf tests, both servers are virtually completely idle with zero load.

Server A is assigned the IP 10.0.0.1 and it is the OpenVPN server. Server B is assigned the IP 10.0.0.2 and it is the OpenVPN client.

The OpenVPN configuration for Server A is as follows:

port 1194
proto tcp-server
dev tun0
ifconfig 10.0.0.1 10.0.0.2
secret static.key
comp-lzo
verb 3

The OpenVPN configuration for Server B is as follows:

port 1194
proto tcp-client
dev tun0
remote 204.11.60.69
ifconfig 10.0.0.2 10.0.0.1
secret static.key
comp-lzo
verb 3

What I've noticed:

1. My first thought was that I was bottlenecking the CPU on the server. OpenVPN is single-threaded and both of these servers run Intel Xeon L5520 processors which aren't the fastest. However, I ran a top command during one of the iperf tests and pressed 1 to view CPU utilization by core and found that the CPU load was very low on each core:

top - 14:32:51 up 13:56,  2 users,  load average: 0.22, 0.08, 0.06
Tasks: 257 total,   1 running, 256 sleeping,   0 stopped,   0 zombie
Cpu0  :  2.4%us,  1.4%sy,  0.0%ni, 94.8%id,  0.3%wa,  0.0%hi,  1.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu3  :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    946768k total,   633640k used,   313128k free,    68168k buffers
Swap:  4192188k total,        0k used,  4192188k free,   361572k cached

2. Ping times increase considerably over the OpenVPN tunnel while iperf is running. When iperf is not running, ping times over the tunnel are consistently 60ms (normal). But when iperf is running and pushing heavy traffic, ping times become erratic. You can see below how the ping times are stable until the 4th ping when I've started the iperf test:

PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=60.1 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=60.1 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=60.2 ms
** iperf test begins **
64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=146 ms
64 bytes from 10.0.0.2: icmp_seq=5 ttl=64 time=114 ms
64 bytes from 10.0.0.2: icmp_seq=6 ttl=64 time=85.6 ms
64 bytes from 10.0.0.2: icmp_seq=7 ttl=64 time=176 ms
64 bytes from 10.0.0.2: icmp_seq=8 ttl=64 time=204 ms
64 bytes from 10.0.0.2: icmp_seq=9 ttl=64 time=231 ms
64 bytes from 10.0.0.2: icmp_seq=10 ttl=64 time=197 ms
64 bytes from 10.0.0.2: icmp_seq=11 ttl=64 time=233 ms
64 bytes from 10.0.0.2: icmp_seq=12 ttl=64 time=152 ms
64 bytes from 10.0.0.2: icmp_seq=13 ttl=64 time=216 ms

3. As mentioned above, I ran iperf outside of the OpenVPN tunnel and the throughput was normal -- ~88Mbps consistently.

What I've tried:

1. I thought compression might be fouling things up, so I turned off compression by removing comp-lzo from both configs and restarting OpenVPN. No improvement.

2. Even though I previously found that the CPU utilization was low, I thought the default cipher might be a little too intensive for the system to keep up with. So I added cipher RC2-40-CBC to both configs (a very lightweight cipher) and restarted OpenVPN. No improvement.

3. I read on various forum about how tweaking the fragment, mssfix and mtu-tun might help with performance. I played with a few variations as described in this article, but again, no improvement.

Any ideas on what could be causing such poor OpenVPN performance?

5
  • Do any links, comments from here help? forums.openvpn.net/topic10593.html
    – hookenz
    Apr 28, 2015 at 21:54
  • Most of the suggestions in there are things that I've already tried: 1. check for CPU bottleneck, 2. verify transfer speeds without VPN, 3. toggle compression, 4. choose a faster cipher, etc. No luck with any of them yet :-/ It's bizarre. Aside from the slow speed and high/volatile ping times, I see no other indication of where the bottleneck might be.
    – Elliot B.
    Apr 28, 2015 at 22:04
  • Are both machines in the same datacenter? 60ms within the same data center is pretty high. I might be tempted to try cipher none though I doubt it will help.
    – Zoredache
    Apr 28, 2015 at 23:18
  • @Zoredache Sorry -- I wasn't clear about the locations of the servers -- server A is in Dallas and server B is in Seattle.
    – Elliot B.
    Apr 28, 2015 at 23:22
  • Have you checked MTU? Esp: tun-mtu, fragment and mssfix parameters? Documentation
    – Lenniey
    Apr 29, 2015 at 7:22

5 Answers 5

28

After a lot of Googling and configuration file tweaks, I found the solution. I'm now getting sustained speeds of 60Mbps and burst up to 80Mbps. It's a bit slower than the transfer rates I receive outside the VPN, but I think this is as good as it'll get.

The first step was to set sndbuf 0 and rcvbuf 0 in the OpenVPN configuration for both the server and the client.

I made that change after seeing a suggestion to do so on a public forum post (which is an English translation of a Russian original post) that I'll quote here:

It's July, 2004. Usual home internet speed in developed countries is 256-1024 Kbit/s, in less developed countries is 56 Kbit/s. Linux 2.6.7 has been released not a long ago and 2.6.8 where TCP Windows Size Scaling would be enabled by default is released only in a month. OpenVPN is in active development for 3 years already, 2.0 version is almost released. One of the developers decides to add some code for socket buffer, I think to unify buffer sizes between OSes. In Windows, something goes wrong with adapters' MTU if custom buffers sizes are set, so finally it transformed to the following code:

#ifndef WIN32
o->rcvbuf = 65536;
o->sndbuf = 65536;
#endif

If you used OpenVPN, you should know that it can work over TCP and UDP. If you set custom TCP socket buffer value as low as 64 KB, TCP Window Size Scaling algorithm can't adjust Window Size to more than 64 KB. What does that mean? That means that if you're connecting to other VPN site over long fat link, i.e. USA to Russia with ping about 100 ms, you can't get speed more than 5.12 Mbit/s with default OpenVPN buffer settings. You need at least 640 KB buffer to get 50 Mbit/s over that link. UDP would work faster because it doesn't have window size but also won't work very fast.

As you already may guess, the latest OpenVPN release still uses 64 KB socket buffer size. How should we fix this issue? The best way is to disallow OpenVPN to set custom buffer sizes. You should add the following code in both server and client config files:

sndbuf 0
rcvbuf 0

The author goes on to describe how to push buffer size adjustments to the client if you are not in control of the client config yourself.

After I made those changes, my throughput rate bumped up to 20Mbps. I then saw that CPU utilization was a little high on a single core so I removed comp-lzo (compression) from the configuration on both the client and server. Eureka! Transfer speeds jumped up to 60Mbps sustained and 80Mbps burst.

I hope this helps someone else resolve their own issues with OpenVPN slowness!

4

After some tries, I've reached a good solution. In my case, @Elliot's reply didn't helped. Googling more, I found out this snippet to add in server's configuration that made the job

sndbuf 393216
rcvbuf 393216
push "sndbuf 393216"
push "rcvbuf 393216"

I have a little OpenVPN server running on a Raspberry PI3 and now I get 71 Mbps downlink and 16Mbps uplink. Download is limited since CPU's power. Right now, my configuration is the following:

client-to-client
duplicate-cn
keepalive 10 120
cipher AES-128-CBC
#cipher AES-256-CBC <<<---- lowers the speed to around 50Mbps, still not bad
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
tun-mtu 9000

OpenVPN 2.4.0 arm-unknown-linux-gnueabihf with OpenSSL 1.0.2l

It feels so weird that such a problem about the default configuration of a buffer still exists.

[EDIT] My client.ovpn file is structured like this:

client
dev tun
proto tcp
remote SERVER.IP.ADDRESS.HERE
resolv-retry infinite
nobind
persist-key
persist-tun
mute-replay-warnings
ns-cert-type server
tun-mtu 9000
key-direction 1
cipher AES-128-CBC
comp-lzo
verb 1
mute 20
<ca>
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN RSA PRIVATE KEY-----
[...]
-----END RSA PRIVATE KEY-----
</key>
<tls-auth>
-----BEGIN OpenVPN Static key V1-----
[...]
-----END OpenVPN Static key V1-----
</tls-auth>
5
  • Setting buffer sizes helped me.
    – Rolf
    May 11, 2018 at 20:45
  • what did you put in the client .ovpn file? Jan 13, 2019 at 2:34
  • @Patoshiパトシ I've commented out sndbuf and recbuf, put the according cipher and compression and left the default parameters.
    – Kernel
    Jan 14, 2019 at 10:44
  • @Kernel can you show me what you have in your client? I'm doing an OpenVPN connection from Hong Kong to NYC and its randomly slow and sometimes disconnects. I'm not sure why. Jan 14, 2019 at 20:27
  • @Patoshiパトシ I've edited my reply, check it again. Nonetheless, I would suggest you to try to use UDP instead, since it could help you addressing the problem of an unstable link with your server. Indeed, it's just an assumption, I've never tried to benchmark this solution in this situation.
    – Kernel
    Jan 16, 2019 at 10:04
2

We have two intercontinental servers that link to each other, the speeds between them hovering around 220 Mbit/s.

Inside the (UDP) OpenVPN tunnel however, speeds would average at 21 Mbit/s - roughly 10x slower.

(There is a significant latency between the servers: around 130 ms, and the transfers were measured using Iperf3 in TCP mode.)

Tried all the suggestions on answers here as of this writing, and nothing helped.

The one thing that finally did help was this little bit:

--txqueuelen 4000

According to the OpenVPN reference manual:

–txqueuelen n 
(Linux only) Set the TX queue length on the TUN/TAP interface. Currently defaults to 100.

After setting this parameter on the server and the client, I was able to reach the same 'direct-link' speeds (~250Mbit/s) also under the OpenVPN tunnel.

I was already using the rcvbuf 0 and sndbuf 0, but at least alone, they didn't help at all.

I've found these recommendations in both: this page in the OpenVPN forums, and also in this page in the UDPspeeder wiki.

On another note: I was able to reach higher speeds using UDP transfers in iperf, but doing so would also incur a reasonably high packet-loss.

If by some chance you need to use VPN to tunnel two places with lossy links, I'd advise to consider using some sort of Forward-Error-Correction (FEC) tunnel under the VPN itself. The two I've managed to find and work with are:

  • The aforementioned UDPspeeder, which tunnels UDP connections;
  • kcptun, which tunnels TCP connections;

Both can help a lot with packetloss (by spending more bandwidth in the first place), and ultimately even leading to higher data throughput, even with the added overhead, which is really neat if you ask me.

(That's because packet-loss can really mess up a network, specially TCP. See page 6.)

I would have preferred using OpenVPN on UDP, for all the usual reasons, but I've found it difficult to deal with UDPspeeder when you have both more than 100ms latency & >10 Mbit/s speeds.

kcptun however, worked great with very little tweaking, and actually really increased our servers throughput with each other. =)

On an extended note, here you can find some more detailed explanations about tweaking some parts of OpenVPN performance.

1

According to the Config's you are using TCP as transport for the Tunnel. Consider using UDP instead of TCP since the stacked TCP connections tent to create problems in packet loss situations.

As reference see Why TCP Over TCP Is A Bad Idea

6
  • Unfortunately UDP isn't an option for us. We need to ensure that the data packets we transmit arrive as expected. Nevertheless we did experiment with UDP earlier on and the poor transfer rates were still a problem.
    – Elliot B.
    Apr 28, 2015 at 22:41
  • 7
    We need to ensure that the data packets we transmit arrive as expected. and isn't that handled by the protocol that is being tunneled? Why do you think your tunnel needs to be the thing that enforces that?
    – Zoredache
    Apr 28, 2015 at 23:14
  • That's probably the case, but we're using OpenVPN for a long-distance asynchronous DRBD implementation (i.e., file-system replication). Data integrity is really important, so even though DRBD probably has internal mechanisms for verifying transfer integrity, I'd rather keep it on TCP. Either way, when we had it on UDP we still had the low throughput.
    – Elliot B.
    Apr 28, 2015 at 23:25
  • 4
    @ElliotB. As DRBD itself uses TCP for replication, it will retransmit in case OpenVPN UDP packet is lost. Actually using TCP in this case you will do two retransfers instead of one ... one of which is going to get thrown away. And you may be creating a pretty long window with no DRBD traffic (even get broken replication because of this). Once you get some packetloss on the route, you will see how bad thinking this is.
    – Fox
    Apr 29, 2015 at 22:04
  • @Fox Thank you for providing clarification! Indeed DRBD does use TCP ( drbd.linbit.com/users-guide/s-prepare-network.html ). Lairsdragon earlier suggestion to switch to UDP wasn't relevant at the time because UDP was also experiencing extremely low transfer rates, but since implementing the solution I posted above we did make the switch to UDP and experienced another performance gain of a few Mbps.
    – Elliot B.
    Apr 30, 2015 at 2:49
0

For me I had a VPS server with openvpn server setup in Japan and my client connection was using a DDWRT in OpenVPN client mode in NYC. I was getting only 1-2mbps on a 100mbit connection. The best I was able to optimize it for was 5mbps which was enough for what I needed which is as optimized as I can make it I believe.

My OpenVPN server settings:

tun-mtu 9000
sndbuf 393216
rcvbuf 393216
push "sndbuf 393216"
push "rcvbuf 393216"
comp-lzo
txqueuelen 4000
######
port 10111
proto udp
dev tun
user nobody
group nobody
persist-key
persist-tun
keepalive 10 120
topology subnet
server x.x.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 1.0.0.1"
push "dhcp-option DNS 1.1.1.1"
push "redirect-gateway def1 bypass-dhcp"
dh none
ecdh-curve prime256v1
#tls-crypt tls-crypt.key 0
crl-verify crl.pem
ca ca.crt
cert server_IzA1QdFzHLRFfEoQ.crt
key server_IzA1QdFzHLRFfEoQ.key
auth SHA256
#cipher AES-128-GCM
#cipher AES-128-CBC
#ncp-ciphers AES-128-GCM
#ncp-ciphers AES-128-CBC
#tls-server
#tls-version-min 1.2
#tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
#tls-cipher TLS-DHE-RSA-WITH-AES-128-CBC-SHA
status /var/log/openvpn/status.log
verb 3

My DDWRT OpenVPN client settings also seen in my screenshot:

tun-mtu 9000
comp-lzo
##########
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
verify-x509-name server_IzA1QdFzHLRFfEoQ name
auth SHA256
auth-nocache
setenv opt block-outside-dns # Prevent Windows 10 DNS leak
verb 3

enter image description here

enter image description here

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .