24

I am configuring OpenVPN 2.3.6-1 on my Arch Linux server in order to encrypt SMB traffic over the public Internet. When I test the setup on one of my Linux virtual machine clients, I get the error: TLS Error: TLS handshake failed.

I quickly read (OpenVPN on OpenVZ TLS Error: TLS handshake failed (google suggested solutions not helping)) and tried to switch from the default UDP to TCP, but that only caused the client to repeatedly report that the connection timed out. I also tried disabling the cipher and TLS authentication, but that caused the server to fail with Assertion failed at crypto_openssl.c:523. In both instances, the required changes were made to both the client and server configurations.

I have been following the instructions at (https://wiki.archlinux.org/index.php/OpenVPN) to set up OpenVPN and the instructions at (https://wiki.archlinux.org/index.php/Create_a_Public_Key_Infrastructure_Using_the_easy-rsa_Scripts) to create the keys and certificates. The only deviations I have made from these instructions have been specifying my own computers' names and their corresponding key/certificate file names.

See also my original question about securing SMB traffic over the Internet: (Simple encryption for Samba shares)

Can anybody explain how I can solve this issue?

Details:

Server: Arch Linux (up to date) connected directly to gateway via ethernet cable. No iptables.

Client: Arch Linux (up to date) virtual machine on VirtualBox 4.3.28r100309 Windows 8.1 host, bridged network adapter. No iptables. Windows Firewall disabled.

Gateway: Port forwarding for port 1194 enabled, no firewall restrictions.

Here are the configuration files on the server and client, respectively. I created these according to the instructions on the Arch Wiki.

/etc/openvpn/server.conf (Non-comment lines only):

port 1194
proto udp
dev tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server-name.crt
key /etc/openvpn/server-name.key
dh /etc/openvpn/dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth /etc/openvpn/ta.key 0
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
verb 3

/etc/openvpn/client.conf (Non-comment lines only):

client
dev tun
proto udp
remote [my public IP here] 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client-name.crt
key /etc/openvpn/client-name.key
remote-cert-tls server
tls-auth /etc/openvpn/ta.key 1
comp-lzo
verb 3

Here are the outputs of running openvpn on the machines with the above configurations. I started the server first, then the client.

The output of openvpn /etc/openvpn/server.conf on the server:

Thu Jul 30 17:02:53 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec  2 2014
Thu Jul 30 17:02:53 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 17:02:53 2015 NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x.  Be aware that this might create routing conflicts if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
Thu Jul 30 17:02:53 2015 Diffie-Hellman initialized with 2048 bit key
Thu Jul 30 17:02:53 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 17:02:53 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 17:02:53 2015 ROUTE_GATEWAY 192.168.0.1/255.255.255.0 IFACE=enp5s0 HWADDR=##:##:##:##:##:##
Thu Jul 30 17:02:53 2015 TUN/TAP device tun0 opened
Thu Jul 30 17:02:53 2015 TUN/TAP TX queue length set to 100
Thu Jul 30 17:02:53 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Jul 30 17:02:53 2015 /usr/bin/ip link set dev tun0 up mtu 1500
Thu Jul 30 17:02:53 2015 /usr/bin/ip addr add dev tun0 local 10.8.0.1 peer 10.8.0.2
Thu Jul 30 17:02:53 2015 /usr/bin/ip route add 10.8.0.0/24 via 10.8.0.2
Thu Jul 30 17:02:53 2015 GID set to nobody
Thu Jul 30 17:02:53 2015 UID set to nobody
Thu Jul 30 17:02:53 2015 UDPv4 link local (bound): [undef]
Thu Jul 30 17:02:53 2015 UDPv4 link remote: [undef]
Thu Jul 30 17:02:53 2015 MULTI: multi_init called, r=256 v=256
Thu Jul 30 17:02:53 2015 IFCONFIG POOL: base=10.8.0.4 size=62, ipv6=0
Thu Jul 30 17:02:53 2015 IFCONFIG POOL LIST
Thu Jul 30 17:02:53 2015 Initialization Sequence Completed

The output of openvpn /etc/openvpn/client.conf on the client:

Thu Jul 30 21:03:02 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec  2 2014
Thu Jul 30 21:03:02 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/client-name.key' is group or others accessible
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/ta.key' is group or others accessible
Thu Jul 30 21:03:02 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 21:03:02 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 21:03:02 2015 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Thu Jul 30 21:03:02 2015 UDPv4 link local: [undef]
Thu Jul 30 21:03:02 2015 UDPv4 link remote: [AF_INET][my public IP here]:1194
Thu Jul 30 21:04:02 2015 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Thu Jul 30 21:04:02 2015 TLS Error: TLS handshake failed
Thu Jul 30 21:04:02 2015 SIGUSR1[soft,tls-error] received, process restarting
Thu Jul 30 21:04:02 2015 Restart pause, 2 second(s)
4
  • 2
    Your client never gets a response from the server at all. Either you have a firewall you forgot about, or your port forwarding isn't working. Jul 31, 2015 at 1:02
  • 3
    Do a packet sniff, like: tcpdump -ni eth0 udp and port 1194 on the server and ensure if packets are arriving. If they are there may be problem with firewall dropping packets, if no then most probably there is some problem with port forwarding on the router. You may do it on the router as well. Give a shot and try to use some higher port, it's not common but maybe your ISP messed something up, like port 11194/UDP or 53/UDP. Jul 31, 2015 at 9:55
  • 2
    Yep, it was the forwarding. I don't usually work with UDP, so I forgot to change the protocol when creating the rule. I'll post that as the answer.
    – Kyle
    Aug 1, 2015 at 0:12
  • 3
    If the packets show up in tcpdump on the server, is there a way to ensure that they arrive at openvpn properly?
    – Joost
    Dec 22, 2015 at 14:48

8 Answers 8

10

I had this problem as well.

Am using digitalocean provider for my server and the problem was with floating ip feature.

In order to fix that, you have to update openvpn config setting:

local <ip anchor>

ip anchor should be an ip adress gathered from ip addr command, see example: enter image description here

Credits to this post

1
  • I had this issue as well with a pfsense device. Adding the local <ip address of VPN server> entry in the server configuration fixed it. Note that for TCP this entry was not needed, it was only UDP to give the error. Jan 9, 2017 at 12:28
9

As suggested by Michael Hampton and Michal Sokolowski in the comments on my question, it was a problem with the port forwarding rule I created on my gateway. OpenVPN is configured to use UDP, and I forgot to switch from TCP to UDP on the gateway as I usually don't use that protocol. The forwarding rule now uses UDP, and my VPN is functional.

1

My current configuration would work on some countries but not others. I am suspecting that my current provider is blocking TLS handshake packet. Solution? Since I am the only one using that VPN I have switched to static key authentication which - in my case - proved to be super fast https://openvpn.net/index.php/open-source/documentation/miscellaneous/78-static-key-mini-howto.html

1

This happened to me due to trying some old, invalid certificates.

Trying a newly generated profile config fixed it.

1
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Dec 31, 2021 at 17:49
0

If it appears after updating the OS core. Or the incoming packets show up in tcpdump on the server, but still not works. Try a simple firewall disable/enable. Maybe somebody will help.

sudo ufw disable
sudo ufw enable
0

I was getting this problem due to a misconfigured default gateway on the server side. The OpenVPN server was getting the connection attempt from the client but the response was then being lost because it never reached the right router.

2
  • 1
    Do you remember where you had to change this ? Was this in etc/openvpn/server.conf ? Aug 9, 2019 at 11:54
  • I think something was wrong with the routing tables at the server side. Check with ip route.
    – lanoxx
    Aug 10, 2019 at 14:12
0

I just had this problem. On checking my .ovpn file I saw that the ?.ddns.net had been changed to an IP address, which is why it didn't connect. I changed the IP back to the ?.ddns.net address and it connected.

0

In my case, I was setting up a GrandStream IP Phone when I encountered this. The date on the phone was the culprit. I set the date manually, unplug and plug back the ethernet cable, it connected to the OpenVPN server successfully. The phone always boots with a wrong date because it could not get date time info from the network. This was happening for a particular router.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .