OpenVPN Tunnel not working for subnets after some time

Question

okay this sounds weird, I have one OpenVPN Server with 2 clients, 1 client works fine, the second one drops out after about 1 day.

The server is 10.7.2.1, the client 1 10.7.2.2 (not working) and client 2 10.7.2.3.

The internal networks for each of those are:

server (10.7.2.1):
10.0.96.0/24

client 1 (10.7.2.2):
10.21.0.0/24
192.168.0.0/24

client 2 (10.7.2.3):
10.22.0.0/24

So after the day it's impossible to ping from say 192.168.0.3 to 10.0.96.253:

ping 10.0.96.253
PING 10.0.96.253 (10.0.96.253) 56(84) bytes of data.
^C
--- 10.0.96.253 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7590ms

I am sure the packets are correctly routed:

10.0.96.0/24 via 10.7.2.1 dev tun1 
10.7.2.0/24 dev tun1  proto kernel  scope link  src 10.7.2.2 
10.22.0.0/24 via 10.7.2.1 dev tun1 

and can also verify via tcpdump:

[Interface:eth0:]    10:41:37.446320 IP 192.168.0.3 > 10.0.96.253: ICMP echo request, id 20316, seq 4, length 64
[Interface:tun1:]    10:41:37.446332 IP 192.168.0.3 > 10.0.96.253: ICMP echo request, id 20316, seq 4, length 64
[Interface:eth0:]    10:41:38.446967 IP 192.168.0.3 > 10.0.96.253: ICMP echo request, id 20316, seq 5, length 64
[Interface:tun1:]    10:41:38.446983 IP 192.168.0.3 > 10.0.96.253: ICMP echo request, id 20316, seq 5, length 64

tun1 is the network interface on client 1.

32: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none 
    inet 10.7.2.2/24 brd 10.7.2.255 scope global tun1
       valid_lft forever preferred_lft forever

When doing a tcpdump on the server side, nothing shows up!

BUT it does work if I ping directly from the client:

ping 10.0.96.253
PING 10.0.96.253 (10.0.96.253) 56(84) bytes of data.
64 bytes from 10.0.96.253: icmp_seq=1 ttl=63 time=31.8 ms
64 bytes from 10.0.96.253: icmp_seq=2 ttl=63 time=31.3 ms
64 bytes from 10.0.96.253: icmp_seq=3 ttl=63 time=32.9 ms
64 bytes from 10.0.96.253: icmp_seq=4 ttl=63 time=31.7 ms
64 bytes from 10.0.96.253: icmp_seq=5 ttl=63 time=32.2 ms
64 bytes from 10.0.96.253: icmp_seq=6 ttl=63 time=38.4 ms
64 bytes from 10.0.96.253: icmp_seq=7 ttl=63 time=32.8 ms
^C
--- 10.0.96.253 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6008ms
rtt min/avg/max/mdev = 31.331/33.060/38.428/2.264 ms

Also visible in tcpdump

[Interface:tun1:]    10:44:41.828197 IP 10.7.2.2 > 10.0.96.253: ICMP echo request, id 51791, seq 5, length 64
[Interface:tun1:]    10:44:41.860386 IP 10.0.96.253 > 10.7.2.2: ICMP echo reply, id 51791, seq 5, length 64

After a reconnect of openvpn via systemctl restart openvpn@samba it works again!

Again from 192.168.0.3 to 10.0.96.253:

ping 10.0.96.253
PING 10.0.96.253 (10.0.96.253) 56(84) bytes of data.
64 bytes from 10.0.96.253: icmp_seq=1 ttl=62 time=33.9 ms
64 bytes from 10.0.96.253: icmp_seq=2 ttl=62 time=31.4 ms
64 bytes from 10.0.96.253: icmp_seq=3 ttl=62 time=32.0 ms
64 bytes from 10.0.96.253: icmp_seq=4 ttl=62 time=32.9 ms
^C
--- 10.0.96.253 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3275ms
rtt min/avg/max/mdev = 31.496/32.589/33.915/0.947 ms

and the corresponding tcpdump:

[Interface:eth0:]    10:47:25.230275 IP 192.168.0.3 > 10.0.96.253: ICMP echo request, id 31580, seq 4, length 64
[Interface:tun1:]    10:47:25.230301 IP 192.168.0.3 > 10.0.96.253: ICMP echo request, id 31580, seq 4, length 64
[Interface:tun1:]    10:47:25.262723 IP 10.0.96.253 > 192.168.0.3: ICMP echo reply, id 31580, seq 4, length 64
[Interface:eth0:]    10:47:25.262739 IP 10.0.96.253 > 192.168.0.3: ICMP echo reply, id 31580, seq 4, length 64

The second client doesn't show this behaviour. So I'm quite stumped and don't know where to look, anyone any idea what I could do?

Server Config:

port                  1194
proto                 udp
dev                   tun
ca                    samba.ca.crt
cert                  samba.crt
key                   samba.key
dh                    samba.dh.pem
server                10.7.2.0 255.255.255.0
ifconfig-pool-persist samba.ipp.txt
route                 10.0.96.0 255.255.255.0
route                 192.168.0.0 255.255.255.0 10.7.2.2
route                 10.21.0.0 255.255.255.0 10.7.2.2
route                 10.22.0.0 255.255.255.0 10.7.2.3
topology              subnet
mode                  server

client-to-client

client-config-dir     samba.ccd
ccd-exclusive         samba.ccd
comp-lzo
persist-key
persist-tun
status                samba.openvpn-status.log
log-append            /var/log/samba.openvpn.log
verb                  4
keepalive             10 120

Server OpenVPN Version:

OpenVPN 2.3.8 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug  4 2015

Client Config:

client
dev              tun
proto            udp
remote           xx.xx.xx.xx 1194
resolv-retry     infinite
nobind
persist-key
persist-tun
ca               samba.ca.crt
cert             samba.crt
key              samba.key
comp-lzo
verb             4
status           samba.openvpn-status.log
log-append       /var/log/samba.openvpn.log

Client CCD:

client-to-client
iroute 10.21.0.0 255.255.255.0
iroute 192.168.0.0 255.255.255.0
push "route 10.0.96.0 255.255.255.0"
push "route 10.22.0.0 255.255.255.0"

Client OpenVPN Version:

OpenVPN 2.3.8 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug  4 2015