I have OpenVPN Access Server running on AWS. Here's the configuration:
172.18.16.0/20 Client (172.18.16.101) ----- OpenVPN Server (172.16.0.0/20) | | | | Private Subnet (172.16.16.0/20)
The client can connect to the OpenVPN server. The OpenVPN server can ping and access resources on both the client and also on the private subnet. The client can also access all the resources on the OpenVPN server as well as the private subnet. Additionally, the OpenVPN server can also access resources on the client. However, the boxes on the private subnet can't seem to ping or access resources on the client at all.
OpenVPN Server Network Config:
as0t0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:172.18.0.1 P-t-P:172.18.0.1 Mask:255.255.248.0 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:18 errors:0 dropped:0 overruns:0 frame:0 TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:200 RX bytes:1223 (1.2 KB) TX bytes:968 (968.0 B) as0t1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:172.18.8.1 P-t-P:172.18.8.1 Mask:255.255.248.0 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:200 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr 06:e2:83:cf:4f:27 inet addr:172.16.12.204 Bcast:172.16.15.255 Mask:255.255.240.0 inet6 addr: fe80::4e2:83ff:fecf:4f27/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1 RX packets:1355 errors:0 dropped:0 overruns:0 frame:0 TX packets:1193 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:124194 (124.1 KB) TX bytes:153022 (153.0 KB)
OpenVPN Server's Route Table:
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 eth0 172.16.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0 172.18.0.0 0.0.0.0 255.255.248.0 U 0 0 0 as0t0 172.18.8.0 0.0.0.0 255.255.248.0 U 0 0 0 as0t1 172.18.16.101 0.0.0.0 255.255.255.255 UH 0 0 0 as0t0
A Box on the Private Subnet's Route Table:
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.16.16.1 0.0.0.0 UG 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.16.16.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
I ran tcpdump on the OpenVPN server box's as0t1 interface while pinging from a box on the private subnet to the client. It looks like the requests was forwarded to the client, and the client also sent back responses (seen on the OpenVPN box). However, it seems that it stops there and the responses never got put back on eth0. I feel that the routing table should be correct as the OpenVPN server can reach everything and the client can also reach the private subnet. I thought the issue was at iptables, but after spending hours looking at it, I am starting to give up. Here's the tables:
Filter Table
Chain INPUT (policy ACCEPT) target prot opt source destination AS0_ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED AS0_ACCEPT all -- anywhere anywhere AS0_IN_PRE all -- anywhere anywhere mark match 0x2000000/0x2000000 AS0_ACCEPT udp -- anywhere anywhere state NEW udp dpt:openvpn AS0_ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:https AS0_WEBACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED AS0_WEBACCEPT tcp -- anywhere anywhere state NEW tcp dpt:943 Chain FORWARD (policy ACCEPT) target prot opt source destination AS0_ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED AS0_IN_PRE all -- anywhere anywhere mark match 0x2000000/0x2000000 AS0_OUT_S2C all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination AS0_OUT_LOCAL all -- anywhere anywhere Chain AS0_ACCEPT (5 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain AS0_IN (3 references) target prot opt source destination ACCEPT icmp -- anywhere ip-172-18-0-1.ap-southeast-1.compute.internal icmp echo-request ACCEPT icmp -- anywhere ip-172-18-8-1.ap-southeast-1.compute.internal icmp echo-request AS0_U_TMBDLP_IN all -- ip-172-18-16-101.ap-southeast-1.compute.internal anywhere AS0_IN_POST all -- anywhere anywhere Chain AS0_IN_NAT (1 references) target prot opt source destination MARK all -- anywhere anywhere MARK or 0x8000000 ACCEPT all -- anywhere anywhere Chain AS0_IN_POST (2 references) target prot opt source destination ACCEPT all -- anywhere ip-172-16-0-0.ap-southeast-1.compute.internal/16 AS0_OUT all -- anywhere anywhere DROP all -- anywhere anywhere Chain AS0_IN_PRE (2 references) target prot opt source destination AS0_IN all -- anywhere ip-192-168-0-0.ap-southeast-1.compute.internal/16 AS0_IN all -- anywhere ip-172-16-0-0.ap-southeast-1.compute.internal/12 AS0_IN all -- anywhere ip-10-0-0-0.ap-southeast-1.compute.internal/8 DROP all -- anywhere anywhere Chain AS0_IN_ROUTE (0 references) target prot opt source destination MARK all -- anywhere anywhere MARK or 0x4000000 ACCEPT all -- anywhere anywhere Chain AS0_OUT (2 references) target prot opt source destination AS0_U_TMBDLP_OUT all -- anywhere ip-172-18-16-101.ap-southeast-1.compute.internal AS0_OUT_POST all -- anywhere anywhere Chain AS0_OUT_LOCAL (1 references) target prot opt source destination DROP icmp -- anywhere anywhere icmp redirect ACCEPT all -- anywhere anywhere Chain AS0_OUT_POST (2 references) target prot opt source destination DROP all -- anywhere anywhere Chain AS0_OUT_S2C (1 references) target prot opt source destination AS0_OUT all -- anywhere anywhere Chain AS0_U_TMBDLP_IN (1 references) target prot opt source destination AS0_IN_NAT all -- anywhere ip-172-16-0-0.ap-southeast-1.compute.internal/16 AS0_IN_POST all -- anywhere anywhere Chain AS0_U_TMBDLP_OUT (1 references) target prot opt source destination ACCEPT all -- ip-172-16-0-0.ap-southeast-1.compute.internal/16 anywhere ACCEPT all -- ip-172-18-0-0.ap-southeast-1.compute.internal/20 anywhere ACCEPT all -- ip-172-18-16-0.ap-southeast-1.compute.internal/20 anywhere AS0_OUT_POST all -- anywhere anywhere Chain AS0_WEBACCEPT (2 references) target prot opt source destination ACCEPT all -- anywhere anywhere
NAT Table
Chain PREROUTING (policy ACCEPT) target prot opt source destination AS0_NAT_PRE_REL_EST all -- anywhere anywhere state RELATED,ESTABLISHED Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination AS0_NAT_POST_REL_EST all -- anywhere anywhere state RELATED,ESTABLISHED AS0_NAT_PRE all -- anywhere anywhere mark match 0x2000000/0x2000000 Chain AS0_NAT (3 references) target prot opt source destination MASQUERADE all -- anywhere anywhere Chain AS0_NAT_POST_REL_EST (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain AS0_NAT_PRE (1 references) target prot opt source destination AS0_NAT all -- anywhere anywhere mark match 0x8000000/0x8000000 AS0_NAT_TEST all -- anywhere ip-192-168-0-0.ap-southeast-1.compute.internal/16 AS0_NAT_TEST all -- anywhere ip-172-16-0-0.ap-southeast-1.compute.internal/12 AS0_NAT_TEST all -- anywhere ip-10-0-0-0.ap-southeast-1.compute.internal/8 AS0_NAT all -- anywhere anywhere Chain AS0_NAT_PRE_REL_EST (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain AS0_NAT_TEST (3 references) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere mark match 0x4000000/0x4000000 ACCEPT all -- anywhere ip-172-18-0-0.ap-southeast-1.compute.internal/20 ACCEPT all -- anywhere ip-172-18-16-0.ap-southeast-1.compute.internal/20 AS0_NAT all -- anywhere anywhere
I am almost sure that the issue is in the iptables rules, but I cannot seem to track it down. Just for some extra fun, I am also throwing the output of iptables trace (captured on the OpenVPN box) here.
Box on Private Subnet Accessing MySQL Server on the Client (failed):
Aug 28 17:42:33 localhost kernel: [21906.075591] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) Aug 28 17:42:33 localhost kernel: [21906.075603] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) Aug 28 17:42:33 localhost kernel: [21906.075608] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) Aug 28 17:42:33 localhost kernel: [21906.075616] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) Aug 28 17:42:33 localhost kernel: [21906.075620] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) Aug 28 17:42:33 localhost kernel: [21906.075625] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) Aug 28 17:42:33 localhost kernel: [21906.075629] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.115440] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.115452] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.115458] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.115466] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.115470] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.115475] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.115480] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) Aug 28 17:42:34 localhost kernel: [21907.175467] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) Aug 28 17:42:34 localhost kernel: [21907.175474] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) Aug 28 17:42:34 localhost kernel: [21907.175479] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) Aug 28 17:42:34 localhost kernel: [21907.175486] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) Aug 28 17:42:34 localhost kernel: [21907.175490] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) Aug 28 17:42:34 localhost kernel: [21907.175494] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) Aug 28 17:42:34 localhost kernel: [21907.175498] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) Aug 28 17:42:36 localhost kernel: [21909.077995] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) Aug 28 17:42:36 localhost kernel: [21909.078007] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) Aug 28 17:42:36 localhost kernel: [21909.078012] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) Aug 28 17:42:36 localhost kernel: [21909.078021] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) Aug 28 17:42:36 localhost kernel: [21909.078025] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) Aug 28 17:42:36 localhost kernel: [21909.078030] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) Aug 28 17:42:36 localhost kernel: [21909.078034] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) Aug 28 17:42:36 localhost kernel: [21909.287922] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) Aug 28 17:42:36 localhost kernel: [21909.287932] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) Aug 28 17:42:36 localhost kernel: [21909.287937] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) Aug 28 17:42:36 localhost kernel: [21909.287945] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) Aug 28 17:42:36 localhost kernel: [21909.287949] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) Aug 28 17:42:36 localhost kernel: [21909.287954] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) Aug 28 17:42:36 localhost kernel: [21909.287958] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) Aug 28 17:42:40 localhost kernel: [21913.165296] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) Aug 28 17:42:40 localhost kernel: [21913.165308] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) Aug 28 17:42:40 localhost kernel: [21913.165313] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) Aug 28 17:42:40 localhost kernel: [21913.165321] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) Aug 28 17:42:40 localhost kernel: [21913.165326] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) Aug 28 17:42:40 localhost kernel: [21913.165330] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) Aug 28 17:42:40 localhost kernel: [21913.165335] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) Aug 28 17:42:40 localhost kernel: [21913.285405] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) Aug 28 17:42:40 localhost kernel: [21913.285414] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) Aug 28 17:42:40 localhost kernel: [21913.285419] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) Aug 28 17:42:40 localhost kernel: [21913.285427] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) Aug 28 17:42:40 localhost kernel: [21913.285431] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) Aug 28 17:42:40 localhost kernel: [21913.285435] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) Aug 28 17:42:40 localhost kernel: [21913.285440] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) Aug 28 17:42:48 localhost kernel: [21921.097914] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) Aug 28 17:42:48 localhost kernel: [21921.097926] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) Aug 28 17:42:48 localhost kernel: [21921.097931] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) Aug 28 17:42:48 localhost kernel: [21921.097939] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) Aug 28 17:42:48 localhost kernel: [21921.097943] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) Aug 28 17:42:48 localhost kernel: [21921.097948] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) Aug 28 17:42:48 localhost kernel: [21921.097964] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) Aug 28 17:42:49 localhost kernel: [21921.527746] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2530 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026632383250188401030308) Aug 28 17:42:49 localhost kernel: [21921.527756] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2530 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026632383250188401030308) ##### the rest are cut due to exceeding number of characters allowed #####
Forwarding config
$ cat /proc/sys/net/ipv4/ip_forward 1
cat /proc/sys/net/ipv4/ip_forward
say? The traces you've got are incomplete; there should be something afterfilter:AS0_ACCEPT:return
entries in each case, showing the filter-table disposition in each case. Also, pcaps at the router would be useful.