1

I've recently configured a Debian 9 server (Debian 4.9.130-2) to run as a lightweight server, running a series of Docker containers (nextcloud, sync, etc.) alongside basic services like ssh. Services are properly configured and running without issue: I can connect to ssh and docker containers from any device on my LAN without any apparent troubles. However, attempts to connect from outside the network aren't getting to the server. At this point, I'm trying to test whether the failure is on the router or the server, as both seem to be properly configured. Towards this end, I set up packet capture on my router and then made several attempted inbound connection attempts to the forwarded ports from a VPN.

Here's a snippet from wireshark which indicates (I think) that traffic is indeed getting redirected by the router.

First, packet capture at the router:

No.     Time           Source                Destination           Protocol Length Info
   2265 26.624915      196.52.84.12          87.75.107.144         TCP      80     54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326616015 TSecr=0 SACK_PERM=1
   2382 27.746737      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326617017 TSecr=0 SACK_PERM=1
   2470 28.626743      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326618019 TSecr=0 SACK_PERM=1
   2590 29.666995      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326619020 TSecr=0 SACK_PERM=1
   2688 30.687513      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326620023 TSecr=0 SACK_PERM=1
   2719 31.667451      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326621028 TSecr=0 SACK_PERM=1
   2868 33.696000      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326623032 TSecr=0 SACK_PERM=1
   3254 37.657240      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326627033 TSecr=0 SACK_PERM=1
   3861 45.658800      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326635033 TSecr=0 SACK_PERM=1
   4132 48.150464      196.52.84.12          87.75.107.144         TCP      80     57788 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326377039 TSecr=0 SACK_PERM=1
   4152 49.191512      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57788 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326378040 TSecr=0 SACK_PERM=1
   4207 50.160028      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57788 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326379041 TSecr=0 SACK_PERM=1
   4464 52.415812      196.52.84.12          87.75.107.144         TCP      80     57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326381262 TSecr=0 SACK_PERM=1
   4530 53.412326      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326382263 TSecr=0 SACK_PERM=1
   4631 54.373065      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326383263 TSecr=0 SACK_PERM=1
   4684 55.380093      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326384264 TSecr=0 SACK_PERM=1
   4779 56.420386      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326385265 TSecr=0 SACK_PERM=1
   4874 57.420881      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326386265 TSecr=0 SACK_PERM=1
   5161 59.374395      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326388265 TSecr=0 SACK_PERM=1
   5381 61.774499      196.52.84.12          87.75.107.144         TCP      80     [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326651102 TSecr=0 SACK_PERM=1

Packet capture on server:

No.     Time           Source                Destination           Protocol Length Info
  32179 24.444677474   196.52.84.12          192.168.1.208         TCP      78     54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326616015 TSecr=0 SACK_PERM=1
  33778 25.565718159   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326617017 TSecr=0 SACK_PERM=1
  35147 26.445497552   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326618019 TSecr=0 SACK_PERM=1
  36888 27.485382313   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326619020 TSecr=0 SACK_PERM=1
  38683 28.505695805   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326620023 TSecr=0 SACK_PERM=1
  40376 29.485394758   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326621028 TSecr=0 SACK_PERM=1
  43649 31.513421847   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326623032 TSecr=0 SACK_PERM=1
  50623 35.473792067   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326627033 TSecr=0 SACK_PERM=1
  65139 43.473176096   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326635033 TSecr=0 SACK_PERM=1
  69018 45.964529458   196.52.84.12          192.168.1.208         TCP      78     57788 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326377039 TSecr=0 SACK_PERM=1
  70816 47.004900826   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57788 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326378040 TSecr=0 SACK_PERM=1
  72718 47.973061039   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57788 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326379041 TSecr=0 SACK_PERM=1
  77788 50.228672533   196.52.84.12          192.168.1.208         TCP      78     57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326381262 TSecr=0 SACK_PERM=1
  80033 51.224501372   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326382263 TSecr=0 SACK_PERM=1
  82529 52.185037535   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326383263 TSecr=0 SACK_PERM=1
  84789 53.191738933   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326384264 TSecr=0 SACK_PERM=1
  87000 54.231741538   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326385265 TSecr=0 SACK_PERM=1
  88816 55.231936109   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326386265 TSecr=0 SACK_PERM=1
  92836 57.184892614   196.52.84.12          192.168.1.208         TCP      78     [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326388265 TSecr=0 SACK_PERM=1

Quick key to output:

  • 196.52.84.14 is an IP address assigned to my PC while connected to a VPN
  • 192.168.1.208 is the LAN IP address for the server
  • 87.75.107.144 is the WAN IP address on the router (obfuscated)
  • port 2202 is being forwarded to port 22 for ssh on that server via tcp and port 4003 is being forwarded to 443 on the server

Am I right in thinking that the router is behaving as it should and forwarding the packets (e.g. "retransmission")?

Firewall is as follows:

$ sudo iptables-save
# Generated by iptables-save v1.6.0 on Fri Mar 15 20:37:38 2019
*nat
:PREROUTING ACCEPT [3920:488137]
:INPUT ACCEPT [2997:321060]
:OUTPUT ACCEPT [2725:243307]
:POSTROUTING ACCEPT [2735:246173]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-931904c155b2 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 8181 -j MASQUERADE
-A POSTROUTING -s 172.18.0.3/32 -d 172.18.0.3/32 -p tcp -m tcp --dport 7878 -j MASQUERADE
-A POSTROUTING -s 172.18.0.4/32 -d 172.18.0.4/32 -p tcp -m tcp --dport 8686 -j MASQUERADE
-A POSTROUTING -s 172.18.0.5/32 -d 172.18.0.5/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.18.0.6/32 -d 172.18.0.6/32 -p tcp -m tcp --dport 8989 -j MASQUERADE
-A POSTROUTING -s 172.18.0.7/32 -d 172.18.0.7/32 -p tcp -m tcp --dport 4040 -j MASQUERADE
-A POSTROUTING -s 172.18.0.8/32 -d 172.18.0.8/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A POSTROUTING -s 172.18.0.8/32 -d 172.18.0.8/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-931904c155b2 -j RETURN
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 8181 -j DNAT --to-destination 172.18.0.2:8181
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 7878 -j DNAT --to-destination 172.18.0.3:7878
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 8686 -j DNAT --to-destination 172.18.0.4:8686
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 9001 -j DNAT --to-destination 172.18.0.5:9000
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 27021 -j DNAT --to-destination 172.18.0.6:8989
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 4040 -j DNAT --to-destination 172.18.0.7:4040
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 10001 -j DNAT --to-destination 172.18.0.8:8000
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 10000 -j DNAT --to-destination 172.18.0.8:80
COMMIT
# Completed on Fri Mar 15 20:37:38 2019
# Generated by iptables-save v1.6.0 on Fri Mar 15 20:37:38 2019
*filter
:INPUT ACCEPT [6374971:555022347]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [8882591:15858115582]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m comment --comment "Allow SSH" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m comment --comment "Allow HTTPS" -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-931904c155b2 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-931904c155b2 -j DOCKER
-A FORWARD -i br-931904c155b2 ! -o br-931904c155b2 -j ACCEPT
-A FORWARD -i br-931904c155b2 -o br-931904c155b2 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8181 -j ACCEPT
-A DOCKER -d 172.18.0.3/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 7878 -j ACCEPT
-A DOCKER -d 172.18.0.4/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8686 -j ACCEPT
-A DOCKER -d 172.18.0.5/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 9000 -j ACCEPT
-A DOCKER -d 172.18.0.6/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8989 -j ACCEPT
-A DOCKER -d 172.18.0.7/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 4040 -j ACCEPT
-A DOCKER -d 172.18.0.8/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.18.0.8/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-931904c155b2 ! -o br-931904c155b2 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-931904c155b2 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Mar 15 20:37:38 2019

Routing table:

$ ip route
0.0.0.0/1 via 10.1.10.9 dev tun0
default via 192.168.1.1 dev eno1 onlink
10.1.10.1 via 10.1.10.9 dev tun0
10.1.10.9 dev tun0 proto kernel scope link src 10.1.10.10
128.0.0.0/1 via 10.1.10.9 dev tun0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-931904c155b2 proto kernel scope link src 172.18.0.1
172.98.67.82 via 192.168.1.1 dev eno1
192.0.0.0/8 dev eno1 proto kernel scope link src 192.168.1.208
192.168.1.0/24 via 192.168.1.1 dev eno1
  • I recommend using the output of iptables-save instead of iptables -nvL. It's much more useful for troubleshooting iptables chaining issues. – pepoluan Mar 15 '19 at 20:31
  • That's really helpful - I've updated the post. – Jeremy Kidwell Mar 15 '19 at 20:43
0

The [TCP Retransmission] mean the packet was sent, and the PC still try to resend it, as it got no answer.

It usually mean the receiver didn't sent back a AC to aknowledge the data was received.

It could be bad route on the receiver's side, as often the receiver might not have a route back to your IP 196.52.84.14

I suggest debugging from the receiver's side, I suggest 192.168.1.208 as you can enable a packet sniffer there easily. Collect the log and check if the remote receiver know a default route to your VPN IP.


Edit 1

The wireshark show the machine get the retransmission but it don't answer. As such please make sure the firewall on the server does not block that traffic, winpcap/wireshark capture inbound packet BEFORE the local firewall discard those packets.

  • Just did a quick edit to the original post to include packet capture at both devices. What's the best way to test whether the server can find a route back to that VPN IP? – Jeremy Kidwell Mar 15 '19 at 16:59
  • That's really strange. Pretty sure the firewall is open... will add above. – Jeremy Kidwell Mar 15 '19 at 18:27
  • @JeremyKidwell please provide a route -n, to see if the computer can answer to 196.*.*.*, and it will show the default route 0.0.0.0 – yagmoth555 Mar 15 '19 at 18:31
  • good call. Done! – Jeremy Kidwell Mar 15 '19 at 18:36
  • @JeremyKidwell I sense a route for 196 is missing, 196.52.84.0/24 via 192.168.1.1 dev eno1 I would guess, if it come from router at LAN IP 192.168.1.1, as the default route 0.0.0.0 is 10.1.10.9, which sound incorrect for your packet – yagmoth555 Mar 15 '19 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.