SFTP connections to docker container break when using packet forwarding instead of port binding

Question

Setup

I am trying to make a setup that allows me to directly connect to docker containers from my LAN without binding individual ports to the docker hosts IP addresses.

I have turned on IP forwarding with sysctl -w net.ipv4.ip_forward=1.

Symptoms

I can connect with SSH and use SFTP to transfer really small files. However if I try to send a file that's bigger than around 30K the transfer just hangs. I need to kill sftp to stop it.

EDIT: The file size limit seems to vary. Now it is somewhere between 50-100KB. It can be changed further with different variations of -B, -l and -R of sftp

Small files reach 100% and stall, bigger ones read less. Speed slowly moves to zero and times reads - stalled -

sftp> put testfile35k
Uploading testfile35k to /home/user/testfile35k
testfile35k                                   100%   35KB   0.4KB/s   01:29

(killall -9 sftp)

sftp> put testfile
Uploading testfile to /home/user/testfile
testfile                                       20% 2080KB   0.0KB/s - stalled -

After 15 minutes I get Write failed: connection timed out and eventually Write failed: Broken pipe

If instead of trying to forward from eth0 to docker0 I use docker's port binding (-p) SFTP works fine.

Network settings:

SSH Server:

[root@b6c9d8f9d30d Python-3.4.3]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
112: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:0c brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.12/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:c/64 scope link
       valid_lft forever preferred_lft forever

Docker/"router":
[root@docker ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:21:68:9b brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.193/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.0.121/24 brd 192.168.0.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe21:689b/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::5484:7aff:fefe:9799/64 scope link
       valid_lft forever preferred_lft forever

SSH Client:
[root@test ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:99:0b:64 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.201/24 brd 192.168.0.255 scope global eth1
    inet6 fe80::5054:ff:fe99:b64/64 scope link
       valid_lft forever preferred_lft forever

Docker host iptables: EDIT2: Flushed all iptables rules and NAT without any effect

[root@docker ~]# iptables-save
# Generated by iptables-save v1.4.21 on Sat Sep 12 15:09:53 2015
*nat
:PREROUTING ACCEPT [5:488]
:INPUT ACCEPT [2:284]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [3:204]
:DOCKER - [0:0]
COMMIT
# Completed on Sat Sep 12 15:09:53 2015
# Generated by iptables-save v1.4.21 on Sat Sep 12 15:09:53 2015
*filter
:INPUT ACCEPT [513:54486]
:FORWARD ACCEPT [280:161095]
:OUTPUT ACCEPT [325:35618]
:DOCKER - [0:0]
COMMIT
# Completed on Sat Sep 12 15:09:53 2015

I know that containers are not normally supposed to be running sshd. This is a remote python development host and the SSH is for running a remote interpreter.

Answer 1

The problem was not with docker or even ip forwarding on the docker host, but with another router instead.

I realised that if I manually set a static route to 172.17./16 via 192.168.0.193 SFTP starts to work.

I had configured my actual router/internet gateway to redirect traffic to 172.17/16 to the docker host on LAN instead of going to the internet. After playing a bit with the settings of that router this now works. It might have had something to do with TCP state tracking and asymmetric routing.