(Before getting into details, I'm presenting this problem with Apache and SSH as an example, but this is not specific to TCP traffic, it is the same problem with both TCP and UDP based protocols.)
I have a multilinked, multihomed server running Ubuntu 9.04, with eth0 connected to an outside network and eth1 connected to an inside network. The outside network is presented to the "rest of the world" and the inside network contains all of the developers workstations and workhorse servers. There is a firewall blocking traffic from the "rest of the world" to the inside network, but not blocking outgoing requests.
$ /sbin/ifconfig eth0 Link encap:Ethernet HWaddr 00:30:18:a5:62:63 inet addr:xxx.yyy.159.36 Bcast:xxx.yyy.159.47 Mask:255.255.255.240 [snip] eth1 Link encap:Ethernet HWaddr 00:02:b3:bd:03:29 inet addr:xxx.zzz.109.65 Bcast:xxx.zzz.109.255 Mask:255.255.255.0 [snip] $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface xxx.yyy.159.32 0.0.0.0 255.255.255.240 U 0 0 0 eth0 xxx.zzz.109.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 0.0.0.0 xxx.yyy.159.33 0.0.0.0 UG 100 0 0 eth0
Apache is listening on port 80, sshd listening on 22:
$ netstat --tcp -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:www *:* LISTEN tcp 0 0 *:ssh *:* LISTEN [snip]
From my development machine on the inside, xxx.zzz.109.40, I can connect to the inside address and all is well. From the outside I can connect to the outside address and everything works like it should.
But for some kinds of testing I would like to connect to the outside address from my development machine, yet the server is refusing the connection request. I'm guessing that it is looking in its routing tables and since the incoming data is coming from an address that should be on eth1 but is arriving on eth0 that it's dropping it, probably as a security precaution.
Is there a way I can relax this restriction?
The odd thing is that this used to work on 8.04, but does not work on 8.10 or 9.04, so sometime during the last year the kernel is doing some extra checking. For the connection to work, the return path needs to be the same as the source path, so that means that messages from my development machine arriving on eth0 would have to go back out on eth0 to be routed back to my machine.
Here is a diagram, there is no NAT anywhere.