Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

(Before getting into details, I'm presenting this problem with Apache and SSH as an example, but this is not specific to TCP traffic, it is the same problem with both TCP and UDP based protocols.)

I have a multilinked, multihomed server running Ubuntu 9.04, with eth0 connected to an outside network and eth1 connected to an inside network. The outside network is presented to the "rest of the world" and the inside network contains all of the developers workstations and workhorse servers. There is a firewall blocking traffic from the "rest of the world" to the inside network, but not blocking outgoing requests.

$ /sbin/ifconfig
eth0  Link encap:Ethernet  HWaddr 00:30:18:a5:62:63  
      inet addr:xxx.yyy.159.36  Bcast:xxx.yyy.159.47  Mask:255.255.255.240
      [snip]

eth1  Link encap:Ethernet  HWaddr 00:02:b3:bd:03:29  
      inet addr:xxx.zzz.109.65  Bcast:xxx.zzz.109.255  Mask:255.255.255.0
      [snip]

$ route -n 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
xxx.yyy.159.32  0.0.0.0         255.255.255.240 U     0      0        0 eth0
xxx.zzz.109.0   0.0.0.0         255.255.255.0   U     0      0        0 eth1
0.0.0.0         xxx.yyy.159.33  0.0.0.0         UG    100    0        0 eth0

Apache is listening on port 80, sshd listening on 22:

$ netstat --tcp -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 *:www                   *:*                     LISTEN     
tcp        0      0 *:ssh                   *:*                     LISTEN     
[snip]

From my development machine on the inside, xxx.zzz.109.40, I can connect to the inside address and all is well. From the outside I can connect to the outside address and everything works like it should.

But for some kinds of testing I would like to connect to the outside address from my development machine, yet the server is refusing the connection request. I'm guessing that it is looking in its routing tables and since the incoming data is coming from an address that should be on eth1 but is arriving on eth0 that it's dropping it, probably as a security precaution.

Is there a way I can relax this restriction?

The odd thing is that this used to work on 8.04, but does not work on 8.10 or 9.04, so sometime during the last year the kernel is doing some extra checking. For the connection to work, the return path needs to be the same as the source path, so that means that messages from my development machine arriving on eth0 would have to go back out on eth0 to be routed back to my machine.


Here is a diagram, there is no NAT anywhere.

Diagram

share|improve this question
    
You should probably explain why the data should come from your developement machine .109.40 arrives via eth0 on the way to the outside server. You should also post your iptables settings for the forward table. A simple iptables -L should suffice. –  towo Jul 7 '09 at 15:48
    
The external interface requires authentication for HTTP, but the internal one doesn't. Most of the server development is relatively simple, but some of it requires authenticating as one of a collection of test users. I could add another network card to 'd' and connect it to the outside, but I'd rather not. –  Joel Jul 8 '09 at 4:05
add comment

7 Answers

up vote 1 down vote accepted

Assuming you don't have any iptables rules that would prevent this, you need to disable the return path filtering. You can do this using:

# echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

You can also use a particular interface name instead of all and there is also a default, which would affect newly created interfaces.

From:

reverse path filter; it is a check to see if, for a packet arriving on an interface, a packet sent to the original packet's source address would be sent out on that interface; if not, the arriving packet is dropped. it can be considered an attempt at detecting packets with spoofed source addresses.

share|improve this answer
    
Given that the source address isn't being spoofed, rp_filter isn't going to help a bit. –  womble Jul 4 '09 at 2:01
1  
The packet will be received on the external interface, but the outgoing packet will be routed out of the internal interface. This seems to match the description of rp_filter pretty accurately. –  David Pashley Jul 4 '09 at 2:09
    
@David: If the development machines are connected to a single network, and their gateway is the machine in question, then the packet will come from the internal interface. –  derobert Jul 4 '09 at 3:52
    
@David: How will the packet be received on the external interface? –  womble Jul 4 '09 at 8:03
    
I don't understand the downvotes. This seems to be the most probable explanation. –  bortzmeyer Jul 5 '09 at 20:58
show 1 more comment

You haven't shown your firewall config, but I'll bet there's a rule in there that's doing something untoward with filtering connections to the external IP address from inside. A badly screwed up NAT config could also be at fault, but I'm having trouble envisaging something so pathological resulting from an ordinary attempt at configuration.

Just to clarify one point, too: just because a packet comes from one interface addressed to the IP address of a different interface will not cause the kernel to reject that packet. Whatever's happening, it's not the kernel's fault.

Also, a tcpdump of traffic (both on the internal and external interfaces of the server, and on the client) would be diagnostically useful.

share|improve this answer
    
I have confirmed with Wireshark that requests from the inside are making it through the firewall to the outside interface, but there is no response from the server. I'll run tcpdump on the server on the eth0 interface, but I'm pretty confident they'll show up there too. –  Joel Jul 4 '09 at 2:18
1  
You saw the packets on the external interface? That shouldn't happen. I smell NAT insanity. –  womble Jul 4 '09 at 3:23
    
No, no NAT, hopefully the diagram will make it clearer. –  Joel Jul 8 '09 at 4:06
add comment

What is default gateway for your development machine? The default gateway assuming Layer 3 Switch / Router should know how to reach the other interface of the server.

You should have something like

ip route xxx.yyy.159.36 netmask 255.255.255.240 gw xxx.zzz.109.65

This should make things work. But in case this is not sufficient, then enable IP forwarding on machine xxx.zzz.109.65 using

sysctl net.ipv4.ip_forward = 1

Also edit /etc/sysctl.conf and enable ip forwarding there too.

Make sure iptables FORWARD chain of filter table on xxx.zzz.109.65 allows packet forwarding for your development machine.

share|improve this answer
    
Like the answer posted by Whisk, it's pretty clear I've got some forwarding rules/patterns/behavior to investigate. –  Joel Jul 8 '09 at 4:10
add comment

I'm guessing you're using that box as your internet nat router using iptables NAT - obviously if that isn't the case then this won't help, otherwise you'll probably need something like:

iptables -t nat -I POSTROUTING -s xxx.zzz.109.0/24 -d xxx.yyy.159.36 -j ACCEPT

That will skip the source natting that you're doing for packets travelling from inside to outside for anything destined for the external IP of that server.

share|improve this answer
    
No NAT, but my iptables NAT experience fits in the period at the end of this sentence. –  Joel Jul 8 '09 at 4:08
add comment

Just to be anal I would not call the above network configuration multihomed. It's merely forwarding between two networks within the same AS. Multihomed these days really means connected to two or more neighbours (AS's), or at least multiple routes to the same upstream neighbour (AS), in the case of a leaf network with a single upstream connectivity provider.

share|improve this answer
    
I like to think that I'm a stickler for using correct terminology, could you please comment with a reference to what 'AS' refers to in this context? I edited the question to add diagram, if that helps clarify anything. –  Joel Jul 8 '09 at 4:13
    
AS == Autonomous System. –  Jason Tan Jul 17 '09 at 11:11
add comment

Lots of useful ideas already given. I would a) Double-check it's not the firewall/NAT. Add a -j LOG rule before your REJECT and DROP rules and see if anything suspicious gets caught. b) Run tcpdump on both interfaces (-i any) and look for your packets.

BTW, what exactly do you mean by "the server is refusing the connection request"? Do you see some (immediate) refusal message or does the connection just time out?

share|improve this answer
    
There is no response from the server, it's as if the request never arrived. The client eventually times out...and tcpdump has been my friend for a long time :-). –  Joel Jul 8 '09 at 4:16
add comment

This has got to be a problem with your firewall rules. Most likely the box's own external IP is getting lumped in with 'all external traffic' and so isn't being allowed back through to the internal client. Try putting an exception in the firewall rules for that IP address; that is, explicitly allow the webserver's external IP address to send traffic to the internal networks.

More debugging will require you to post your firewall rules.

share|improve this answer
    
As it turns out it was not a firewall problem, but thank you. –  Joel Jul 8 '09 at 4:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.