I'm using Ubuntu 14.04 (standard server install + OpenSSH, biosdevname=0 during instal to keep the network names simple) in an environment where I have two non-overlapping subnets M and N on different VLANs.

For the purpose of debugging an issue on one of the subnets, I want to configure my server such that it can receive traffic on eth0 from an IP on subnet M, and on eth1 from an IP on subnet N.

I realize I can only have one gateway, so I use the gateway from subnet M. I set up /etc/network/interfaces such that subnet M is fully specified with an IP A.B.C.42 (which works normally) and from subnet N I've chosen an IP address X.Y.Z.88 and provided the appropriate netmask for that subnet (it doesn't overlap with subnet M at all).

I can bring up eth1 via sudo ip addr flush dev eth1; sudo ifdown eth1; sudo ifup eth1... this appears to work and my routing table is:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface         A.B.C.1         UG    0      0        0 eth0
A.B.C.0 U     0      0        0 eth0
X.Y.Z.64 U     0      0        0 eth1

Note that the adapters are valid and working per sudo lshw -c network etc.

At this point, assuming there is no blockage on subnet N, should I be able to ping or ssh to this server via subnet N (using IP X.Y.Z.88)? (Subnet M with IP A.B.C.42 definitely works.) If not, what obvious steps / tests am I missing?

(I've seen similar questions around here and elsewhere but so far haven't found one that has this configuration (totally separate subnets for example). I'm really trying t determine if subnet N is blocking traffic or is otherwise acting up.)


Well there are some things you can test.

First and for all, in your topic you write about vlans. Is the server connected to a switch? If so, are the ports correctly configured in the right VLAN (access port?) etc...

Is your server a virtual machine? Is the VLAN properly configured on the virtual NIC (portgroup on ESX f.e.)?

2nd: Is your server able to reach another device in subnet N's IP range?

3rd: Is there a firewall installed on the server? If so does it allow communication coming from or going to subnet N?

4th: For SSH, is the SSH server listening on any address or is it configured to just listen on subnet M's IP?

For some more debugging you could use like tcpdump or tshark. This would allow you to check if traffic is actually reaching the server.

  • It's a real machine (not a VM - though it was problems with a more complex config with VMs that led to this simplified debugging setup). No, actually I cannot ping other IPs in that range (but I can ping the gateway address for that range... however, I chalked that up to not having a route to the others). There may be blockage/firewalling on the network, but the server is open enough (I can easily get through on eth0, but not on eth1 - and yes, both adapters are functional). SSH - not special, I can definitely ssh from the machine to itself via its eth0 and eth1 IPs. – MartyMacGyver Oct 15 '14 at 19:59
  • If M and N are directly connected to the server then no Default Gateway is needed on either adapter and no routing occurs for communication with hosts on either subnet. NIC M should be able to communicate with M hosts and NIC N should be able to communicate with N hosts without any DG or additional routing table entries. – joeqwerty Oct 15 '14 at 20:19
  • The default gateway is there to get to the outside world (for the sake of this, assume I can route outside via the subnet M's gateway). That said, I can't seem to route between hosts on subnet N which I'd like to resolve as well (perhaps because subnet M has no knowledge of subnet N)... not sure what route I can add to satisfy this. (I tried sudo ip route add X.Y.Z.64/27 via X.Y.Z.65 dev eth1 after deleting the original X.Y.Z.64 route but that leads to an "RTNETLINK answers: Network is unreachable" error.) – MartyMacGyver Oct 15 '14 at 20:42
  • OK. I think I get it. The server is meant to be a router between subnets M and N and not to simply be itself dual-homed. Is that correct? – joeqwerty Oct 15 '14 at 20:49
  • In the desired config the server is supposed to be able to be accessed from either subnet but isn't meant to necessarily route traffic from one subnet to the other (though it could, that's not the point). In this case, I am testing whether there are valid routes to both subnets from some point (or points) external to those subnets. The big assumption is that subnet N is able to pass the traffic... I'm just trying to rule out an error (in principle or in practice) in my networking config of the server itself - is this possible as shown, or are there more steps? – MartyMacGyver Oct 15 '14 at 21:03

Actually, the answer wasn't as messy as I expected but it's not exactly obvious.

It was clear that naively adding the route + the second gateway wasn't working (nor should it work - you can't have multiple gateways in this way):

sudo ip route add X.Y.Z.64/27 via X.Y.Z.65 dev eth1
RTNETLINK answers: File exists

A friend pointed me to a page describing how to handle a similar situation. Bottom line: you need to create ip rules and then you can add a second gateway.

Assuming you have a current OS and kernel iproute2 and policy routing should be baked in. With that in mind, we can simply execute the following:

sudo bash -c 'echo "1 admin" >> /etc/iproute2/rt_tables'
sudo ip route add X.Y.Z.64/27 dev eth1 src X.Y.Z.88 table admin
sudo ip route add default via X.Y.Z.65 dev eth1 table admin
sudo ip rule  add from X.Y.Z.88/32 table admin
sudo ip rule  add to   X.Y.Z.88/32 table admin
sudo ip route flush cache

And it just works. I can now ping /ssh to either interface!

Note: the "RTNETLINK answers: Network is unreachable" error in a previous comment was due to a misconfigured switch (I kept getting "RTNETLINK answers: No such process" when I was trying the solution above for the same reason apparently). The sudo tcpdump -i eth1 -n -vv -c 1 'ether[20:2] == 0x2000' command was invaluable in identifying that problem. Once I realized what was wrong I was able to use the correct IP ranges for that VLAN to get this to work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.