0

I am setting up a cloud environment on Ubuntu and I am running into an interesting problem. Out of the box Ubuntu (all linux really) doesn't allow you to specify more than one gateway in /etc/network/interfaces

Since I am not using DHCP for any of the NICs and I am using vLANs on my switch and router (reference image below) I need each NIC on this server to not just have it's own IP but have it's own gateway specified. Since vLAN 2 which is 10.0.1.0/24 cannot access 10.0.0.1 which is the default gateway but inaccessible due to netmask 255.255.255.0.

All my research has lead me to understand that I need to configure the routing tables by hand, however, when I try to add routing for more than the first eth0 NIC I get the following error:

# ip route add default via 10.0.1.1 dev eth1 table eth1
RTNETLINK answers: File exists

enter image description here

At this point I am lost for things to try... I cannot add the routes to the new route tables, and without explicit routes each NIC card tries to use 10.0.0.1 as the gateway since it's the default gateway for 10.0.0.0/24

  • 2
    You already have a default gateway, and you can only have one! What exactly are you trying to route where? – Michael Hampton Apr 8 '15 at 23:09
  • Read my entire post :-) – Ethode Apr 8 '15 at 23:10
  • 2
    I did. It doesn't make sense. Which is why I asked you to clarify. – Michael Hampton Apr 8 '15 at 23:10
  • I need each NIC to operate on separate networks, and vLAN's do not share a gateway, therefore if you want them to make it to the router for things like MAC binding it needs to use the vLAN gateway address which is respectively 10.0.0.1 (management network) 10.0.1.1 (vLAN 2) 10.0.2.1 (vLAN 3) The only way to use 10.0.0.1 as the gateway for all three NIC's and keep my IP scheme is to move the netmask t0 255.255.0.0 which would in effect eliminate the security of the separate networks in the first place – Ethode Apr 8 '15 at 23:13
  • I should also mention it really doesn't matter if I'm dealing with vLAN's or literally 2 physical networks, I can't imagine it's THAT uncommon to have NIC's communicate on different networks lol – Ethode Apr 8 '15 at 23:15
2

So here's the answer after taking Matt's suggestion and going the trial and error route.

The command I used had the following pattern ip route add via dev

The final command actually was

# ip route add 10.0.1.31 via 10.0.1.1 dev eth1
# ip route add 10.0.2.31 via 10.0.2.1 dev eth2

After adding these I had no issues pinging between the IP's on the vLAN.. Fantastic...

Essentially the difference here is that I added a static route so that interaction next hop is sent to the vLAN gateway ID instead of going to the default gateway.

Also so these are permanent I wound up adding them to the /etc/network/interfaces config file as post-up configs. So I wound up with this line for eth1 and eth2, but I left eth0 alone since it can use the default gateway

post-up route add 10.0.1.31 via 10.0.1.1 dev eth1
  • 1
    If this solved the problem, why did you keep going on about default gateways, which have nothing to do with this? You should have automatically gotten routes for these if you configured the netmask correctly in /etc/network/interfaces. – Michael Hampton Apr 9 '15 at 1:31
  • The route I added IS the gateway. Just as @zoredash mentioned below, an alternative would have been to add multiple routing tables for each subnet and then adding a default GW to each routing table; thus giving me three default gateways at the end. I just happen to prefer the simple one liner I can add to interfaces instead. Changing the netmask wouldn't fix anything because I need the network separation. Had I opened up the netmask to be 255.255.0.0 instead of 255.255.255.0 sure it would have found the route, but then I would have lost the network separation. I WANT /24 it's not an accident – Ethode Apr 9 '15 at 2:13
0

You can only have one default gateway, that's why it's called default. You can create static routes for the other NICs for particular networks reachable on that NICs VLAN.

  • That's precisely what I attempted to do, and reference in my question. However, I receive that error message which will not allow me to add the route which I thought was odd... That being said, it's actually incorrect to state that all NIC cards CANNOT have separate gateways. Using route tables it's possible and there are about 1000 posts showing how to do it on Google but most of not applicable to Debian flavors and the ones that are debian based have me run the command I posted in my question (which fails), so I am stuck :-( – Ethode Apr 8 '15 at 23:19
  • 1
    Nonsense, the Linux kernel allows you to specify multiple default gateways and weight them. – Matt Apr 8 '15 at 23:20
  • 2
    You can only have one useful 'gateway' per route table. Sure you can have backups by setting metrics that will take over if the main gateway fails, and on Linux you can have a gateway that is a weighted set of addresses. But really you can only have one effective 'default' gateway per route table. – Zoredache Apr 9 '15 at 0:54
  • @Zoredache you're correct, if I add multiple routing tables I also can verify that they work, as they should provided I have the proper ordering. However, I have to admit, I kind of like the 1 liner I add to the /etc/network/interfaces config file a bit better, just seems cleaner than adding brand new route tables and setting up default gw in each. Do you see a potential downside to using a static route in the post-up vs separate route tables with a gw in each? – Ethode Apr 9 '15 at 1:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.