I am setting up a cloud environment on Ubuntu and I am running into an interesting problem. Out of the box Ubuntu (all linux really) doesn't allow you to specify more than one gateway in /etc/network/interfaces
Since I am not using DHCP for any of the NICs and I am using vLANs on my switch and router (reference image below) I need each NIC on this server to not just have it's own IP but have it's own gateway specified. Since vLAN 2 which is 10.0.1.0/24 cannot access 10.0.0.1 which is the default gateway but inaccessible due to netmask 255.255.255.0.
All my research has lead me to understand that I need to configure the routing tables by hand, however, when I try to add routing for more than the first eth0 NIC I get the following error:
# ip route add default via 10.0.1.1 dev eth1 table eth1 RTNETLINK answers: File exists
At this point I am lost for things to try... I cannot add the routes to the new route tables, and without explicit routes each NIC card tries to use 10.0.0.1 as the gateway since it's the default gateway for 10.0.0.0/24