Two NIC (eth0,eth1), two global IP on same subnet. Became unreachable to IP on eth1 after reboot

Question

I configured a CentOS 6.5 virtual server and deployed it to a remote VMware vSphere environment. We needed two IP address, so I added a virtual NIC and assigned a second IP on eth1.

I didn't configure a source routing or somewhat in the OS, but it seemed to be working. So we went into production without it. Several months later, we had a maintenance and rebooted the virtual server.

After the reboot, we could not reach to the IP assigned to eth1. It was still reachable from a other machine on the same subnet, but not from outside. We couldn't resolve the problem, but it got back reachable again the next day even though I did nothing.

My Questions

1. Did this happen because I didn't configure the routing on the OS?
2. It doesn't matter that all the traffic was going through eth0 but do I still need to configure a routing table?
3. I know I can configure One NIC with Two IP (using alias device eth0:1).
   • Is this better solution?
   • Do I still need to configure a static routing with this approach?
4. Why did eth1 came back though I did nothing.
5. Is ARP table involved in this problem?
6. Why did my setup work but failed occasionally?

Here are some more information.

• MAC addresses are manually configured as static so it doesn't change.
• There is a firewall in front (We don't have control of it).
• IP on eth0 was reachable all the time.
• I needed Two IP to serve HTTPS(Apache) and RTMPS(Wowza).
• VMware Tools is installed on the virtual machine.

My config files.

/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPADDR=xxx.xxx.xxx.94
NETMASK=255.255.255.0
GATEWAY=xxx.xxx.xxx.254

/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPADDR=xxx.xxx.xxx.95
NETMASK=255.255.255.0
GATEWAY=xxx.xx.xxx.254

/etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:??:??:??", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:??:??:??", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Any advice will be appreciated!

Answer 1

Some gotchas:

• The NICs have been 'swapped' at boot, i.e. the system thought eth1 was eth0 and vice versa. Google "Enumerating network interface linux" for some hints. After that some other policy interfered (i.e. your internal firewall?) and traffic for one IP if coming in from the other NIC is blocked to prevent spoofing.

• Something changed the settings on the vmware network control panel so that one network card does not support 'tcp' any more, but only VMware's internal protocols.

Whatever you do, remove one default gateway if they are on the same subnet. It's useless and likely to confuse the system. Use ip to set up multi-gateways based on traffic policies if you need to.