CentOS 6, 2 NICs, 1 gateway, 1 NIC doesn't work right

Question

I have a CentOS 6 install that had 1 NIC, eth0, working fine

I added another NIC, eth1. Both NICs are on the same switch, and same LAN.

I copied ifcfg-eth0 to ifcfg-eth1, and then simply changed the DEVICE=, IPADDR=, and HWADDR= iines.

After setting up /etc/sysconfig/network-scripts/ifcfg-eth1, I can ping eth1 from an external machine all day and it is fine.

But if I try to ping from inside the server (ping -Ieth1 google.com), I get no response, whereas ping -Ieth0 google.com works fine.

I have removed the GATEWAY= lines from both ifcfg-eth0 and ifcfg-eth1, and put GATEWAY= in /etc/sysconfig/network.

Here are my obfuscated config files:

eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.1.40
NETMASK=255.255.255.0
HWADDR=00:00:23:45:67:8F
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
DNS1=192.168.1.240
DNS2=192.168.1.244

eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=192.168.1.41
NETMASK=255.255.255.0
HWADDR=00:00:23:45:67:99
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
DNS1=192.168.1.240
DNS2=192.168.1.244

network
NETWORKING_IPV6=no
HOSTNAME=myserver
NETWORKING=yes
GATEWAY=192.168.1.2

My iptables -L -v are currently completely empty. And I am not using SELinux.

Answer 1

Never ever set two interfaces to the same subnet, this will probably not work as desired. If you examine the output of ip ro, you'll understand that Linux is predominantly based on destination-based routing. Although source-based routing is possible, it is very rare in practice. I can only recall one time when I needed to do it in my life, whatever was the reason.

You misunderstood what ping -I does, hence the confusion. From man:

-I interface address

Set source address to specified interface address.

Ping doesn't send the packet trough -I interface. When a packet is ready to be sent out, kernel looks at destination IP and ip ro, and based on these two things it decides which interface to use. It doesn't look at packet's source IP.

-I if for setting the source IP of a packet. As a packet flows out, the switch knows both your IPs came from MAC of eth0, so it sends responses there. But eth0 drops everything that isn't addressed to its own IP (or broadcast). There is no "magical internal bridging" between eth0 and eht1, outside of what you see in the ip ro. So the answer never reaches eth1.

The ip ro is a modern replacement for deprecated route.

Since you use the same switch/gateway, you probably want to have bonding of eth0 and eth1. Then set your two IP addresses on bond0 interface (ifcfg-bond0). This is a standard practice and would work as a charm.

To completely separate the traffic between eth0 and eth1, it's best to set eth1 to a different IP subnet, such as 192.168.2.40 and use a different router/gateway :)

Answer 2

net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2

Packets discarded when the route for outbound traffic differs from the route of incoming traffic

Answer 3

Daniel -

It appears that your problem exists because of the intented configuration's limitation. You have two interfaces on the same LAN/subnet, but one of them must be the default route. Run this command to verify your route is defaulting to eth0, this will indicate that you will not be able to reach any devices outside your network when you attempt to force traffic through eth1.

route

It should look something like this:

Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.1.0 * 255.255.255.0 U 0 0 0 eth0

link-local * 255.255.0.0 U 1002 0 0 eth0

default 192.168.1.0 0.0.0.0 UG 0 0 0 eth0

For sanity checking, try pinging something local to your LAN through that port:

ping -Ieth1 192.168.1.2