Second network card configuration not working

Question

I have 4 servers running Centos 5. All of them have two ethernet network cards.

I have configured 192.168.1.x IP addresses on their eth0 card. They are all connected to the same switch using their eth0 card and they are all working.

I have configured 10.72.11.x IP addresses on their eth1 card.They are all connected to the same switch - a different one from the switch used with eth0 card - using their eth1 card and they are NOT all working.

Their configuration files is like:

DEVICE=eth1
BOOTPROTO=static
IPADDR=10.72.11.236
BROADCAST=10.72.11.191
NETMASK=255.255.255.192
NETWORK=10.72.11.128
HWADDR=84:2B:2B:55:4B:98
IPV6INIT=yes
IPV6_AUTOCONF=yes
ONBOOT=yes

The interface is starting and configured as I need.

[root@sql1 network-scripts]# ifconfig
**eth0**  Link encap:Ethernet  HWaddr 84:2B:2B:55:4B:97
          inet addr:192.168.1.105  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::862b:2bff:fe55:4b97/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2981 errors:0 dropped:0 overruns:0 frame:0
          TX packets:319 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:386809 (377.7 KiB)  TX bytes:66134 (64.5 KiB)
          Interrupt:36 Memory:da000000-da012800

**eth1**  Link encap:Ethernet  HWaddr 84:2B:2B:55:4B:98
          inet addr:10.72.11.236  Bcast:10.72.11.191  Mask:255.255.255.192
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:48 Memory:dc000000-dc012800  

I also added a route-eth1 file that looks like:

10.0.0.0/8 via 10.72.11.254

Routing looks fine to me:

[root@sql1 network-scripts]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.72.11.192    0.0.0.0         255.255.255.192 U         0 0          0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
10.0.0.0        10.72.11.254    255.0.0.0       UG        0 0          0 eth1
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0

But I cannot ping one server from the other.

[root@sql1 network-scripts]# ping 10.72.11.235
PING 10.72.11.235 (10.72.11.235) 56(84) bytes of data.
From 10.72.11.236 icmp_seq=1 Destination Host Unreachable
From 10.72.11.236 icmp_seq=2 Destination Host Unreachable
From 10.72.11.236 icmp_seq=3 Destination Host Unreachable
From 10.72.11.236 icmp_seq=4 Destination Host Unreachable
From 10.72.11.236 icmp_seq=5 Destination Host Unreachable
From 10.72.11.236 icmp_seq=6 Destination Host Unreachable
^C
--- 10.72.11.235 ping statistics ---
7 packets transmitted, 0 received, +6 errors, 100% packet loss, time 6033ms, pipe 3

What am I doing wrong?

Answer 1

It should be noted that your NICs aren't properly 'working'. Your ifconfig output doesn't show the 'RUNNING' required for there to be a properly working link. note the line UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 from your eth0 interface. There's no 'RUNNING' evident in the eth1 output. In my experience, this indicates a low level issue. eg: Bad/disconnected cabling.

Is is possible the NIC you have plugged your cable into is NOT eth1? Servers can get some weird naming going on. Try checking the output of /var/log/messages, to see if it reports an interface getting (dis)connected. Likewise, see if you get any diagnostic info there when you connect/disconnect, or during boot. Check and see if this is occurring on all 5 boxes. You may also want to try the graphical configuration tool, just in case you've got a typo in the configuration file. (system-config-network)

Answer 2

Think a bit about your subnet setup:

IPADDR=10.72.11.236
BROADCAST=10.72.11.191
NETMASK=255.255.255.192
NETWORK=10.72.11.128

In base 2:

236 is 1110 1100
191 is 1011 1111
192 is 1100 0000
128 is 1000 0000

Your IPs are outside your subnet - 10.72.11.236 is not in 10.72.11.128/26, 10.72.11.172 is. Neither is 17.72.11.235 but 10.72.11.171 is. Either change your IPs or change your broadcast to .255 and your network to .192. Alternatively (unless it's being dictated to you), run in the lower half of 10.72.11.0 (i.e. 10.72.11.0/26) or just use a /24 and save yourself the hassle.

Answer 3

Try using ipcalc from http://jodies.de/ipcalc, it is a big help when configuring networks and subnetworks.

# ipcalc 10.72.11.236/255.255.255.192
Address:   10.72.11.236         00001010.01001000.00001011.11 101100
Netmask:   255.255.255.192 = 26 11111111.11111111.11111111.11 000000
Wildcard:  0.0.0.63             00000000.00000000.00000000.00 111111
=>
Network:   10.72.11.192/26      00001010.01001000.00001011.11 000000
HostMin:   10.72.11.193         00001010.01001000.00001011.11 000001
HostMax:   10.72.11.254         00001010.01001000.00001011.11 111110
Broadcast: 10.72.11.255         00001010.01001000.00001011.11 111111
Hosts/Net: 62                    Class A, Private Internet

Richard

Answer 4

I've seen this problem before. Not 100% sure of the fix, but I remember there being something in sysctl to tell ARP to consider these different devices - or something along these lines.

There have been a few posts about in on serverfault before if I remember correctly.

Answer 5

Inbound traffic that goes to eth0 should come back via eth0 and vice versa. Try doing a TCPdump to check where traffic is going.

You also need policy based routing to do this : See Linux 2 network card routing depending on the interface used