Wrong Gateway assigned

Question

Hello people of serverfault. I am currently running into some issues. Currently I have a total of 7 servers. 6 of them have Ubuntu 12.04. The other one is a clearos. Currently the clearos machine connects to the modem/internet and then my poweredge 3348 managed switch.

3 of those machines have no issues and I can ping the gateway address 10.0.1.1 and ping the ip's of all 3 nic's of each server from the gateway with no issues.

The other 3 servers have issues. Only 1 nic from each server is pingable from the gateway and the other two are unavailable.

Here is an example of my mail server.

miverson@mail:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0b:db:e2:96:5a  
          inet addr:10.0.1.220  Bcast:192.168.255.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fee2:965a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:26013 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17550 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3801680 (3.8 MB)  TX bytes:1921219 (1.9 MB)
          Interrupt:28 

eth1      Link encap:Ethernet  HWaddr 00:0b:db:e2:96:5b  
          inet addr:10.0.1.221  Bcast:192.168.255.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fee2:965b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8131 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:597342 (597.3 KB)  TX bytes:2222 (2.2 KB)
          Interrupt:29 

eth2      Link encap:Ethernet  HWaddr 00:11:25:bf:42:59  
          inet addr:10.0.1.222  Bcast:192.168.255.255  Mask:255.255.255.0
          inet6 addr: fe80::211:25ff:febf:4259/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6581 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:459462 (459.4 KB)  TX bytes:3672 (3.6 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:54 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4644 (4.6 KB)  TX bytes:4644 (4.6 KB)

My sudo route command from that server

miverson@mail:~$ sudo route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.1.1     0.0.0.0         UG    0      0        0 eth0
10.0.1.0        *               255.255.255.0   U     0      0        0 eth0
10.0.1.0        *               255.255.255.0   U     0      0        0 eth1
10.0.1.0        *               255.255.255.0   U     0      0        0 eth2

and my arp command

miverson@mail:~$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
10.0.1.210                  ether   00:0b:db:e7:56:e8   C                     eth0
10.0.1.1              ether   00:0d:56:b9:33:be   C                     eth0
10.0.1.144               ether   90:27:e4:f4:9d:60   C                     eth0
MySQL.therabbitry.org    ether   00:0b:db:e7:56:e2   C                     eth0

When I ping 10.0.1.220 I get a response and it's fine, but if I ping 10.0.1.221 or 10.0.1.222 from the gateway I get 100% packet loss.

Help me out and let me know what my solution is, or if you have any other questions.

EDIT: I just realized that my Bcast address doesn't look correct. Any recommendations? Eth0 Eth1 Eth2 are all set via dhcp and are not static. Matt

Answer 1

A couple of things:

Your broadcast address is incorrect for all your interfaces. It should be set to 10.0.1.255, assuming your netmask of 255.255.255.0 is correct.

Your default route seems to be set correctly however it will only work with a single interface. Think of it like this: Your default route is the route of last resort. If your server has no idea how to get to where you want your traffic to go, it'll try the default route. Your default route says, try to find the gateway at 10.0.1.1/24 via eth0, he should know where to go from there.

Your other interfaces eth1 and eth2 have routes but they are not involved in the default route. This is probably why you pings are successfully returned to 10.0.1.220. Assuming you are not doing any VLANing, as long as all of your interfaces have the same correct subnet mask they should be able to communicate "link-local", that is without having to involve a router since all the interfaces are presumably in the same subnet.

This is your clue. Either your router is misconfigured or your switch is doing something to disrupt the continuity of Layer-2. Additionally check out this fantastic answer on how subnetting works.

Answer 2

As already noted by kce, The broadcast address is incorrect, although this would not cause any of the issues mentioned in your post.

The broadcast address is an option to set within your DHCP server scope (DHCP option code 28) - if it is currently set to 192.168.255.255, this is easily changed.

As you already have seen with your other "working" machines, there is no specific magic to make IP connectivity to work - as soon as the interfaces are configured and connected, the IP addresses are reachable. The simplest debugging option would be to just run a tcpdump -i eth1 -v -n to take a look at the traffic (preferably while running a ping to eth1's address from 10.0.1.1). Given the symptoms I would strongly suspect a connectivity problem - you likely would not see any traffic from 10.0.1.1 at at all.