I'm having a server with network connectivity issues that I presume come from issues with arp protocol handling.
Let's say the network topology is as follows:
- network 192.168.106.0, netmask 255.255.255.0
- router at 192.168.106.1
- "problem server" at 192.168.106.2
- another server at 192.168.106.3
Now, assume that the "problem server" may be silent on the network for periods long enough for its arp entry on the router to expire.
When someone from outside this network attempts to connect to the "problem server", all attempts time out. Connections from within the network to the "problem server" succeed.
If the "problem server" itself attempts to connect to some other address outside the network, the connection succeeds -- and after this, also connections from outside the network to the "problem server" succeed for a while. Also, connections from the "problem server" to "another server" are ok.
Looking at arp traffic in the case where the "problem server" has been silent for a long time, I can see arp requests on the network for the "problem server" address, but the "tell" address on these is the network address (192.168.106.0) instead of the router address (192.168.106.1) -- and this is what I assume to be the reason for this problem: for some reason the router has wrong reply address in its arp requests.
The "another server" remains reachable, but there I assume the reason to be that it frequently makes connections to outside the local network, and thus keeps its arp entry at the router from expiring.
Any comments / suggestions?
The servers in question are running Linux (CentOS 5.x?), and are running as VMs within VMWare ESXi (5.0?) (I'll check/fill in version details once I get back to work on Monday). The router make/model is unknown for me.
Responses to questions, further findings
Apologies for being slow to return this.
Unfortunately my visibility to the network side (anything beyond the VMWare platform itself) is severely limited.
Based on the arp request packets from the router, it is a Juniper product (guessing by requestor MAC address).
This is a small network, so consider topology as a router, switch, and a single VMWare server hosting several virtual machines.
As for the originator of the odd arp requests, it pretty much has to be the network gateway: they only appear when I try to connect to the "problem" machine from outside the network - and cease when the attempt times out or is cancelled. A minor oddity is that the MAC address in these requests is not the same that is seen for the router in the server arp table after establishing an outbound connection. However, both the MAC address present in these "odd" requests as well as the MAC address shown in the server arp table have a Juniper-assigner OUI.
Then one possibly related finding; it seems that Linux won't respond to arp requests where "tell" address is the network address, whereas Windows (Vista at least) does. This I wasn't able to test in the actual problem environment, but with my own toys at home.
Also, it looks like I'm not completely alone with this issue; a similar experience can be found here: alpacapowered.wordpress.com
ping -c3 192.168.106.1
command to cron on your servers to be run every 10 minutes or so, this way you ensure that the arp entries won't time out. But this will not get to the cause of the problem.