Windows 2008 R2 Servers Sending Arp Requests for IPs outside Subnet

Question

By running a packet capture on my my routers I see some of my servers sending ARP requests for IPs that exist outside of its network.

For example if my network is:

Network: 8.8.8.0/24  
Gateway: 8.8.8.1 (MAC: 00:21:9b:aa:aa:aa)  
Example Server: 8.8.8.20 (MAC:  00:21:9b:bb:bb:bb)

By running a capture on the interface that has 8.8.8.1 I see requests like:

Sender Mac: 00:21:9b:bb:bb:bb 
Sender IP: 8.8.8.20
Target MAC: 00:21:9b:aa:aa:aa
Target IP: 69.63.181.58

Anyone seen this behavior before? My understanding of ARP is that requests should only go out for IPs within the subnet... Am I confused in my understanding of ARP? If I am not confused, anyone seen this behavior?

Also, these seem to happen in bursts and it doesn't happen when I do something like ping an IP outside of the network.

Update:
In response to Ian's questions. I am not running anything like Hyper-V. I have multiple interfaces but only one is active (Using BACS failover teaming). The subnet mask is 255.255.255.0 (Even if it were something different it wouldn't explain an IP like 69.63.181.58).

When I run MS Network Monitor or wireshark I do not see these ARP requests. What happens is that on the router capturing I see a burst of about 10 requests for IPs outside of the network from the host machine. On the machine itself using wireshark or NetMon I see a flood of ARP responses for all the machines on the network. However, I don't see any requests in the capture asking for those responses.

So it seems like maybe it is maybe refreshing the arp cache but including IPs that outside of the network. Also when it does this NetMon doesn't show the ARP requests?

Answer 1

If you don't see the ARP requests in Wireshark/Netmon then two additional sources for the ARP frames could be Broadcom's teaming driver (BASP) and OEM network manageability (Dell's DRAC and HP's iLo for example).

The Broadcom teaming driver includes a feature called "LiveLink" which uses ARP frames to verify network connections to remote systems (see http://support.dell.com/support/edocs/network/p29352/english/teaming.htm). If the user sets a LiveLink probe for an IP address outside of the local subnet then BASP will happily generate an ARP for that address. Of course, if the address is bogus then the team should indicate a failure on one or more of the NIC's used in the team.

Enterprise servers often have a dedicated Ethernet port for manageability. Lower cost servers may piggyback on the LOM port and send traffic through the same RJ-45 connector as Windows. If the management feature of the server is enabled but not configured correctly it may generate ARPs outside of the IP subnet used by the host. These ARP frames would also be invisible to Wireshark/Netmon. Most management solutions also work while the system is off so if you continue to see ARPs generated by a system when it's turned off then the management function may be the source.

Answer 2

Your understanding of ARP is correct. A host shouldn't ARP for an ip address not in it's local subnet because it should know that the ip address is not local and therefore should know that the data needs to be sent to the default gateway. The only time I've ever seen this is with a host infected with malware.

Answer 3

The obvious question is: Whats your network config? Do you have multiple interfaces? Do you have hyper-v or vmware on your server?

In any case your understanding of arp is indeed correct. A network config mistake seems like the most likely explanation.

Try installing MSNetmon on your server and capturing arp traffic. It should give you an indication of where its coming from. In all likelyhood it will be system and not some malware process. If it is malware as mentioned above then it should identify it quickly.