arp table filling up

Question

I have a CentOS 5 system which serves as an IPSec VPN connector between my network and a couple of remote networks. Lately it is running into this problem once or twice a day where its ARP cache gets filled up.

The local network on which this guy sits is 10.51.0.0/16 and IPSec connects it to 10.53.0.0/16 and 10.54.0.0/16. It has 2 interfaces, eth0 connected to the internet and eth1 which is connected to the local network with ip 10.51.1.15.

The ARP cache gets filled up with addresses like 10.51.119.x and it seems to methodically go on filling it up completely. I ran tcpdump while it was happening and saw ARP requests for all these non-existent addresses originating from the local ip 10.51.1.15 so it's almost like somebody is doing a network scan, but how do I figure out where it is originating? It is unlikely that it actually comes from the box itself, nobody should be running scans like that from it, but it might be coming from the IPSec networks? How do I figure out where it's coming from?

Problem solved: it was Kaspersky antivirus doing system discovery on our network.

Answer 1

For the scanner to gain anyting (other than creating a DOS attack) he has to receive the packets. If the scan appears to origin from a locally attached subnet, then he cannot be anywhere on a route between the source and the origin, because there's nothing in-between the CentOS box and 10.51.1.15.

You could:

1. Run tcpdump with -e header. This will tell you MAC address of the originator of the scan.
2. Ping 10.51.1.15 at various times and see, if the arp antry corresponds to what link layer headers held in the tcpdump.
3. If you have a managed switch, then you can log onto it and see from where do the frames come. This will tell you the socket, into which the scanner is plugged. This method will work also if the scanner really wants just to flood your ARP cache.

It is unlikely, that the scan originates outside the 10.51.0.0/16 network. Typical firewall rules drop packets that claim to origin from a locally-connected network that come through wrong interface. Then, maybe you have atypical firewall rules ;).

I'm not sure if I got your question right, but if 10.51.1.15 is the IP of the CentOS box, then apart of tcpdump -e the ideas above dont' make sense. If the packets appear to originate from the local box then it has packets to deliver to these addresses. In that case you could set up iptables rules that log packets to invalid addresses of 10.51.0.0/16 network. By invalid addresses I mean addresses which have not been assigned to any host. If something is trying to access them, then probably it has no legitimate reason to do so. Such logging rule (with corresponding limit rule to prevent log filling) will tell you origin of such a packet -- if it was from a local box, or any of the remote networks.

If the packets appear to originate from the CentOS box itself you could also take a look at ps output during a suspected scan and compare it to ps output saved when there was no scanning. Maybe you'll find some obvious answers.

Answer 2

I found this post to be incredibly helpful in debugging a similar situation. The recommendation made there is to do the follwing:

tcpdump -l -n arp | egrep 'arp who-has' | head -100 | awk '{ print $NF }' |sort | uniq -c | sort -n

This will give you the ip that made the most "who-has" requests out of the last 100. This should give you what you need to know to find the offending system.

Answer 3

Not sure if it applies, but I had the same exact issue on a Cisco router and the issue was that on the default route I had the interface name specified rather than the IP address. The ARP table would fill until it crippled the router. As soon as I specified an IP it cleared up. Maybe completely unrelated but you never know.

ip route 0.0.0.0 0.0.0.0 FastEthernet0/1 <-wrong
ip route 0.0.0.0 0.0.0.0 192.1668.99.254 <-right