Linux Server Virtual Ethernet interface stops responding to arp requests

Question

I'm hoping someone has had this issue before as it is driving me nuts. I have 2500 linux embedded Linux servers. We use these servers partly to act as gateways to other devices sat on a private network. The configuration contains a series of virtual interfaces (not vlans) with DNAT and SNAT rules to pass requests on to said devices.

For years this has worked fine. Last week we added about 25 virtual interfaces and about 50 plus iptables rules. At this point we ran into an issue that we did not expect. Our embedded Linux servers randomly stopped answering requests for IP virtual IP addresses configured against the main Ethernet interface eth0. ARP requests go unanswered eg:

tcpdump -vvvnn arp  -- *x.x.x.23 is the virtual IP address and x.x.x.16 is the router the request came from*
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
16:15:10.404513 arp who-has x.x.x.23 tell x.x.x.16 
16:15:12.403592 arp who-has x.x.x.23 tell x.x.x.16
16:15:14.407617 arp who-has x.x.x.23 tell x.x.x.16

The device that x.x.x.23 is natted to is pingable from the Linux Server and the IPtables entry looks correct but never gets evaluated.

Does anyone know if there is some sort of buffer/cache in Linux that could limit the number of interfaces defined against a single Ethernet interface? We have about 50 per Linux Server now.

Our firewall configuration is getting larger but I would expect an operating system error if that was causing a problem.

I can't see an operating system errors in dmesg or /var/log/messages.

We have a mixture of different ethernet hardware RLT8139 and RDC6040. 2 different flavours of Linux, Debian 6 and Puppy Linux 3(older sites). Both have exhibited the same problem.

It is almost like the server does not know that it has the interface. Flushing the firewall rules doesn't seem to do anything. The server should reply to the ping when the nat rule is disabled but it doesn't.

I'm banging my head now. This is what our interfaces look like. The IPs have been masked to protect the innocent.......

eth0      Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.1  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1036673 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4606675 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:164871619 (157.2 MiB)  TX bytes:354806888 (338.3 MiB)
          Interrupt:10 Base address:0xde00 

eth0:1    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.3  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:10   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.13  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:14   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.14  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:15   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.15  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:16   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.16  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:17   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.17  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:18   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.18  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:19   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.19  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:2    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.11  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:20   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.20  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:21   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.21  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:22   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.22  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:23   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.23  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:24   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.24  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:25   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.25  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:26   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.26  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:27   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.27  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:3    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.12  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:33   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.33  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:34   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.34  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:39   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.39  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:4    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.4  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:40   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.40  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:41   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.41  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:42   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.42  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:43   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.43  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:44   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.44  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:45   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.45  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:46   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.46  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:47   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.47  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:48   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.48  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:49   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.49  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:5    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.5  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:50   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.50  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:51   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.51  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:52   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.52  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:53   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.53  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:54   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.54  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:55   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.55  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:56   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.56  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:57   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.57  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:58   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.58  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:59   Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.59  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:6    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.6  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:7    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.7  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:8    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.8  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00 

eth0:9    Link encap:Ethernet  HWaddr 00:1B:EB:43:24:05  
          inet addr:x.x.x.9  Bcast:x.x.x.63  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0xde00

Xavier Nicollet · Accepted Answer · 2015-03-25 14:07:24Z

0

According to what you report: x.x.x.16 is both eth0:16 and the router.

Have you masked too many things ?

Anyway, I would look at /proc/sys/net/ipv4/conf/ [all|default|eth0] / arp_ignore and arp_announce .

Regards,

answered Mar 25, 2015 at 14:07

Xavier Nicollet

6305 silver badges11 bronze badges

Add a comment |

Stack Exchange Network

Linux Server Virtual Ethernet interface stops responding to arp requests

1 Answer 1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
linux
linux-networking
.

Hot Network Questions

Linux Server Virtual Ethernet interface stops responding to arp requests

1 Answer 1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged linuxlinux-networking.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
linux
linux-networking
.