2

I have a Linux box acting as a router with no iptables or other firewall and no networking applications running on it, just pure router. I've put it in a test environment that generates many TCP connections, each having unique source and destination IP, and those connections go through this router. I'm observing that number of connections successfully created rise to approximately 500 and then no more connections can be created for several minutes, then another 100 connections can be created and there is another pause, and so on. If 10 connections for each source-destination pair are created, then maximum numbers go about 10 times up, so the problem is probably with many connections from different IPs.

As traffic is simply routed, it doesn't have to do with number of file descriptors, iptables connection tracking and other things often proposed to check in similar cases. The box has plenty of free RAM and CPU, both NICs are gigabit. The kernel is 2.6.32.

I've already tried increasing net.core.*mem_max, net.core.netdev_max_backlog and txqueuelen on both NICs, with completely no effect. What else should I check ? Is there some rate-limit in the kernel itself ?

Improve this question

2 Answers 2

Reset to default
0

It's not clear why you are sure the problem is not caused by one of your endpoints (client or server), what type of traffic are you generating? Does it work with different device?

Improve this answer
1
  • I'm sure because the problem disappears when I remove the linux router, I'm getting up to 50k simultaneous connections then. The traffic is TCP, after establishing each connection the client and server exchange 1 byte and leave the connection idle. I've also tried using Win7 router and I get about 9k connections with the default settings.
    – Eugene
    Dec 4, 2012 at 10:21
0

I've found the answer and it was quite funny - ARP table overflow. The traffic in the test environment was gerenated from many IPs that resided in directly-connected networks, so the system had to use ARP first to figure out MACs, and the default hard limit of ARP table in Linux is just 1024 entries, which gives number of connections between networks connected to 2 different interfaces close to 512. When I increased net.ipv4.neigh.gc_thresh1 and also .gc_thresh2 and .gc_thresh3, the problem was solved.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .