Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This morning I received this is my dmesg log:

[3184815.656881] nf_conntrack: table full, dropping packet.
[3184821.442351] net_ratelimit: 282 callbacks suppressed

FWIW, here's the current max:

cat /proc/sys/net/netfilter/nf_conntrack_max 
65536

So, I can gather that the server is dropping TCP connections because it can't keep track of them all. I know I can just set a new max number in sysctl.conf or echo it to /proc/sys/net/netfilter/nf_conntrack_max but I'm not sure that's the best course to take.

Would I be setting up the kernel for certain death? Is there a better approach to handling LOTS of network connections?

Thanks in advance

share|improve this question
add comment

2 Answers

In terms of nf_conntrack_max, you are only limited by the amount of available kernel memory. Kernel memory isn't swappable so it is actual RAM.

TLDR: On modern systems I set this to 128k with seemingly no ill effect.

Another tunable that you may wish to tweak, however, is /proc/sys/net/netfilter/nf_conntrack_buckets. Every time a packet is examined it is hashed into one of the hashsize buckets (based on src/dst IP and port). Each bucket contains a linked list that then has to be searched for the particular connection. Increasing the number of buckets should (up to a certain point) decrease the number a nodes in the linked list that need to be traversed and therefore increase performance. The tradeoff is that the more buckets you have the higher the probability of empty buckets. An empty bucket takes kernel memory. Thats why you don't set hashsize = nf_conntrack_max. I was unable to find any recommendations on setting the hashsize except the statement "Under normal circumstances ip_conntrack_max equals 8 * hashsize."

Another thing to note is that when your connections are short-lived (e.g., a web server) you may have many connections in the TIME_WAIT state with corresponding entries in the conntrack hash. cat /proc/net/nf_conntrack | grep TIME | wc -l (ip_conntrack on older kernels) or iptstate should tell you this information. To reduce the timeout for these change /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait. You'd probably want to change /proc/sys/net/ipv4/tcp_fin_timeout as well.

Finally, if you really need performance and you can't afford the RAM you could disable the firewall completely and put a dedicated firewall in from of the server.

share|improve this answer
    
Thanks @embobo, this is super useful. I'll carefully play around with each of the params you suggested. –  Bryce Apr 5 '11 at 17:41
add comment

Do you actually need connection tracking? Take a look at your firewall rules and see if you manage to not use it. If you've just got a simple web server, you almost certainly don't actually need to use connection tracking.

In general, if you are handling lots of traffic, you want the firewall to have to do as little as possible. Disabling connection tracking would be a good start there. (It's a lot easier to look at a packet and say 'Is this going to port 80? If it's not, drop it.' then 'Is this going to port 80, and have I seen this connection from the same IP/Port/TCP Session # before?'

share|improve this answer
    
That's a really good point @devicenull. Being that this is a frontend to a stack of webheads, doesn't it make sense to keep the conn tracking enabled to reduce the total amount of TCP traffic through the device? Removing the tracking would cause the box to re-negotiate the TCP session for every connection wouldn't it? –  Bryce Apr 5 '11 at 17:43
1  
Connection tracking in this case only refers to IPTables keeping track of it, it does not effect the actual TCP stack. Unless you have some firewall rules that require this (which may exist in a load balancer), you can disable connection tracking with no ill effects. You might try switching to something like HAProxy if you need a load balancer that doesn't require connection tracking. –  devicenull Apr 6 '11 at 0:02
    
that's really funny. I am using HAProxy and was afraid that disabling conntrack would screw up its round robin balancing strategy. Thanks a lot @devicenull –  Bryce Apr 6 '11 at 0:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.