In terms of
nf_conntrack_max, you are only limited by the amount of available kernel memory. Kernel memory isn't swappable so it is actual RAM.
TLDR: On modern systems I set this to 128k with seemingly no ill effect.
Another tunable that you may wish to tweak, however, is
/proc/sys/net/netfilter/nf_conntrack_buckets. Every time a packet is examined it is hashed into one of the
hashsize buckets (based on src/dst IP and port). Each bucket contains a linked list that then has to be searched for the particular connection. Increasing the number of buckets should (up to a certain point) decrease the number a nodes in the linked list that need to be traversed and therefore increase performance. The tradeoff is that the more buckets you have the higher the probability of empty buckets. An empty bucket takes kernel memory. Thats why you don't set
nf_conntrack_max. I was unable to find any recommendations on setting the hashsize except the statement "Under normal circumstances ip_conntrack_max equals 8 * hashsize."
Another thing to note is that when your connections are short-lived (e.g., a web server) you may have many connections in the TIME_WAIT state with corresponding entries in the conntrack hash.
cat /proc/net/nf_conntrack | grep TIME | wc -l (
ip_conntrack on older kernels) or
iptstate should tell you this information. To reduce the timeout for these change
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait. You'd probably want to change
/proc/sys/net/ipv4/tcp_fin_timeout as well.
Finally, if you really need performance and you can't afford the RAM you could disable the firewall completely and put a dedicated firewall in from of the server.