Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm looking to block IP addresses in a relatively automated fashion if they look to be 'screen scraping' content from websites that we host. In the past this was achieved by some ingenious perl scripts and OpenBSD's pf. pf is great in that you can provide it nice tables of IP addresses and it will efficiently handle blocking based on them. However for various reasons (before my time) they made the decision to switch to CentOS. iptables doesn't natively provide the ability to block large numbers of addresses (I'm told it wasn't unusual to be blocking 5000+), and I'm a bit cautious over adding that many rules into an iptable.

ipt_recent would be awesome for doing this, plus it provides a lot of flexibility for just severely slowing down access, but there is a bug in the CentOS kernel that is stopping me from using it (reported, but awaiting fix).

Using ipset would entail compiling a more up-to-date version of iptables than comes with CentOS which whilst I'm perfectly capable of doing it, I'd rather not do from a patching, security and consistency perspective.

Other than those two it looks like nfblock is a reasonable alternative. Is anyone aware of other ways of achieving this? Are my concerns about several thousand IP addresses in iptables as individual rules unfounded?

share|improve this question
The more I think on this the more I think what's probably going to be best is some form of efficiently breaking up the rules, trying to ensure each packet gets checked by the least possible, e.g. first octet, 255 can be split into 5 blocks of 51 addresses, so create separate chains for them, then split those chains up into further blocks themselves etc. etc. – Twirrim Mar 8 '10 at 16:44

iptables is the userland tool for manipulating netfilter. netfilter is the code in the kernel that handles the packet filtering. Contextually, changing the userland tool would only change your experience not the way the filtering operates.

I have never hit a limitation in Linux with the quantity of rules specified and I've been using Linux for firewalling since the userland tool was ipfwadm. It's notable that Netfilter wasn't introduced to Linux until the 2.4.x kernel and ipfwadm was the interface for ipfw not netfilter. OpenBSD is great for a firewall if you can continue using it in your environment.

The limitation will likely be a physical limitation based on the system resources, with a focus on the quantity of RAM available. If you encounter issues, you may have to tune Linux's max ip connection settings in the kernel. You're unlikely to encounter either of these situations on modern hardware with a modern Linux distribution.

If you would like to discuss the finer details of netfilter, you might be better off furthering this dialog on the netfilter mailing lists, as they will be the subject matter experts.

If this does not answer your question, please feel free to clarify and I will be happy to revise it.

Netfilter user hits memory limitation testing netfilter limits

share|improve this answer
Hmm, so for the most part it doesn't really matter what I'm using, I'm manipulating the same thing in the end and should see comparable performance. The only thing that niggles me about that is ipset, which appears to abstract that away from netfilter: If I'm reading that man page correctly ipset stores the IP addresses itself in memory, as you can specify the method e.g. iphash, rather than through normal netfilter method (or is it pushing the iphash into the netfilter?) – Twirrim Mar 8 '10 at 15:57

The best way to block large amount of IP addresses without causing to much stress on the kernel is to null route them with command:

root@machine# ip route add blackhole

iptables is not suited well for many rules.

share|improve this answer

I think the bug he referenced is in the netfilter code, specifically the ipt_recent module. I seems limited to 4k (32bit) and 8k (64bit).


share|improve this answer

5000 rules is not a problem, however, there is which does do quite well at handling enormous numbers of rules.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.