Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Is there efficient and easy way to use IP blacklist (like this one, but perhaps 2-3 times larger, which can't be simplified to CIDRs much) to deny access to a web server (either lighttpd or Apache) using standard/userspace software (no kernel recompilation) on Linux?

AFAIK Deny rules in Apache config and iptables have linear lookup time. There are kernel modules that efficiently handle large lists of IPs, but installation of those requires patching of the kernel.

share|improve this question

5 Answers 5

You could try using moblock (google it - can't add links yet, new user). Disable all downloaded blocklists, and use only a local blocklist that you generate. You may need to add NFQUEUE (netlink queue) support to your kernel, but it may already be there by default.

The general setup is: for all SYN packets on the ports you want to filter, use netfilter's NFQUEUE action to push them to moblock, sitting in userspace. Moblock does efficient matching and sends back either an ACCEPT or DROP response to netlink.

The moblock config file format is quite simple: on each line, give a name and an IP range, in the form 123.123.123.42-123.123.124.56. When moblock loads the ranges, it builds an efficient data structure to match against those ranges. When a packet is dropped because of a match, the range name and actual source IP is logged (or not, if you disable logging of matches).

I've used moblock in its default configuration (downloaded blocklists) with about 230000 IP ranges, and have observed no discernible performance hit (filtering only the SYN packet is important to keep kernel/userspace traffic down though).

One caveat: if moblock is not running, I believe NFQUEUE's default action is to DROP, resulting in a denial of service of your application. That said, I've had moblock running continuously without any problem for over 6 months. Still, you may want to set up a monitoring probe that alerts you if a known-good IP can no longer connect to :80 on your server. You definitely don't want to use moblock to filter ssh, unless you have explicitely whitelisted some trusted IPs in netfilter to recover.

share|improve this answer

AFAIK Deny rules in Apache config and iptables have linear lookup time.

Sort of. You can use chains in IPTables to break it up, i.e. a simplistic approach would be to have a chain for each A class block of addresses (e.g. 1.0.0.0/8, 2.0.0.0/8, etc.) and the rules for that block within the chain, reducing the look up time significantly (i.e. worst case ~200 rule evaluations to get to the A class rule and then however many rules within that block. You can also use "iptables -L -v -n" to see which rule sets are being evaluated the most heavily and move them to the top. There are other better ways to do this, iptables documentation covers this.

share|improve this answer
    
Heh... that was going to be my answer :) –  MikeyB Jun 1 '09 at 1:45

Yeah, on your system, there should be a /etc/hosts.allow. Take a look, it has some very easy examples to lock out IPs even on a per service basis.

In a nutshell, in /etc/hosts.allow:

ALL : ALL : allow

httpd : /etc/hosts.httpd.deny : deny

sshd : /etc/hosts.sshd.deny : deny
share|improve this answer

I think you cannot do it without paching you kernel.

ipset seems to be a good solution.

Quoting from its webpage:

If you want to

  • store multiple IP addresses or port numbers and match against the collection by iptables at one swoop;
  • dynamically update iptables rules against IP addresses or ports without performance penalty;
  • express complex IP address and ports based rulesets with one single iptables rule and benefit from the speed of IP sets

then ipset may be the proper tool for you.

share|improve this answer

You can also do it in .htaccess

Here's an example which blocks by useragent, but the idea is the same: http://jetfar.com/trap-content-scraper-spam-harvester-bots-using-honeypot-wordpress-htaccess/

AFAIK Deny rules in Apache config and iptables have linear lookup time.

I would try it and then do some testing before you rule this out - the impact might be unnoticeable, and it would be very easy to maintain.

Similarly you could try adding the list to /etc/hosts.deny

I wouldn't assume which is faster without testing both first.

Cheers / Richy

share|improve this answer
    
I've measured and there is very big negative impact. –  porneL Jun 1 '09 at 22:09
    
Really, I'd be interested to know some more detail about that... –  Rich Jun 2 '09 at 1:03
    
It's easy, Rich. .htaccess is evaluated on each request and when enabled it checks in each folder for a .htaccess file. It's a huge performance hit. –  Till Jun 16 '09 at 13:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.