Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I manage Ruby on Rails servers and need a way to have the loadbalancer reject bots and along with anyone who hits the site more than say, 60 times per minute (minus static file calls). I know CPANEL has some sort of IP blacklisting setup in it that is pretty easy to manage and let's you set rates and all, but is there a good free combo?

Also is there anything for keeping an up-to-date list of bots I should automatically reject (for making web requests)? The load balancer is running nginx and it is a PAIN to have to manually enter the bot to reject in the conf file after one of them takes an app server down with constant requests.

share|improve this question
    
depending on your needs, fail2ban might work, though you'd need to create a rule for it. –  Journeyman Geek Dec 9 '10 at 5:13
add comment

2 Answers 2

I just went through all of this last week as my server was his by a DDoS, I practically created a tool kit, and wrote my own apache module tailored(which I may end up publishing the source if needed.) to my own needs. I would love to hear someone else's methods, but here is what you can easily setup:

Limit the amount of inbound connections coming in to your webserver, KeepAlive should be enabled on apache by default. This will limit your inbound tcp connections on port 80 to 4 times in two seconds.

iptables -A INPUT [any custom places you want checked] -j burst_check (for me, I did this:)
iptables -A INPUT -d my.internal.lan.ip -p tcp --dport 80 -j burst_check
iptables -N burst_check
iptables -A burst_check -m state –-state NEW -m recent –-set
iptables -A burst_check -m state –-state NEW -m recent -–update –-seconds 2 -–hitcount 4 -j LOG --log-prefix "hitcount "
iptables -A burst_check -m state –-state NEW -m recent -–update –-seconds 2 -–hitcount 4 -j DROP

Another tool that I found useful was mod_evasive.

The winning combination was I originally wrote a script that was able to identify the bots from the access_log that looped indefinitely adding it to be blocked on iptables. My apache module does the same thing except handles it as the connection comes in and blocks it in iptables before they even get a chance to reconnect. Plus having the flood check in place allowed them to connect only a few times a second until so my script (originally) would have a second to ban them.

As far as blacklists go:
http://www.spamhaus.org/drop/drop.lasso http://www.cymru.com/Documents/bogon-bn-agg.txt APF can add these automated, but I don't use them.

Instead, I'm currently using a list of the 30,000 IP's that hit me over the weekend, plus all of the subnets in thailand. (a large percentage of these bots were in thailand, and they really have no business on my server as-is.)

share|improve this answer
    
Ouch, sorry to hear about that! I am running nginx, so I guess I could adapt the script to work with it.Do you know of any automated solutions to get those blacklists into your IP tables, or I suppose I can just write a crobjob and shellscript it. –  tesmar Dec 10 '10 at 15:56
    
apf adds those automatically (although I haven't used it), there was another blacklist I've seen since then to grab the top 1000 attacker's from dshield.org's website that you could probably add into it as well or do an automated cron. –  user63068 Dec 10 '10 at 18:27
add comment

you may want to look at something like iptables or pf

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.