Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My server is getting killed by Baiduspider and no matter what I put in my robots.txt file nothing happens. So temporarily I need to block as many of it's IP addresses as I can via iptables. I am getting the IP addresses via:

grep -ri Baidu /var/log/apache2/access.log | cut -f1 -d' ' | sort | uniq

And my iptables rules look like:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            ctstate RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp-data 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:www 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:snpp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:mysql 
LOG        all  --  anywhere             anywhere            limit: avg 5/min burst 5 LOG level debug prefix `iptables denied: ' 
DROP       all  --  anywhere             anywhere 

I'm planning on adding the new rules with something similar to:

for ip in $(grep -ri Baidu /var/log/apache2/access.log | cut -f1 -d' ' | sort | uniq); do iptables -A INPUT -s $ip -j DROP; done

But I don't think I can just add them. I think they need to be inserted in a specific location. Where do they need to be inserted to take effect?

share|improve this question
1  
In the INPUT chain. First entry. –  mailq Dec 27 '11 at 19:30
1  
But why not block 119.63.192.0/21 with one rule? –  mailq Dec 27 '11 at 19:35
    
There were 5 different subnets that had user agents claiming to be "Baiduspider/2.0". I ended up blocking all of them, rather than doing it by individual IP address. –  nachito Dec 27 '11 at 19:43

1 Answer 1

Your first rule looks to be allowing everything, making all of the rules below it pointless (is this intended?).

You'd need to insert the rule above that for it to have any effect; change your -A INPUT to a -I INPUT.

You might also consider just blocking them in Apache based on user agent instead - a 403 response might get the message across better than a failed connection, and won't require the resources that you're using now to respond to those requests with content.

share|improve this answer
1  
The question is too localized. I queried my logs right now and see thousands of forged Baidu requests whereas they don't originate from the Baidu IP range. –  mailq Dec 27 '11 at 19:42
    
No Shane, the first rule wasn't intended. I think someone added that a while back when they complained that the firewall wasn't working properly. I'm trying to block those requests as soon as possible as they don't seem to be legitimate, otherwise I would consider using Apache to do it. –  nachito Dec 27 '11 at 19:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.