Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In my .htaccess file I'm trying to block some bots using:

RewriteCond %{HTTP_USER_AGENT} ^badbot [NC]

But, I want to change these conditions to something more inclusive like:

Method #1: RewriteCond %{HTTP_USER_AGENT} .*badbot [NC,OR]

OR

Method #2: RewriteCond %{HTTP_USER_AGENT} ^.*badbot [NC,OR]

OR

Method #3: RewriteCond %{HTTP_USER_AGENT} badbot [NC,OR]

The question is, which one of those 3 methods would be best for server performance?

Thanks!

share|improve this question
    
Sorry, I'm new to this website. I didn't realize I was supposed to do that. I will go back and accept the best answers. –  Zero Nov 13 '10 at 13:08
    
Then apologies for the tone of my comment. Some users just don't accept answers at all. –  James Lawrie Nov 13 '10 at 13:19
    
It may take a while for that to update - I think some details are refreshed every so often. –  James Lawrie Nov 13 '10 at 13:43

2 Answers 2

up vote 1 down vote accepted

As far as I'm aware, there is little performance difference between these. Logically, the /.*badbot/ and /badbot/ should be equivalent, and as regular expressions are parsed from left to right I would also have thought the ^ made no difference.

I did test these with Perl (which may use a slightly different regex engine to mod_rewrite) and it did show /badbot/ to be quite a bit quicker than the others, but this could be down to the strings I was using. Also, bearing in mind that USER_AGENT has a maximum length of 200, this is probably unnecessary optimization. You'd get more optimization by placing these inside the httpd.conf instead of .htaccess.

The results I got from testing are:
[james@aladdin ~]$ time perl -e 'my $x = "bad" x 99999999; $x .= "badbot"; print "true" if $x =~ /^.*badbot/;'
true
real 0m2.575s
user 0m1.847s
sys 0m0.712s
[james@aladdin ~]$ time perl -e 'my $x = "bad" x 99999999; $x .= "badbot"; print "true" if $x =~ /badbot/;'
true
real 0m2.110s
user 0m1.378s
sys 0m0.691s
[james@aladdin ~]$ time perl -e 'my $x = "bad" x 99999999; $x .= "badbot"; print "true" if $x =~ /.*badbot/;'
true
real 0m2.522s
user 0m1.866s
sys 0m0.647s

share|improve this answer
    
Thanks, that was very helpful. So you're saying that method #3 would be best? Without the .* OR ^.* –  Zero Nov 13 '10 at 13:39
    
From testing, yes it seems so, but I can't understand why. It could be down to the strings being used - if it's using the Boyer-Moore algorithm then it will search for the last character of the string first and work back from there (so would be fast with the above example) - see lists.freebsd.org/pipermail/freebsd-current/2010-August/… - although I did replace the "bad" with "bot" to see if that made a difference and #3 was still faster. I wish I knew why. –  James Lawrie Nov 13 '10 at 13:46
    
I guess I'll go with method #3 then. I'm just wondering if using a rewrite rule to "block" the thousands of bot requests that I'm receiving would increase the server's cpu usage versus the bandwidth that would be saved by using my .htaccess file. In other words, ignore the bots and let them waste bandwidth or block the bots and use cpu power to block them. Am I correct to think this? –  Zero Nov 13 '10 at 18:45

If your looking for optimal performance... Firstly have Apache ignore/skip looking for .ht files.. Disable .htaccess.. :D

AllowOverride None

And put your stuff/rewrites/etc directly in each vhost..

And then choose 1,2 or 3 :D

Then you have already taken a step towards "faster"

Hope this helps.. :D

share|improve this answer
    
I'm on a shared server so I don't have access to the Apache config. Would AllowOverride None still benefit me? From what I understand it just disables .htaccess files in all sub-directories, but keeps the main .htaccess file active, right? –  Zero Nov 13 '10 at 18:15
    
Yes... you can also turn it off completely.. Which stops reading these files dynamically on EVERY call.. -----First it stats the disk in every folder, reads the file, parses the content, applies the rules.. ( you can see this being heavy across a busy server ) in fact, its completely stupid by design.. –  Arenstar Nov 13 '10 at 18:20
    
Okay, so according to the Apache docs I can only have AllowOverride None in <Directory>. I want my main .htaccess file in public_html to still apply to all directories (my entire site), but I don't want the server to search for any .htaccess files in those sub-directories. So what would I put in <Directory>? –  Zero Nov 13 '10 at 18:36
    
I don't think that the local .htaccess scheme is stupid, since it will only parse the files if they are present, which should be false for most directories. Also, it's very useful in shared hosting, since it's a good and simple way of change some server settings without touching the global settings. Of course, if you put the same configuration in each folder instead of using the root folder I agree that it's a bit stupid, but is the admin fault. –  Alberto Martinez Nov 13 '10 at 20:24
    
Do you think stat'ing the disk for every folder is intelligent/logical considering that there is a VERY high chance it doesnt exist? Basically any serious Apache deployment will turn this off (IMHO it should be default) –  Arenstar Nov 14 '10 at 4:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.