htaccess noob needs your opinion and elite skills :)

Question

brand new with .htaccess and I have read the howto's for weeks along with a load of experimenting. I desperately need someone to look at the thing for me and tell me if anything is wrong or just plain won't work :) Here it is:

I think some of this is repetitive and can or should be combined but I don't get how to do it. The AddHandler stuff was for some php website issues I was having, the order allow deny was attempts at blocking linking from bad sites and all the Rewrites were second attempts at block all except a few that are good one. This was pieced together over weeks so I don't doubt I got it wrong somewhere. Any help would be great. Thanks in advance.

<AddHandler server-parsed .htm
AddHandler server-parsed .html
Addhandler application/x-httpd-php .html .php>

<order allow,deny
allow from all
deny from *.ru
deny from tamanria.com>

<RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?ceplocal2003.org(/)?.*$     [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yellowpages.ca(/)?.*$     [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?cep.ca(/)?.*$     [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?goldbook.ca(/)?.*$     [NC]
RewriteRule .*\.(gif|jpg|jpeg|bmp|png|pdf)$ http://www.ceplocal2003.org/images/Refererblock.gef [R,NC]>

<RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} .ru [NC,OR]
RewriteCond %{HTTP_REFERER} refblock\.com
RewriteRule .* - [F]>

<RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} tamanria.com [NC,OR]
RewriteCond %{HTTP_REFERER} refblock\.com
RewriteRule .* - [F]> 

<ErrorDocument 404 /Custom404.html>

enter code here

Answer 1

First of all, your syntax is wrong.

.htaccess files don't need angle brackets ("<>") around directives; they're needed only when you're creating subsections and specifying per-section settings, as in

<Directory /some/dir>
    allow from all
</Directory>

Also, it's redundant to state "RewriteEngine on" so many times; one will suffice (unless you're using it in a subsection).

Last but not least, your "order" directive tells Apache that "allow" directives should take precedence over "deny" ones, so you're effectively allowing access to your site from anywhere, because your "deny" directives are never even processed after the "allow from all" kicks in.

This should work better:

AddHandler server-parsed .htm   
AddHandler server-parsed .html
Addhandler application/x-httpd-php .html .php

order deny,allow
deny from *.ru
deny from tamanria.com
allow from all

RewriteEngine on    

RewriteCond %{HTTP_REFERER} !^$    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?ceplocal2003.org(/)?.*$     [NC]    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yellowpages.ca(/)?.*$     [NC]    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?cep.ca(/)?.*$     [NC]    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?goldbook.ca(/)?.*$     [NC]    
RewriteRule .*\.(gif|jpg|jpeg|bmp|png|pdf)$ http://www.ceplocal2003.org/images/Refererblock.gef [R,NC]

RewriteCond %{HTTP_REFERER} .ru [NC,OR]    
RewriteCond %{HTTP_REFERER} refblock\.com    
RewriteRule .* - [F]

RewriteCond %{HTTP_REFERER} tamanria.com [NC,OR]    
RewriteCond %{HTTP_REFERER} refblock\.com    
RewriteRule .* - [F]

ErrorDocument 404 /Custom404.html

Answer 2

deny from *.ru
deny from tamanria.com

This will force a DNS lookup for EVERY request going to your site. As you also have referrer blocking set up below, I'm guessing that you were trying to block content that was hotlinked from your site. If so, these rules wouldn't have worked to prevent that.

Since you are just trying to block sites that are hotlinking, you can put either the rules that set the sites you want to allow, or, deny the ones that have been hotlinking. I prefer preventing the problem rather than seeing a spike in the bandwidth, then frantically trying to figure out what site has hotlinked content.

I would think at a minimum, the following would do

AddHandler server-parsed .htm
AddHandler server-parsed .html
Addhandler application/x-httpd-php .html .php

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} ~Refererblock.gef$ [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*(yellowpages\.ca|cep\.ca|goldbook\.ca)(:[0-9
]+)?(/.*)?$ [NC]
RewriteRule .*.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg|zip)$ http://www.ceplocal2003.org/images/Refererblock.gef [NC,F,L]

Answer 3

While Massimo has just corrected the wiki formatting, there are still errors in the code.

RewriteCond %{HTTP_REFERER} .ru [NC,OR]

will block anything that contains somecharacter(ru) within the string.

so, any url that contains aru would be declined. A more restrictive ruleset should be used for that.

RewriteRule .* - [F]> 

will cause an error 500 as it is an invalid string. removing the > would fix that.

While I had more than 50 rep earlier, I have 1 now, so, I cannot comment on your answer. Please merge the answer into a comment. Thanks.

Answer 4

Thank you user46528 and Massimo, hopefully I didn't miss anyone. I am not sure how to merge my answer into a comment though :( For the .ru I was trying to actually block any domain ending in .ru from linking. I have found a lot of .ru nasty little sites linking for some reason.

The < code was added by mistake in here when I was figuring out how to format the help request properly :)

So I see the combining piece but I see two different types, does it matter which way you write it, both listed below.

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} ~Refererblock.gef$ [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*(yellowpages\.ca|cep\.ca|goldbook\.ca)(:[0-9
]+)?(/.*)?$ [NC]
RewriteRule .*.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg|zip)$ http://www.ceplocal2003.org/images/Refererblock.gef [NC,F,L]

OR

RewriteCond %{HTTP_REFERER} !^$    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?ceplocal2003.org(/)?.*$     [NC]    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yellowpages.ca(/)?.*$     [NC]    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?cep.ca(/)?.*$     [NC]    
RewriteCond %{HTTP_REFERER} !^http://(www\.)?goldbook.ca(/)?.*$     [NC]    
RewriteRule .*\.(gif|jpg|jpeg|bmp|png|pdf)$ http://www.ceplocal2003.org/images/Refererblock.gef [R,NC]>    

If I had to guess I would say both are fine but the first one is cleaner looking.

Second I am fuzzy on whether I had the allow,deny simply backwards or should I be getting rid of it altogether, I don't want to create a DNS lookup that much.

This ceplocal2003 is a work website that I maintain and while recently checking my SEO and searching google for ceplocal2003 I found alot of useless sites that seem to be just link farms that were linking back to our site. This is where I started reading up on .htaccess.

So, any chance with that information that the above suggestions could be compiled into a block that would work for me.

If I get what you two are saying I should block all "hotlinking" and other site stealing completely and then just allow the ones that need to be allowed like yellowpages and goldbook etc.

Boy do I ever feel like Alice tumbling down the rabbit hole now LOL, a week ago I would have thought htaccess was something intel designed for a motherboard :)

Answer 5

the order deny,allow is for blocking IPs/Hosts (and should not really be used for blocking hosts as it will do a DNS lookup for every request that could be blocked).

My personal feeling is that you should just preemptively block any attempt at hotlinking rather than blocking the ones that abuse it. if you wanted to block all .ru referrers, they would be covered under the other rules for your images.

If you wanted to block them from linking to your site at all, you would need to write a more general rule and include these below the rules that block the image hotlinking:

RewriteCond %{HTTP_REFERER} ^http://(.*@)?([a-z0-9]+\.)*(refblock\.com|tamanria\.com|\.ru)(:[0-9]+)?(/.*)?$ [NC]
RewriteRule .* - [F,L]

Answer 6

While the StackOverflow boys club again ranks an incorrect answer and buries the correct answer, I've consolidated the data into a single post that can be copied. I don't have enough rep to edit my previous post, nor, comment on Massimo's post which is technically valid, but, would cause performance issues down the line. Using hostnames in a deny from block is a very common, novice mistake and 99% of the time doesn't actually fix the problem. The deny block would prevent any host that contains a reverse DNS ending in .ru or tamanria.com from accessing your site. That is assuming that the reverse dns is properly configured. Many hosting companies use their own domain for the reverse mapping rather than the domain name that is being served. The deny from .ru would have blocked any surfer that had a reverse DNS lookup that ended in .ru, which may have been an acceptable compromise. However, anyone that has an IP address without a proper reverse DNS would experience a delay while accessing the site. If their site had a linkchecker or spider to find content, then that would have been valid, but, it would have been safer to block the IP address of the server or the IP address range. Another common but fatal error in a deny statement is

deny from 1.2.3.*

While it looks innocent enough, the * at the end means that the value needs to be checked against 1.2.3.domain.com, which again forces a reverse DNS lookup for each file request. To do partial matches with a deny block, the proper format is:

deny from 1.2.3.

or

deny from 1.2.3.0/24

In any case, the following is what should be used in the .htaccess

AddHandler server-parsed .htm
Addhandler application/x-httpd-php .html .php
AddType image/gif .gef

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*(yellowpages\.ca|cep\.ca|goldbook\.ca)(:[0-9
]+)?(/.*)?$ [NC]
RewriteRule .*.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg|zip)$ http://www.ceplocal2003.org/images/Refererblock.gef [NC,F,L]

#include this line only if you want to block ALL referring access from refblock.com, tamanria.com and all .ru sites.
RewriteCond %{HTTP_REFERER} ^http://(.*@)?([a-z0-9]+\.)*(refblock\.com|tamanria\.com|\.ru)(:[0-9]+)?(/.*)?$ [NC]
RewriteRule .* - [F,L]

I've removed the check for the Refererblock.gef, as it is not one of the blocked extensions and was a needless check. When I wrote the rules earlier, I misread it as .gif rather than .gef. Also, as a matter of verifying the above .htaccess, I noticed that your .gef file is served with the incorrect mime type of text/plain rather than image/gif. I've included the AddType directive above to fix it so that Firefox users won't be prompted to save the image to disk.

I don't know whether you wanted to parse .html files for SSI and for php. I've removed the .html parsing as Apache doesn't really handle that too well. With Apache2, you should use the Handlers to parse the page if you need to do both PHP and SSI on the same page.

Hopefully you get this information quickly before it gets edited/downvoted into oblivion. Though, if history has taught me anything about this site, we'll see the answer merged into someone's response that has much higher karma.