Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have experienced some instability on our server lately. Checking log/messages there is a lot of messages like this

Apr 25 20:27:17 lillevinkel2 drupal: http://skobloggen.lillevinkelsko.no|1335378437|page not found|122.232.5.21|http://skobloggen.lillevinkelsko.no/http://www.magicptp.com/promote.php?id=dongxh|http://pulsreklamy.pl/|0||promote.php

They are from a lot of different ip adresses. I added in .htaccess that everything with http in should be denied. pasting the url from the log results in access denied in browser, but the log is still filling with this. What can I do to prevent this kind of spidering or stop the requests before they result in a page not found?

share|improve this question

1 Answer 1

If it's a matter of the logs filling up, you can stop logging those entries using SetEnvIf. Something like:

SetEnvIf Request_URI "http" NoLog
CustomLog logs/access_log common env=!NoLog

More documentation here:

http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html and http://httpd.apache.org/docs/2.2/env.html

share|improve this answer
    
That will clean up the log and is very useful. Thx. But I would also like to stop it with "access denied" before there is a "page not found". My access denied rewrite rule works when I paste the URL into the browser, but still it is coming new "page not found" messages. –  Tom Apr 26 '12 at 20:00
    
Look at: httpd.apache.org/docs/2.2/howto/access.html#env You can do a similar Deny from env=NoLog with the same SetEnvIf pattern match. –  cjc Apr 26 '12 at 20:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.