Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

For my staging server, I'm trying to figure out a way to block search engine bots entirely, rather than an individual .htaccess file or robots.txt file. The idea is that it's out of sight and out of mind when creating a new site on the staging server. Is there a way to detect the bot's user agent using the apache module and block that connection on a sever-wide level?

Thanks!

share|improve this question

1 Answer 1

My suggestion would be to block everything OTHER than a known-good testing agent string. That way you block bots you've never heard of. You could also block all but a known good set of IPs with %{REMOTE_ADDR}.

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} !YourTestingAgent
RewriteRule (.*) /404.html
share|improve this answer
    
That's a great idea. Are you able to place server-wide rewrite rules in the apache configuration file? –  Jesse Bunch Jun 29 '11 at 20:09
    
Yeah, I think so. The docs say it can go in at the "server config" level: httpd.apache.org/docs/2.0/mod/mod_rewrite.html#rewritecond –  Mark Jun 29 '11 at 20:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.