Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For my staging server, I'm trying to figure out a way to block search engine bots entirely, rather than an individual .htaccess file or robots.txt file. The idea is that it's out of sight and out of mind when creating a new site on the staging server. Is there a way to detect the bot's user agent using the apache module and block that connection on a sever-wide level?


share|improve this question

My suggestion would be to block everything OTHER than a known-good testing agent string. That way you block bots you've never heard of. You could also block all but a known good set of IPs with %{REMOTE_ADDR}.

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} !YourTestingAgent
RewriteRule (.*) /404.html
share|improve this answer
That's a great idea. Are you able to place server-wide rewrite rules in the apache configuration file? – Jesse Bunch Jun 29 '11 at 20:09
Yeah, I think so. The docs say it can go in at the "server config" level: – Mark Jun 29 '11 at 20:24

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.