I know can create ONE robots.txt file for all domains on an Apache server*, but I want to append to each domain's (if pre-existing) robots.txt. I want some general rules in place for all domains, but I need to allow different domains to have their unique rules.

Is there a way to accomplish this?

(*In my case Apache 2.2.x)


From Apache's point of view, robots.txt is just an asset to be served. You can alter the content returned when robots.txt is requested by passing it through an output filter.

If you want to append some text, you could define an external filter. Assuming that Apache is running on Unix-like operating system, the filter configuration could be

ExtFilterDefine appendRobotstxt cmd="/bin/cat - /var/www/html/robots-tail.txt"
<Location /robots.txt>
    SetOutputFilter appendRobotstxt

That would concatenate robots-tail.txt to the end of the response.

  • Apache is on CentOS 6.3. I will give it a try. Thanks. – Gaia Nov 3 '12 at 14:39
  • Do you know how should I define intype so that it only filters robots.txt ? – Gaia Nov 3 '12 at 15:54
  • @Gaia The filter is activated not by the intype of the filter definition, but by a SetOutputFilter directive. <Location /robots.txt> SetOutputFilter appendRobotsTail </Location> – 200_success Nov 4 '12 at 9:39
  • ExtFilterDefine appendRobotstxt cmd="cat - /root/robots-tail.txt" <Location /robots.txt> SetOutputFilter appendRobotstxt </Location> inside httpd.conf gets me a 500 for robots.txt. I gave /root/robots-tail.txt 777 for testing purposes, and apache runs suexec/fcgid. – Gaia Nov 5 '12 at 20:10
  • SELinux is disabled. placing the file in public_html in one of the vhosts allows for an error to be displayed: No such file or directory: couldn't create child process to run cat followed by No such file or directory: can't initialise output filter appendrobotstxt: aborting. Server is running suExec. – Gaia Nov 6 '12 at 1:48

Note that you'd probably have to incorporate the changes. If a domain already has

User-agent: *
Disallow: /search

and you want to add for all domains

User-agent: *
Disallow: /admin/

you'd have to make it

User-agent: *
Disallow: /search
Disallow: /admin/

because robots.txt parsers stop as soon as they found a block that matches them.

  • I want to disallow completely a few user-agents. Thanks. – Gaia Nov 3 '12 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.