I have multiple physical sub-domains and I don't want to change any robots.txt file of any of that sub-domains.

Is there any way to disallow all the sub-domains from my main domain's physical robots.txt file without using any sub-domain's physical file?

Any common server (Apache's) file which can access all the sub-domains and main domain, too?

closed as unclear what you're asking by Jenny D, peterh, Ward, mdpc, MadHatter Jun 15 '17 at 13:35

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.


It's impossible to say anything about subdomain.example.com on example.com/robots.txt.

The robots.txt has really limited syntax, e.g.

User-agent: Google
Disallow: /administrator

User-agent: *
Disallow: /

Where the User-agent: defines the search engine and Disallow: the path related to server root. In this example Google allowed crawl anything but /administrator, all disallowed for rest. As always with robots.txt it doesn't limit anything; it's merely a beautiful wish not to go there.

The syntax simply has no place for a subdomain and the Web Robot only looks for /robots.txt i.e. subdomain.example.com/robots.txt, not example.com/robots.txt.


Assuming that by 'domain', you mean something like example.com and by sub-domain you mean blerf.example.com, then I believe the answer to be 'you can't do that'.

The problem is that when a crawler tries to crawl blerf.example.com, it looks at blerf.example.com/robots.txt to see what it shouldn't crawl. It doesn't look at example.com/robots.txt, because that's a different domain.

One explanation of robots.txt operation can be found at http://www.robotstxt.org/robotstxt.html .

Not the answer you're looking for? Browse other questions tagged or ask your own question.