Questions tagged [robots.txt]
Convention to prevent webcrawlers from indexing your website.
3
questions
10
votes
4
answers
22k
views
How to create robots.txt file for all domains on Apache server
We have a XAMPP Apache development web server setup with virtual hosts and want to stop serps from crawling all our sites. This is easily done with a robots.txt file. However, we'd rather not ...
- 203
23
votes
5
answers
22k
views
How Can I Encourage Google to Read New robots.txt File?
I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.
Is there any way I can encourage Google to re-read my robots....
- 1,434
1
vote
2
answers
551
views
How to block Spiders / Scrapers by REMOTE hostname / domain for all Virtual Hosts in Apache?
I've seen plenty of robot.txt stuff, and some mod-rewrite solutions that looked promising… but haven't been able to find a simple solution to block Spiders / Scrapers / whoever the hell I want to ...
- 1,431