Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [robots.txt]

Convention to prevent webcrawlers from indexing your website.

Filter by
Sorted by
Tagged with
1
vote
2answers
477 views

How to block Spiders / Scrapers by REMOTE hostname / domain for all Virtual Hosts in Apache?

I've seen plenty of robot.txt stuff, and some mod-rewrite solutions that looked promising… but haven't been able to find a simple solution to block Spiders / Scrapers / whoever the hell I want to ...
3
votes
3answers
176 views

How to prevent discovery of a secure URL?

If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible ...
1
vote
2answers
829 views

Is there a way to disallow robots crawling through IIS Management Console for entire site

Can I do the same as robots.txt through IIS settings? Telling User-agent: * Disallow: / in host header or through web.config?
3
votes
2answers
4k views

Is it good idea to ban amazonaws.com [closed]

Site are crawled by anonymous bot hosted on amazon ec2. This robot doesn't respect robots.txt and creates high load on web server so I added check if reverse IP for request ends with "amazonaws.com" ...
2
votes
1answer
88 views

Use robots.txt to prevent crawlers from getting old versions of Trac pages

looking at my Apache access.log I see that crawlers tend to get old versions of pages and documents, like: 119.63.196.86 - - [10/Jun/2011:10:36:31 +0200] "GET /wiki/News?version=14 HTTP/1.1" 200 6073 ...
1
vote
2answers
425 views

Weird entry in access.log on Apache 2.2

I'm running Apache 2.2, and my server runs well. Noticed this weird anomaly in my access.log file, how should I prevent it? robots.txt doesn't seem to be working. 127.0.0.1 - - [17/Apr/2011:12:17:00 +...
0
votes
1answer
298 views

Disallow xml robots.txt

Google webmaster FAQs suggest that this will exclude all xml files from search: User-agent: Googlebot Disallow: /*.xml$ Is this legal for other bots as well? User-agent: * Disallow: /*.xml$
0
votes
1answer
417 views

Cross-submission robots.txt for multiple domains on single host

We are running a site with multiple languages hosted in a single environment on IIS7. For example, oursite.com - english oursite.de - german oursite.es - spanish This is a single-host environment....
10
votes
4answers
13k views

How to create robots.txt file for all domains on Apache server

We have a XAMPP Apache development web server setup with virtual hosts and want to stop serps from crawling all our sites. This is easily done with a robots.txt file. However, we'd rather not ...
0
votes
2answers
1k views

301 redirect or disallow on robots.txt?

I recently asked for 301 redirection on ServerFault and I didn't get a proper solution to my problem, but now I have a new idea: use the robots.txt to disallow certain URLs from my site to be "crawled"...
1
vote
3answers
719 views

Should I ban spiders?

A rails template script that I've been looking at automatically adds User-Agent: and Dissalow: in robots.txt thereby banning all spiders from the site What are the benefits of banning spiders and why ...
0
votes
2answers
102 views

Is this a valid robots.txt file?

I have this robots.txt file: User-agent: * Sitemap: Path_to_sitemap.xml My Q is, should I have something else in there as well? Like allow All or something? Thanks
22
votes
4answers
20k views

How Can I Encourage Google to Read New robots.txt File?

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update. Is there any way I can encourage Google to re-read my robots....
14
votes
5answers
2k views

Which bots and spiders should I block in robots.txt?

In order to: Increase security of my website Reduce bandwidth requirements Prevent email address harvesting
6
votes
4answers
9k views

How do you create a single robots.txt file for all sites on an IIS instance

I want to create a single robots.txt file and have it served for all sites on my IIS (7 in this case) instance. I do not want to have to configure anything on any individual site. How can I do this?
0
votes
2answers
143 views

Quick Robots.txt question

Will the following robots.txt syntax correctly block all pages on the site that end in "_.php"? I don't want to accidentally block other pages. User-Agent: * Disallow: /*_.php Also, am I allowed ...
2
votes
6answers
15k views

What happens if a website does not have a robots.txt file?

If the robots.txt file is missing in the root directory of a website, how are things treated as: the site is not indexed at all the site is indexed without any restrictions It should logically be ...
1
vote
2answers
133 views

robots.txt file with more restrictive rules for certain user agents

I'm a bit vague on the precise syntax of robots.txt, but what I'm trying to achieve is: Tell all user agents not to crawl certain pages Tell certain user agents not to crawl anything (basically, ...
0
votes
1answer
223 views

seems to block /my-beautiful-sef-url-123

I have robots.txt that looks like this: User-agent: * Disallow: /system/ Disallow: /admin/ Disallow: /index.php The obvious goal has been to prevent all the ugly URLs from being indexed, as they all ...
2
votes
3answers
335 views

How much HDD space would I need to cache the web while respecting robot.txts? [closed]

I want to experiment with creating a web crawler. I'll start with indexing a few medium sized website like Stack Overflow or Smashing Magazine. If it works, I'd like to start crawling the entire web. ...
2
votes
2answers
23k views

Meaning of Disallow: /*? in robots.txt

Yahoo's robots.txt contains: User-agent: * Disallow: /p/ Disallow: /r/ Disallow: /*? What does the last line mean? ("Disallow: /*?")
5
votes
6answers
11k views

Blocking yandex.ru bot

I want to block all request from yandex.ru search bot. It is very traffic intensive (2GB/day). I first blocked one C class IP range, but it seems this bot appear from different IP ranges. For example:...
1
vote
3answers
361 views

Does GoogleBot respect User-agent: *

I blocked a page in robots.txt under User-agent: *, and tried to do a manual removal of that URL from Google's cache in the webmasters tools. Google said it wasn't being blocked in my robots.txt, so I ...
1
vote
7answers
170 views

Robots.txt command

I have a bunch of files at www.example.com/A/B/C/NAME (A,B,C change around, NAME is static) and I basically want to add a command in robots.txt so crawlers don't follow any such links that have NAME ...
0
votes
1answer
245 views

Make Google Apps site publicly accessible while disabling crawlers with robots.txt?

I would like to create a publicly accessible Google Apps site (i.e. users do not need to be authenticated to access the content) while maintaining a policy crawlers and bots exclusion with Robots.txt. ...
0
votes
1answer
120 views

How to prevent search engines indexing a specific url

I have a url which i don't want indexed: http://www.mysite.com/moduleA?param=secretkey So when i google search for "mysite.com", i don't want the above link to appear in the search results. However,...
1
vote
3answers
5k views

robots.txt and other .txt returning 404 on IIS?

We have an IIS site running Dotnetnuke that we took over from another group. We have added a robots.txt file to the root but it returns a 404. Actually any txt file in the root seems to return 404. ...
0
votes
2answers
793 views

Ideal robots.txt for WordPress?

I browsed the web trying to find the ideal robots.txt content for a hosted WordPress blog. I found several options, for example here and here. I thought this would be a good question for ServerFault: ...
3
votes
2answers
1k views

Robots.txt - no follow, no index

Please can someone explain to me the difference between setting allow and disallow in a robots.txt file and create No follow, No index meta tags! Is it possible to set no follow and no index within ...