Convention to prevent webcrawlers from indexing your website.
1
vote
1answer
15 views
Ideal robots.txt for a gitweb installation?
I host a few git repositories at git.nomeata.de using gitweb (and gitolite). Occasionally, a search engine spider comes along and begins to hammer the interface. While I generally do want my git ...
0
votes
2answers
52 views
Dynamic robots.txt based on hostname
Is there a way to swap out a robots.txt file in nginx based on hostname? I currently have www.domain.com and backup.domain.com pointing at the same nginx server, but I don't want Google indexing ...
0
votes
1answer
32 views
Is there a chance to block images spiders / bots on dedicated servers without using robots.txt or .htaccess?
We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf.
But that requires to edit may be a large ...
0
votes
1answer
57 views
Blocking bad bots
I found this script and was wondering if this is just overkill and even worth using?
Is it better for me to just use mod_security?
# Generated using http://solidshellsecurity.com services
# Begin ...
0
votes
1answer
23 views
robots.txt for subdomain iis7
I have two different sites in iis7
both point to the same folder
they have different subdomains
www.sitename.com
foo.sitename.com
they are essentially the same website, but it runs different logic ...
0
votes
1answer
33 views
If I redirect all users (Except me) from within .htaccess, do I need a robots.txt file
So.. I have my live version of my site e.g v1.0 at domain.com
I then have my development testing version at testing.domain.com
I want testing.domain.com to only be accessible to me for testing, and ...
2
votes
3answers
114 views
Block Offline Browsers
Is there a way to block offline browsers (like Teleport Pro, Webzip, etc...) that are showed in the logs as "Mozilla"?
Example:
Webzip is showed in my site logs as "Mozilla/4.0 (compatible; MSIE 8.0; ...
2
votes
2answers
56 views
IIS Spikes in anonymous users - crippling my server
I have a server running windows server 2008 R2, recently my websites have becoming unresponsive at least once a day, seemingly at random intervals.
I have installed some monitoring software and ...
0
votes
0answers
44 views
Valid robots.txt? [closed]
I am waiting for Google to crawl my site and display the results in search. Is my robots.txt alright and will it let google, bing etc crawl my site? Thanks!
User-agent: *
Disallow: /cgi-bin/
...
1
vote
0answers
82 views
How to block urls that request for robots.txt in lighttpd?
We have a dedicated development server which runs only test PHP applications on a public network.
We have setup session-based authentication for the site.
The issue we have is there are lots of 404s ...
1
vote
2answers
239 views
How to create a global robots.txt that gets appended to each domain's own robots.txt on Apache?
I know can create ONE robots.txt file for all domains on an Apache server*, but I want to append to each domain's (if pre-existing) robots.txt. I want some general rules in place for all domains, but ...
1
vote
1answer
38 views
How to Disallow Particular Path in robots.txt
I want to disallow /path but also wanna allow /path/another-path in robots.txt. I already tried:
Disallow: /path
Or:
Disallow: /path$
But doesn't work, I mean it blocked /path/another-path too. ...
-2
votes
2answers
152 views
how to rewrite or redirect old or missing or invalid url to 404 page [duplicate]
Possible Duplicate:
Everything You Ever Wanted to Know about Mod_Rewrite Rules but Were Afraid to Ask?
I recently upgraded a site and almost all URLs have changed. I have redirected all of ...
3
votes
3answers
589 views
How do I use robots.txt to disallow crawling for only my subdomains?
If I want my main website to on search engines, but none of the subdomains to be, should I just put the "disallow all" robots.txt in the directories of the subdomains? If I do, will my main domain ...
0
votes
1answer
230 views
webcrawler bots load test my website and it fails the test
we run a commerical website with a relavitely few number of customer at any one time ~30 users .
Frequently a webcrawler such as google bot, bing bot, and 80legs will bring our site to a grinding ...
0
votes
0answers
82 views
Robots.txt cannot be found
I am having a bit of problem viewing my robots.txt file - when I try to check my robots.txt file in IE it brings up "Internet Explorer cannot find the Webpage". In Google Chrome it says "This web page ...
1
vote
3answers
965 views
Why are Facebook's servers hitting my site nearly once per minute (and the time gap is decreasing)?
For about 4 hours now Facebook's servers have been hitting my site every 5 mins at the start, but now at least once per minute.
Getting this in logs
66.220.153.246 - - [25/Apr/2012:20:08:49 +0000] ...
1
vote
1answer
38 views
If denying crawlers access to a directory via robots.txt, will it still index a file in that directory if I direct link?
I am denying indexing to a folder called pdf via robots.txt. However, I do direct link to a few files that exist in that directory.
Will search engines such as Google index those files, or ignore ...
0
votes
2answers
327 views
How can I use varnish to generate a robots.txt file even for subdomain of the same site?
I want to generate a robots.txt file using Varnish 2.1.
That means that domain.com/robots.txt is served using Varnish and also subdomain.domain.com/robots.txt is also served using Varnish.
The ...
0
votes
1answer
137 views
Does a forward web proxy exist that checks and obeys robots.txt on remote domains?
Does there exist a forward proxy server that will lookup and obey robots.txt files on remote internet domains and enforce them on behalf of requesters going via the proxy?
e.g. Imagine a website at ...
1
vote
2answers
243 views
How to block Spiders / Scrapers by REMOTE hostname / domain for all Virtual Hosts in Apache?
I've seen plenty of robot.txt stuff, and some mod-rewrite solutions that looked promising… but haven't been able to find a simple solution to block Spiders / Scrapers / whoever the hell I want to ...
3
votes
3answers
139 views
How to prevent discovery of a secure URL?
If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible ...
1
vote
2answers
143 views
Is there a way to disallow robots crawling through IIS Management Console for entire site
Can I do the same as robots.txt through IIS settings?
Telling
User-agent: * Disallow: /
in host header or through web.config?
0
votes
2answers
669 views
Is it good idea to ban amazonaws.com [closed]
Site are crawled by anonymous bot hosted on amazon ec2. This robot doesn't respect robots.txt and creates high load on web server so I added check if reverse IP for request ends with "amazonaws.com" ...
2
votes
1answer
65 views
Use robots.txt to prevent crawlers from getting old versions of Trac pages
looking at my Apache access.log I see that crawlers tend to get old versions of pages and documents, like:
119.63.196.86 - - [10/Jun/2011:10:36:31 +0200] "GET /wiki/News?version=14 HTTP/1.1" 200 6073 ...
1
vote
2answers
242 views
Weird entry in access.log on Apache 2.2
I'm running Apache 2.2, and my server runs well. Noticed this weird anomaly in my access.log file, how should I prevent it? robots.txt doesn't seem to be working.
127.0.0.1 - - [17/Apr/2011:12:17:00 ...
0
votes
1answer
98 views
Disallow xml robots.txt
Google webmaster FAQs suggest that this will exclude all xml files from search:
User-agent: Googlebot
Disallow: /*.xml$
Is this legal for other bots as well?
User-agent: *
Disallow: /*.xml$
0
votes
1answer
176 views
Cross-submission robots.txt for multiple domains on single host
We are running a site with multiple languages hosted in a single environment on IIS7. For example,
oursite.com - english
oursite.de - german
oursite.es - spanish
This is a single-host ...
5
votes
4answers
2k views
How to create robots.txt file for all domains on Apache server
We have a XAMPP Apache development web server setup with virtual hosts and want to stop serps from crawling all our sites. This is easily done with a robots.txt file. However, we'd rather not ...
0
votes
1answer
456 views
301 redirect or disallow on robots.txt?
I recently asked for 301 redirection on ServerFault and I didn't get a proper solution to my problem, but now I have a new idea: use the robots.txt to disallow certain URLs from my site to be ...
1
vote
3answers
224 views
Should I ban spiders?
A rails template script that I've been looking at automatically adds User-Agent: and Dissalow: in robots.txt thereby banning all spiders from the site
What are the benefits of banning spiders and why ...
0
votes
2answers
82 views
Is this a valid robots.txt file?
I have this robots.txt file:
User-agent: *
Sitemap: Path_to_sitemap.xml
My Q is, should I have something else in there as well? Like allow All or something?
Thanks
10
votes
4answers
8k views
How Can I Encourage Google to Read New robots.txt File?
I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.
Is there any way I can encourage Google to re-read my ...
12
votes
5answers
826 views
Which bots and spiders should I block in robots.txt?
In order to:
Increase security of my website
Reduce bandwidth requirements
Prevent email address harvesting
2
votes
4answers
2k views
How do you create a single robots.txt file for all sites on an IIS instance
I want to create a single robots.txt file and have it served for all sites on my IIS (7 in this case) instance.
I do not want to have to configure anything on any individual site.
How can I do this?
...
0
votes
2answers
99 views
Quick Robots.txt question
Will the following robots.txt syntax correctly block all pages on the site that end in "_.php"? I don't want to accidentally block other pages.
User-Agent: *
Disallow: /*_.php
Also, am I ...
0
votes
6answers
597 views
What happens if a website does not have a robots.txt file?
If the robots.txt file is missing in the root directory of a website, how are things treated as:
the site is not indexed at all
the site is indexed without any restrictions
It should logically be ...
1
vote
2answers
95 views
robots.txt file with more restrictive rules for certain user agents
I'm a bit vague on the precise syntax of robots.txt, but what I'm trying to achieve is:
Tell all user agents not to crawl certain pages
Tell certain user agents not to crawl anything
(basically, ...
0
votes
1answer
206 views
seems to block /my-beautiful-sef-url-123
I have robots.txt that looks like this:
User-agent: *
Disallow: /system/
Disallow: /admin/
Disallow: /index.php
The obvious goal has been to prevent all the ugly URLs from being indexed, as they ...
2
votes
3answers
220 views
How much HDD space would I need to cache the web while respecting robot.txts? [closed]
I want to experiment with creating a web crawler. I'll start with indexing a few medium sized website like Stack Overflow or Smashing Magazine. If it works, I'd like to start crawling the entire web. ...
2
votes
2answers
2k views
Meaning of Disallow: /*? in robots.txt
Yahoo's robots.txt contains:
User-agent: *
Disallow: /p/
Disallow: /r/
Disallow: /*?
What does the last line mean? ("Disallow: /*?")
4
votes
7answers
6k views
Blocking yandex.ru bot
I want to block all request from yandex.ru search bot. It is very traffic intensive (2GB/day).
I first blocked one C class IP range, but it seems this bot appear from different IP ranges.
For ...
1
vote
3answers
296 views
Does GoogleBot respect User-agent: *
I blocked a page in robots.txt under User-agent: *, and tried to do a manual removal of that URL from Google's cache in the webmasters tools. Google said it wasn't being blocked in my robots.txt, so I ...
0
votes
7answers
117 views
Robots.txt command
I have a bunch of files at www.example.com/A/B/C/NAME (A,B,C change around, NAME is static) and I basically want to add a command in robots.txt so crawlers don't follow any such links that have NAME ...
0
votes
1answer
224 views
Make Google Apps site publicly accessible while disabling crawlers with robots.txt?
I would like to create a publicly accessible Google Apps site (i.e. users do not need to be authenticated to access the content) while maintaining a policy crawlers and bots exclusion with Robots.txt. ...
0
votes
1answer
79 views
How to prevent search engines indexing a specific url
I have a url which i don't want indexed:
http://www.mysite.com/moduleA?param=secretkey
So when i google search for "mysite.com", i don't want the above link to appear in the search results.
...
0
votes
3answers
1k views
robots.txt and other .txt returning 404 on IIS?
We have an IIS site running Dotnetnuke that we took over from another group. We have added a robots.txt file to the root but it returns a 404. Actually any txt file in the root seems to return 404.
...
0
votes
2answers
660 views
Ideal robots.txt for WordPress?
I browsed the web trying to find the ideal robots.txt content for a hosted WordPress blog. I found several options, for example here and here.
I thought this would be a good question for ServerFault: ...
3
votes
2answers
1k views
Robots.txt - no follow, no index
Please can someone explain to me the difference between setting allow and disallow in a robots.txt file and create No follow, No index meta tags!
Is it possible to set no follow and no index within ...
2
votes
2answers
547 views
What does the arxiv.org anti-bot “search and destroy” actually do?
The lanl.arxiv.org math and scientific preprint service (formerly known as xxx.lanl.gov) has a strict policy against bots that ignore its robots.txt, Robots Beware. On that page, the have a link ...
