Questions tagged [robots.txt]
Convention to prevent webcrawlers from indexing your website.
89
questions
0
votes
0
answers
57
views
What to do with borderline DoS traffic? [closed]
My site has changed systems. It used to hold json files with details of content that would be processed by JavaScript for table displays. Those have been removed for a couple weeks and now my logs ...
- 101
30
votes
5
answers
79k
views
How to set robots.txt globally in nginx for all virtual hosts
I am trying to set robots.txt for all virtual hosts under nginx http server.
I was able to do it in Apache by putting the following in main httpd.conf:
<Location "/robots.txt">
SetHandler ...
- 103
0
votes
0
answers
80
views
How to block some bots in .htaccess but except access robots.txt?
I've blocked some bad bots using .htaccess.
For example:
SetEnvIfNoCase User-Agent "^bot1" bots
SetEnvIfNoCase User-Agent "^bot2" bots
SetEnvIfNoCase User-Agent "^bot3" ...
- 1
1
vote
0
answers
79
views
Traefik, docker swarm and portainer. Serving robots.txt file
I'm playing around with my homelab and I'm trying to include robots.txt file.
I'm launching traefik and portainer using this docker_compose file. This is using Docker swarm mode
version: "3.3&...
- 111
0
votes
1
answer
543
views
Is there a way of telling web crawlers / robots about number of requests limit per second / minute / etc
I was thinking on a way similar to robots.txt that is used by good bots crawling a website. In the robots.txt I can define the User-agent, Allow and Disallow.
My goal is to pass the message about the ...
- 37.6k
0
votes
1
answer
416
views
Alias in apache httpd.conf
I am trying to serve robots.txt on an existing https site being served via reverse proxy on an apache windows server.
Here is what I added to httpd.conf file however this is not working. What am I ...
- 1,849
0
votes
2
answers
2k
views
how to write RewriteCond for a url with subdomain
I am trying to ban some bots by writing a RewriteCond rule in htaccess file. Is the following ruleset correct if I add the following:
## block traffic from particular referrers
RewriteEngine On
...
CommunityBot
- 1
6
votes
6
answers
30k
views
What happens if a website does not have a robots.txt file?
If the robots.txt file is missing in the root directory of a website, how are things treated as:
the site is not indexed at all
the site is indexed without any restrictions
It should logically be ...
CommunityBot
- 1
0
votes
1
answer
364
views
Wordpress - Yoast SEO - robots.txt
I was just reading up on Yoast website here on how the correct robots.txt file should look like with the latest SEO practices. In their example Yoast uses the following:
User-Agent: *
Disallow: /...
0
votes
0
answers
1k
views
nginx configuration for robots.txt
I've read other answers and Nginx docs, and I can't figure out why this works:
location = /robots.txt {
alias /<fullpath>/static/robots.allow.txt;
}
and this don't:
location = /robots.txt {
...
- 101
0
votes
0
answers
97
views
Make webserver to prevent parsing of certain HTML elements
MediaWiki content management system creates many links which their webpages I want not to be discovered by search engine crawlers.
It's not only that I don't want them indexed and more so not only ...
10
votes
4
answers
22k
views
How to create robots.txt file for all domains on Apache server
We have a XAMPP Apache development web server setup with virtual hosts and want to stop serps from crawling all our sites. This is easily done with a robots.txt file. However, we'd rather not ...
CommunityBot
- 1
3
votes
2
answers
43k
views
Meaning of Disallow: /*? in robots.txt
Yahoo's robots.txt contains:
User-agent: *
Disallow: /p/
Disallow: /r/
Disallow: /*?
What does the last line mean? ("Disallow: /*?")
- 1
-1
votes
1
answer
264
views
Disallow: /?q=search/ in robot.txt
Does /?q=search/ means I can't web scrape the search websites that ends with =search/ ? Can I scrape an URL ends with =0#search ?
- 1,342
0
votes
0
answers
478
views
With Nginx / Node.js reverse proxy how does Nginx serve robots.txt despite txt files not being referenced in Nginx config's location blocks?
In Chrome when I enter https://www.example.com/robots.txt my robots.txt file is served and works fine. I'm happy that it works but I'm not sure why it does.
In the config below I thought that my last ...
1
vote
1
answer
401
views
Tons of Access from Google Proxy
I freaquently have a lots of access from google proxy. It says it is Google Favicon bot and I've checked it by host command. User-agent is like following.
"Mozilla/5.0 (X11; Linux x86_64) ...
- 11
1
vote
0
answers
28
views
What are they trying to get with "GET /public-projects"
I haven't even shared my website with anyone yet and I have already started seeing attempts to GET /public-projects. However, I couldn't get any information about it, what are they trying to get? The ...
- 111
2
votes
2
answers
4k
views
Rewrite robots.txt based on host with htaccess
I'm trying to rewrite a filename based on the server's domain.
This code below is wrong / not working, but illustrates the desired effect.
<If "req('Host') != '*.mydevserver.com'">
...
- 31
-1
votes
1
answer
108
views
How to write disallow paths for comments when their urls keep on changing
I want to disallow pages that develop after someone comments on my post. For example, recently I got a comment (number 12) on my post page and a new page was formed with URL:
https://example.com/post/#...
- 4,514
23
votes
5
answers
22k
views
How Can I Encourage Google to Read New robots.txt File?
I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.
Is there any way I can encourage Google to re-read my robots....
- 71
1
vote
0
answers
47
views
How to block bad url path that is not part of my site from showing in google search?
I have got a site that is running on Node.js (Express) , and Apache httpd.
Hundreds of requests are coming in from malicious IP's, which I'm proactively blocking. (I have a script that looks at the ...
- 123
8
votes
3
answers
11k
views
How do I use robots.txt to disallow crawling for only my subdomains?
If I want my main website to on search engines, but none of the subdomains to be, should I just put the "disallow all" robots.txt in the directories of the subdomains? If I do, will my main domain ...
1
vote
3
answers
3k
views
Dynamic robots.txt based on hostname
Is there a way to swap out a robots.txt file in nginx based on hostname? I currently have www.domain.com and backup.domain.com pointing at the same nginx server, but I don't want Google indexing ...
- 111
3
votes
3
answers
1k
views
robots.txt is redirecting to default page
Hullo,
Typically, if I type into my address bar, "oneofmysites.com/robots.txt", any browser will display the content of robots.txt. As you can see, this is pretty standard behaviour.
I have just ...
- 111
1
vote
1
answer
1k
views
robots.txt route requires a backslash when behind an Application Load Balancer
I have a rails site using an AWS ALB and all routes appear to work except one, robots.txt.
I am getting the error "ERR_TOO_MANY_REDIRECTS", link to example: https://www.mamapedia.com/robots.txt
...
- 80.3k
1
vote
1
answer
567
views
Redirect "robots.txt" on specific domain
I want to redirect all requests on "robots.txt" if the domain contains ".our-internal-devel-domain.de". It should be server-wide, because when we develop a website and publish it over our test-domain, ...
- 8,926
-1
votes
2
answers
2k
views
How to disallow crawling for all subdomains using my main domain's physical robots.txt file [closed]
I have multiple physical sub-domains and I don't want to change any robots.txt file of any of that sub-domains.
Is there any way to disallow all the sub-domains from my main domain's physical robots....
- 49.8k
1
vote
1
answer
329
views
High no of hits by facebook crawler on server
There are daily about 3000 404 hits or more from facebook crawler. Log is as
X.X.X.X Y.Y.Y.Y - - [24/May/2017:03:43:35 +0000] "GET /health-and-medicine/trumps-2018-budget-cuts-funding-for-cancer-...
- 110k
2
votes
1
answer
1k
views
Remove ?=collcc from url
Google Webmasters Tools has notified me about too many duplicated URLs. Some parameters have been added that I don't know about and I need to remove it, for example:
http://example.com/5454/my-utr....
- 1,272
5
votes
1
answer
10k
views
Nginx robots.txt configuration
I can't seem to properly configure nginx to return robots.txt content. Ideally, I don't need the file and just want to serve text content configured directly in nginx. Here's my config:
server {
...
CommunityBot
- 1
0
votes
2
answers
3k
views
301 redirect or disallow on robots.txt?
I recently asked for 301 redirection on ServerFault and I didn't get a proper solution to my problem, but now I have a new idea: use the robots.txt to disallow certain URLs from my site to be "crawled"...
CommunityBot
- 1
2
votes
1
answer
520
views
What's with random-character queries coming from googlebot, e.g., vvytnoxvontwusz.html?
One of my sites has been getting queries from googlebot, on the order of:
example-log:66.249.79.216 - - [06/Apr/2016:15:36:56 -0700] "GET /vvytnoxvontwusz.html HTTP/1.1" 404 15136 "-" "Mozilla/5.0 (...
CommunityBot
- 1
-1
votes
1
answer
399
views
Robots.txt - disallow crawling one directory on subdomain
I have placed my product showcase on subdomain such as http://demo.domain.com/productname/. Demo version of product is located at http://demo.domain.com/productname/demo/. I would like to disallow ...
- 103
0
votes
2
answers
2k
views
How to serve robots.txt for all my own subdomains but not other hosts on Apache?
We develop websites and we host the QA environment on the same server as the production environment. I want to serve a specific robots.txt for all QA sites but not for the production sites. We have a ...
- 3,053
3
votes
1
answer
843
views
Baidu Spider causing 3Gb of traffic a day - but I do business in China
I'm in a difficult situation, the Baidu spider is hitting my site causing about 3Gb a day worth of bandwidth. At the same time I do business in China so don't want to just block it.
Has anyone else ...
- 22.5k
1
vote
1
answer
886
views
Custom robots.txt being overwritten in Azure IIS 8 by something
We have a custom robots.txt in the root of our IIS cloud service Azure website that does not display correctly when navigating to www.oursite.com/robots.txt . A “different” robots.txt file displays ...
- 101
2
votes
1
answer
1k
views
Google-Bot fell in love with my 404-page
Every day my access-log looks kind of this:
66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot....
- 30.7k
-3
votes
2
answers
112
views
Restricting Access from BOTS
I would like to protect my server from too many hits from Bots.
Considering a scenario, where in a server (physical) located in a private network and hitting my server continuously. Do i have a ...
- 4,757
1
vote
0
answers
2k
views
How to block fake google spider and fake web browser access?
Recently I found that someguys are trying to mirror my website. They are doing this in two ways:
Pretend to be google spiders . Access logs are as following:
89.85.93.235 - - [05/May/2015:20:23:16 +...
- 99.6k
-1
votes
1
answer
1k
views
Block all bots from .ru domains via robots.txt or htaccess? [closed]
Is there any way to write rules for robots.txt or htaccess that will block all bots that come from a .ru domain?
Thanks
- 3,663
0
votes
1
answer
1k
views
Robots.txt with several VirtualHosts
My web server is running (apache 2.4.10) different virtual hosts for the following domain name :
foo.example.com
bar.example.com
www.example.com
example.com
Here is the configuration file for my ...
- 116k
0
votes
1
answer
706
views
Robots file behaviour
I've noticed something odd today.
if i go to http://www.google.com/robots.txt, IE11 shows me the contents of google's robot file. However if i go to my site (still in development) using the same ...
- 2,970
2
votes
1
answer
262
views
Ideal robots.txt for a gitweb installation? [closed]
I host a few git repositories at git.nomeata.de using gitweb (and gitolite). Occasionally, a search engine spider comes along and begins to hammer the interface. While I generally do want my git ...
- 193
0
votes
0
answers
52
views
Allocating percent resources for all robots
As many who posted here, my apache server is getting brutally hammerred to near death by robots, most of which are good robots.
There is no way to change their crawl rate.
Anyone have a suggestion ...
- 101
-4
votes
1
answer
203
views
How Can I Encourage Google to scan New robots.txt File?
I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update.
my last robots.txt had a "disallow: all" raw.
Is there any way I ...
- 176
1
vote
1
answer
1k
views
apache robots.txt with SSL
I have an .htaccess file with a rewrite rule to get a redirect of every HTTP request to HTTPS.
But now I have a problem that my robots.txt is not recognized by some online checker.
If I remove the ...
- 6,662
1
vote
2
answers
2k
views
Thousands of robots.txt 404 errors from bots trying to crawl old multisite
Current situation is that we are getting thousands and thousands of 404 errors from bots looking for robots.txt in different places on our site due to domain redirects.
Our old website was a ...
- 30.7k
1
vote
1
answer
1k
views
Google-bot trips on a perfectly normal robots.txt, then on a nonexistent robots.txt [duplicate]
I have two domain names pointing to the same virtual server. One of them, http://ilarikaila.com, is a working brochure website I made for a friend. I used the other one, http://teemuleisti.com, to ...
- 123
1
vote
1
answer
5k
views
Why is my robots.txt not working?
I have this robots.txt:
User-Agent: *
Disallow: /files/
User-Agent: ia_archiver
Allow: /
User-agent: Googlebot
Disallow:
User-agent: googlebot-image
Disallow:
User-agent: googlebot-mobile
...
- 723
3
votes
1
answer
319
views
Why is googlebot requesting robots.txt from my SSH server?
I run ossec on my server and periodically I receive a warning like this:
Received From: myserver->/var/log/auth.log
Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
- 80.4k