Questions tagged [robots.txt]

Convention to prevent webcrawlers from indexing your website.

Filter by
Sorted by
Tagged with
0 votes
0 answers
38 views

nginx configuration for robots.txt

I've read other answers and Nginx docs, and I can't figure out why this works: location = /robots.txt { alias /<fullpath>/static/robots.allow.txt; } and this don't: location = /robots.txt { ...
user avatar
  • 101
0 votes
0 answers
38 views

Make webserver to prevent parsing of certain HTML elements

MediaWiki content management system creates many links which their webpages I want not to be discovered by search engine crawlers. It's not only that I don't want them indexed and more so not only ...
user avatar
0 votes
0 answers
38 views

Cannot block YandexBot with mod_rewrite

We have an Apache httpd 2.4 server as our point of entry for about 20 web sites and each site has its own virtualhost configuration. A lot of settings are probably redundant but it suits our needs. ...
user avatar
-1 votes
1 answer
77 views

Disallow: /?q=search/ in robot.txt

Does /?q=search/ means I can't web scrape the search websites that ends with =search/ ? Can I scrape an URL ends with =0#search ?
user avatar
0 votes
0 answers
225 views

With Nginx / Node.js reverse proxy how does Nginx serve robots.txt despite txt files not being referenced in Nginx config's location blocks?

In Chrome when I enter https://www.example.com/robots.txt my robots.txt file is served and works fine. I'm happy that it works but I'm not sure why it does. In the config below I thought that my last ...
user avatar
1 vote
0 answers
23 views

What are they trying to get with "GET /public-projects"

I haven't even shared my website with anyone yet and I have already started seeing attempts to GET /public-projects. However, I couldn't get any information about it, what are they trying to get? The ...
user avatar
  • 111
-1 votes
1 answer
67 views

How to write disallow paths for comments when their urls keep on changing

I want to disallow pages that develop after someone comments on my post. For example, recently I got a comment (number 12) on my post page and a new page was formed with URL: https://example.com/post/#...
user avatar
1 vote
1 answer
214 views

Tons of Access from Google Proxy

I freaquently have a lots of access from google proxy. It says it is Google Favicon bot and I've checked it by host command. User-agent is like following. "Mozilla/5.0 (X11; Linux x86_64) ...
user avatar
  • 11
1 vote
0 answers
38 views

How to block bad url path that is not part of my site from showing in google search?

I have got a site that is running on Node.js (Express) , and Apache httpd. Hundreds of requests are coming in from malicious IP's, which I'm proactively blocking. (I have a script that looks at the ...
user avatar
  • 123
3 votes
3 answers
687 views

robots.txt is redirecting to default page

Hullo, Typically, if I type into my address bar, "oneofmysites.com/robots.txt", any browser will display the content of robots.txt. As you can see, this is pretty standard behaviour. I have just ...
user avatar
  • 145
1 vote
1 answer
782 views

robots.txt route requires a backslash when behind an Application Load Balancer

I have a rails site using an AWS ALB and all routes appear to work except one, robots.txt. I am getting the error "ERR_TOO_MANY_REDIRECTS", link to example: https://www.mamapedia.com/robots.txt ...
user avatar
0 votes
1 answer
189 views

Wordpress - Yoast SEO - robots.txt

I was just reading up on Yoast website here on how the correct robots.txt file should look like with the latest SEO practices. In their example Yoast uses the following: User-Agent: * Disallow: /...
user avatar
1 vote
1 answer
371 views

Redirect "robots.txt" on specific domain

I want to redirect all requests on "robots.txt" if the domain contains ".our-internal-devel-domain.de". It should be server-wide, because when we develop a website and publish it over our test-domain, ...
user avatar
1 vote
2 answers
3k views

Rewrite robots.txt based on host with htaccess

I'm trying to rewrite a filename based on the server's domain. This code below is wrong / not working, but illustrates the desired effect. <If "req('Host') != '*.mydevserver.com'"> ...
user avatar
  • 149
-1 votes
2 answers
1k views

How to disallow crawling for all subdomains using my main domain's physical robots.txt file [closed]

I have multiple physical sub-domains and I don't want to change any robots.txt file of any of that sub-domains. Is there any way to disallow all the sub-domains from my main domain's physical robots....
user avatar
1 vote
1 answer
286 views

High no of hits by facebook crawler on server

There are daily about 3000 404 hits or more from facebook crawler. Log is as X.X.X.X Y.Y.Y.Y - - [24/May/2017:03:43:35 +0000] "GET /health-and-medicine/trumps-2018-budget-cuts-funding-for-cancer-...
user avatar
2 votes
1 answer
419 views

What's with random-character queries coming from googlebot, e.g., vvytnoxvontwusz.html?

One of my sites has been getting queries from googlebot, on the order of: example-log:66.249.79.216 - - [06/Apr/2016:15:36:56 -0700] "GET /vvytnoxvontwusz.html HTTP/1.1" 404 15136 "-" "Mozilla/5.0 (...
user avatar
-1 votes
1 answer
323 views

Robots.txt - disallow crawling one directory on subdomain

I have placed my product showcase on subdomain such as http://demo.domain.com/productname/. Demo version of product is located at http://demo.domain.com/productname/demo/. I would like to disallow ...
user avatar
2 votes
1 answer
1k views

Remove ?=collcc from url

Google Webmasters Tools has notified me about too many duplicated URLs. Some parameters have been added that I don't know about and I need to remove it, for example: http://example.com/5454/my-utr....
user avatar
3 votes
1 answer
823 views

Baidu Spider causing 3Gb of traffic a day - but I do business in China

I'm in a difficult situation, the Baidu spider is hitting my site causing about 3Gb a day worth of bandwidth. At the same time I do business in China so don't want to just block it. Has anyone else ...
user avatar
  • 327
1 vote
1 answer
823 views

Custom robots.txt being overwritten in Azure IIS 8 by something

We have a custom robots.txt in the root of our IIS cloud service Azure website that does not display correctly when navigating to www.oursite.com/robots.txt . A “different” robots.txt file displays ...
user avatar
  • 11
0 votes
2 answers
1k views

How to serve robots.txt for all my own subdomains but not other hosts on Apache?

We develop websites and we host the QA environment on the same server as the production environment. I want to serve a specific robots.txt for all QA sites but not for the production sites. We have a ...
user avatar
-3 votes
2 answers
85 views

Restricting Access from BOTS

I would like to protect my server from too many hits from Bots. Considering a scenario, where in a server (physical) located in a private network and hitting my server continuously. Do i have a ...
user avatar
-1 votes
1 answer
1k views

Block all bots from .ru domains via robots.txt or htaccess? [closed]

Is there any way to write rules for robots.txt or htaccess that will block all bots that come from a .ru domain? Thanks
user avatar
  • 1
0 votes
1 answer
939 views

Robots.txt with several VirtualHosts

My web server is running (apache 2.4.10) different virtual hosts for the following domain name : foo.example.com bar.example.com www.example.com example.com Here is the configuration file for my ...
user avatar
5 votes
1 answer
8k views

Nginx robots.txt configuration

I can't seem to properly configure nginx to return robots.txt content. Ideally, I don't need the file and just want to serve text content configured directly in nginx. Here's my config: server { ...
user avatar
  • 225
1 vote
0 answers
1k views

How to block fake google spider and fake web browser access?

Recently I found that someguys are trying to mirror my website. They are doing this in two ways: Pretend to be google spiders . Access logs are as following: 89.85.93.235 - - [05/May/2015:20:23:16 +...
user avatar
  • 151
0 votes
1 answer
501 views

Robots file behaviour

I've noticed something odd today. if i go to http://www.google.com/robots.txt, IE11 shows me the contents of google's robot file. However if i go to my site (still in development) using the same ...
user avatar
  • 167
0 votes
2 answers
1k views

how to write RewriteCond for a url with subdomain

I am trying to ban some bots by writing a RewriteCond rule in htaccess file. Is the following ruleset correct if I add the following: ## block traffic from particular referrers RewriteEngine On ...
user avatar
  • 535
0 votes
0 answers
50 views

Allocating percent resources for all robots

As many who posted here, my apache server is getting brutally hammerred to near death by robots, most of which are good robots. There is no way to change their crawl rate. Anyone have a suggestion ...
user avatar
  • 101
-4 votes
1 answer
189 views

How Can I Encourage Google to scan New robots.txt File?

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update. my last robots.txt had a "disallow: all" raw. Is there any way I ...
user avatar
1 vote
1 answer
1k views

apache robots.txt with SSL

I have an .htaccess file with a rewrite rule to get a redirect of every HTTP request to HTTPS. But now I have a problem that my robots.txt is not recognized by some online checker. If I remove the ...
user avatar
1 vote
1 answer
948 views

Google-bot trips on a perfectly normal robots.txt, then on a nonexistent robots.txt [duplicate]

I have two domain names pointing to the same virtual server. One of them, http://ilarikaila.com, is a working brochure website I made for a friend. I used the other one, http://teemuleisti.com, to ...
user avatar
1 vote
1 answer
5k views

Why is my robots.txt not working?

I have this robots.txt: User-Agent: * Disallow: /files/ User-Agent: ia_archiver Allow: / User-agent: Googlebot Disallow: User-agent: googlebot-image Disallow: User-agent: googlebot-mobile ...
user avatar
  • 167
1 vote
2 answers
1k views

Thousands of robots.txt 404 errors from bots trying to crawl old multisite

Current situation is that we are getting thousands and thousands of 404 errors from bots looking for robots.txt in different places on our site due to domain redirects. Our old website was a ...
user avatar
3 votes
1 answer
293 views

Why is googlebot requesting robots.txt from my SSH server?

I run ossec on my server and periodically I receive a warning like this: Received From: myserver->/var/log/auth.log Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
user avatar
  • 766
23 votes
5 answers
61k views

How to set robots.txt globally in nginx for all virtual hosts

I am trying to set robots.txt for all virtual hosts under nginx http server. I was able to do it in Apache by putting the following in main httpd.conf: <Location "/robots.txt"> SetHandler ...
user avatar
  • 637
1 vote
1 answer
307 views

Googlebot cant access my site webmaster tools reply Unreachable robots.txt

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server: tcpdump | grep google It returns that ...
user avatar
2 votes
1 answer
1k views

Google-Bot fell in love with my 404-page

Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot....
user avatar
2 votes
1 answer
241 views

Ideal robots.txt for a gitweb installation? [closed]

I host a few git repositories at git.nomeata.de using gitweb (and gitolite). Occasionally, a search engine spider comes along and begins to hammer the interface. While I generally do want my git ...
user avatar
1 vote
3 answers
3k views

Dynamic robots.txt based on hostname

Is there a way to swap out a robots.txt file in nginx based on hostname? I currently have www.domain.com and backup.domain.com pointing at the same nginx server, but I don't want Google indexing ...
user avatar
  • 1,356
0 votes
1 answer
217 views

Is there a chance to block images spiders / bots on dedicated servers without using robots.txt or .htaccess?

We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf. But that requires to edit may be a large ...
user avatar
  • 171
1 vote
2 answers
3k views

Blocking bad bots

I found this script and was wondering if this is just overkill and even worth using? Is it better for me to just use mod_security? # Generated using http://solidshellsecurity.com services # Begin ...
user avatar
0 votes
1 answer
774 views

robots.txt for subdomain iis7

I have two different sites in iis7 both point to the same folder they have different subdomains www.sitename.com foo.sitename.com they are essentially the same website, but it runs different logic ...
user avatar
  • 207
0 votes
1 answer
142 views

If I redirect all users (Except me) from within .htaccess, do I need a robots.txt file

So.. I have my live version of my site e.g v1.0 at domain.com I then have my development testing version at testing.domain.com I want testing.domain.com to only be accessible to me for testing, and ...
user avatar
2 votes
3 answers
390 views

Block Offline Browsers

Is there a way to block offline browsers (like Teleport Pro, Webzip, etc...) that are showed in the logs as "Mozilla"? Example: Webzip is showed in my site logs as "Mozilla/4.0 (compatible; MSIE 8.0; ...
user avatar
  • 21
2 votes
2 answers
1k views

IIS Spikes in anonymous users - crippling my server

I have a server running windows server 2008 R2, recently my websites have becoming unresponsive at least once a day, seemingly at random intervals. I have installed some monitoring software and ...
user avatar
  • 1,215
1 vote
0 answers
458 views

How to block urls that request for robots.txt in lighttpd?

We have a dedicated development server which runs only test PHP applications on a public network. We have setup session-based authentication for the site. The issue we have is there are lots of 404s ...
user avatar
1 vote
2 answers
2k views

How to create a global robots.txt that gets appended to each domain's own robots.txt on Apache?

I know can create ONE robots.txt file for all domains on an Apache server*, but I want to append to each domain's (if pre-existing) robots.txt. I want some general rules in place for all domains, but ...
user avatar
  • 1,717
1 vote
1 answer
381 views

How to Disallow Particular Path in robots.txt

I want to disallow /path but also wanna allow /path/another-path in robots.txt. I already tried: Disallow: /path Or: Disallow: /path$ But doesn't work, I mean it blocked /path/another-path too. ...
user avatar