Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
35

You can set the contents of the robots.txt file directly in the nginx config: location = /robots.txt { return 200 "User-agent: *\nDisallow: /\n"; } It is also possible to add the correct Content-Type: location = /robots.txt { add_header Content-Type text/plain; return 200 "User-agent: *\nDisallow: /\n"; }


25

In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file. Go to Health -> Fetch as Google [1] and have it fetch /robots.txt That will re-download the file and google will also re-parse the file. [1] in the previous Google UI it was 'Diagnostics -> Fetch as GoogleBot'.


10

Are there other rules that are defined? Maybe common.conf or another conf file in included which is over-riding your config. One of the following should definitely work. location /robots.txt { alias /home/www/html/robots.txt; } location /robots.txt { root /home/www/html/; } Nginx runs all "regexp" locations in order of their appearance. If any "regexp" ...


9

The robots.txt file needs to go in the top level directory of you webserver. If your main domain and each subdomain are on different vhosts then you can put it in the top level directory of each subdomain and include something like User-agent: * Disallow: / Where the robots.txt is located depends upon how you access a particular site. Given a URL like ...


7

The robots.txt file is only parsed by the robot. The webserver does not use it to decide if a client with that user-agent is allowed to access your website or not. A malicious bot will most likely ignore this file. If you want to improve security, mod_security is a much better way to do it.


5

From Apache's point of view, robots.txt is just an asset to be served. You can alter the content returned when robots.txt is requested by passing it through an output filter. If you want to append some text, you could define an external filter. Assuming that Apache is running on Unix-like operating system, the filter configuration could be ...


5

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.


5

If your authentication is working there is no reason to hide the url. Id rather focus on that.


5

It depends on the server configuration, .txt files may not be allowed. It is possible that there is a rule somewhere in the config or some .htaccess that specifies if a url doesn't match a certain pattern (say .html, .php, .htm, etc) it then redirects the rest to the index page of the web root.


4

It is not possible for search engines to find it, if it is not linked anywhere (as this is the only way search engines find other information). In order to make sure a search engine does not find it via a robots.txt and have the secrecy, use a double-hash: http://example.com/asdfghjk/12345678 Your robots.txt would disallow anything below asdfghjk: User-...


4

I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following: first correct your robots.txt to allow the correct pages, then upload a sitemap.xml with your pages as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.


4

location cannot be used inside http block. nginx does not have global aliases (i.e., aliases that can be defined for all vhosts). Save your global definations in a folder and include those. server { listen 80; root /var/www/html; include /etc/nginx/global.d/*.conf; }


4

The robots.txt must be in the root; if you have placed your Demo Version Of product in subdomain's folder then you need to change subdomains robots.txt file which should be aka; http://demo.domain.com/robots.txt robots.txt: user-agent: * disallow: /productname/demo allow: /


4

Looks to me like a Google website property ownership verification attempt, like in Webmaster Tools/Search Console: https://support.google.com/webmasters/answer/35179?hl=en Perhaps someone has inadvertently added your domain name to their account, now Google tries to verify it. No harm I can see. EDIT: You are right, the verification filenames often start ...


3

robots.txt doesn't block anything, it is up to the crawler whether it pays attention to robots.txt or ignores it. There's also no central list of web crawlers, since anyone can run one for any reason and they can appear as ordinary browsing traffic, claiming to come from an ordinary web browser. You can do basic referrer checks to block image hotlinking, ...


3

If you are using trac out of the box then these pages have both NOINDEX and NOFOLLOW on them so much as they will get crawled they won't be indexed.


3

I'm not sure why enforcing compliance with robots.txt would be the job of a proxy: The crawler (robot) is supposed to pull robots.txt and follow the instructions contained in that file, so as long as the proxy returns the correct robots.txt data and the crawler Does The Right Thing with that data, and as long as the crawler supports using a proxy, you'll get ...


3

Have you searched google for <Your IP>:22? I'm sure you haven't published it anywhere as you say, but any old idiot can put up a link to anywhere that googlebot can notice. Have you had this IP block for a while? It seems much less likely, though not impossible, that Google is starting to do something about the 'dark web' that they've talked about ...


3

When doing the sort of redirects you are doing, there is only one HTTP response code which is applicable, namely 301 Moved Permanently. RFC 2616, the standard that defines the HTTP protocol, defines the 301 response code thusly (my emphasis): The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use ...


3

As long as it follows robots.txt you can throttle the requests User-agent: * Crawl-delay: 10


2

No this is not a specified way. Either have the robots.txt or the HTML-Meta-Tag as described in http://en.wikipedia.org/wiki/Meta_element#The_robots_attribute There are only these two options available.


2

You have to put it in your root directory, otherwise it won't be found.


2

Firstly, I can't think of any valid reason to do this - it should definitely be done the 'traditional' way - a file served by your webserver (and if needed, cached by Varnish). Hardcoding a 'file' in a config goes against most practises of trying to keep data and configurations separate. That said, this will output a robots.txt file on Varnish 3.0.2 (...


2

Short Answer: No Long Answer:... Most "Offline Browsers"/Scrapers just download the raw HTML/JS/CSS to be processed by the browser later. These, if their User-Agent Strings look like Legit "Online Browsers" that's all you have to go by and thus can't block them. If they were to execute javascript on their scrape (usefull for sites that use javascript to ...


2

Two options: An if statement based on $http_host that rewrites to one of two text files on the backend. Have robots.txt be rewritten to a dynamic script (robots.php etc.) that responds accordingly.


2

You need to put robots.txt in you root directory The Disallow rules are not domian/sub-domain specific and will apply to all urls For example: Lets assume you are using a sub.mydomain.com and mydomain.com (both are linked to the same ftp folder). For this setup, if you set a Disallow: /admin/ rule then all URL sub.mydomain.com/admin/ and in mydomain....


2

I guess this makes a good community wiki. Please extend this robots.txt if you think it can be improved: User-agent: * Disallow: /*a=search* Disallow: /*/search/* Disallow: /*a=blobdiff* Disallow: /*/blobdiff/* Disallow: /*a=commitdiff* Disallow: /*/commitdiff/* Disallow: /*a=snapshot* Disallow: /*/snapshot/* Disallow: /*a=blame* Disallow: /*/blame/*


2

You deleted the robots.txt file, of course it's "unreachable" - the 404 error that Google's getting in response to the request for it is because the file doesn't exist. If the tool requires a robots.txt to exist, then you probably need one.


2

Looks ok to me, why don't you just try it and see ? If they are all going to be the same you may want to consider using links to a single document so you only have to make changes in one place. Also bear in mind that not all robots will honour a robots.txt file, you may end up having to block them using other means.


2

Yes you can with htaccess and some regex. A quick bit of googling shows up many examples, you could have done that yourself before asking here... http://httpd.apache.org/docs/1.3/howto/auth.html#allowdeny For example: Order deny,allow Deny from .ru Allow from all However this is reliant on a genuine referrer and therefore can be bypassed. For a more ...


Only top voted, non community-wiki answers of a minimum length are eligible