Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to generate a robots.txt file using Varnish 2.1.

That means that domain.com/robots.txt is served using Varnish and also subdomain.domain.com/robots.txt is also served using Varnish.

The robots.txt must be hardcoded into default.vcl file. is that possible?

I know Varnish can generate a maintenance page on error. I'm trying to make it generate a robots.txt file. Can anyone help?

sub vcl_error {
     set obj.http.Content-Type = "text/html; charset=utf-8";
     synthetic {"
 <?xml version="1.0" encoding="utf-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
   <head>
     <title>Maintenance in progress</title>
   </head>
   <body>
     <h1>Maintenance in progress</h1>
   </body>
 </html>
 "};
     return (deliver);
 }
share|improve this question

2 Answers 2

Firstly, I can't think of any valid reason to do this - it should definitely be done the 'traditional' way - a file served by your webserver (and if needed, cached by Varnish). Hardcoding a 'file' in a config goes against most practises of trying to keep data and configurations separate. That said, this will output a robots.txt file on Varnish 3.0.2 (hopefully it works on Varnish 2.1 as well), without making a request to a backend server:

sub vcl_recv {
    #...
        if (req.url ~ "^/robots.txt$") {
            error 702 "OK";
        }
    #...
}

sub vcl_error {
    if (obj.status == 702) {
        set obj.status = 200;
        set obj.http.Content-Type = "text/plain; charset=utf-8";
        synthetic {"# robots.txt
User-agent: *
Disallow: /path/"};
    }
    return (deliver);
}

To verify the output (I did this in a virtual machine setup, without a robots.txt file - the result was a 404 error before using the above config):

curl --head http://www.example.com/robots.txt

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 46
Accept-Ranges: bytes
Date: Tue, 20 Mar 2012 23:28:58 GMT
X-Varnish: 386675282
Age: 0
Via: 1.1 varnish
Connection: close

curl http://www.example.com/robots.txt

# robots.txt
User-agent: *
Disallow: /path/

I am quite curious as to the use case for this. Do let us know why you are taking this approach, as there might be a better solution to the problem.

share|improve this answer

I haven't used Varnish but this really doesn't sound like something that it would be able to do. Varnish is a caching proxy server, not a web crawler. How would it "generate" the robots.txt file?

If your subdomain is pointing at the same IP address as the domain, Varnish should serve it already. You can send different (sub)domains to different backends using varnish but if you haven't specifically configured it to do that then all requests will go to the same backend. You may need to make sure your backend will serve different domains correctly.

Putting the content of files to be served into Varnish configuration files doesn't make sense. You need an HTTP server to serve the file (which exists on the filesystem) and then Varnish can cache that response to avoid hitting the backend for later requests. Varnish is most useful in front of dynamic, generated files and not so useful when caching static files which are already cached by the filesystem anyway.

What problem are you trying to solve here?


Trawling through the docs I discovered that it may be possible to serve a response directly from the configuration using sub vcl_recv and synthetic but cyberx86 beat me to writing up an actual answer.

The purpose of Varnish being able to send responses that it didn't get from an upstream server is that sometimes you get no response at all from the upstream server. Varnish must have its own 502 and/or 504 responses.

It still doesn't make much sense for a robots.txt file, even if it is technically possible.

share|improve this answer
    
I know varnish can generate a maintenance page on error. I'm trying to make it generate robots.txt file. Can anyone help? sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {" <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">; <html> <head> <title>Maintenance in progress</title> </head> <body> <h1>Maintenance in progress</h1> </body> </html> "}; return (deliver); } –  Sam Mar 20 '12 at 22:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.