We're using haproxy on heroku to route between all our various micro-services (also on heroku) so we only need one url in the configs.

I'm trying to use this trick:


to serve up a robots.txt to disallow everything from trawling our API (all auth protected; this is for performance rather than security reasons).

This seems like the most lightweight way of doing this, and it works beautifully locally (launched via "heroku local"). However, going to /robots.txt on our actual heroku environment gives a 503 Server Response Interrupted.

haproxy.cfg snippet:

frontend http-in
 bind *:"${PORT}"
 monitor-uri /robots.txt
 errorfile 200 "${STATIC_PATH}/robots.http"
 errorfile 503 "${STATIC_PATH}/robots.http"

Where STATIC_PATH is defined in heroku as "/app/static".


HTTP/1.0 200 Found
Cache-Control: no-cache
Connection: close
Content-Type: text/plain

User-Agent: *
Disallow: /

I know haproxy is finding robots.http, because if I set the env var to an invalid path, haproxy refuses to start. From the documentation, it should then hold this response in memory and serve it when monitor-uri is matched.

If remove the "errorfile 200" line, haproxy serves back its default server OK page, so monitor-uri is correctly matching, and 200 is the response it's attempting to return.

The "errorfile 503" entry is there because of this: http://comments.gmane.org/gmane.comp.web.haproxy/19803 but it made no difference.

Any other suggestions? Anybody else tried this before?

Thanks, B

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.