1

In nginx location directive, how do I match s3 urls.

for example, the wrong url is:

http://example.com/https://s3.amazonaws.com/mybucket/logo.jpg?1404251306

from Logs, I can see nginx can that it able to service the request from this url & returns a 404.

xx.xx.xx.xx - - [15/Aug/2014:12:38:04 +0000] "GET /https://s3.amazonaws.com/mybucket/logo.jpg HTTP/1.1" 404 151 "-" "Mo

with this, I want to match this url, proxy the request to s3 & return logo.jpg. I have come up with something like this:

location ~* ^/https/(.*) {
  set $s3_host 's3.amazonaws.com';
  set $s3_bucket 'mybucket';

  proxy_set_header       Host $s3_host;
  proxy_set_header       Authorization '';
  proxy_hide_header      x-amz-id-2;
  proxy_hide_header      x-amz-request-id;
  proxy_hide_header      Set-Cookie;
  proxy_ignore_headers   "Set-Cookie";
  proxy_buffering        off;
  proxy_intercept_errors on;

  resolver               8.8.8.8 valid=300s;
  resolver_timeout       10s;

  proxy_pass http://$1;
}

Questions:

  1. What regex should I use in location directive so it matches amazon s3 urls ONLY.
  2. Currently, it handles any bucket. How do I restrict the bucket as well?

[Update]

I get the following error:

==> /var/log/nginx/error.log <==
2014/08/15 13:53:08 [error] 1579#0: *1 invalid port in upstream ":/s3.amazonaws.com/mybucket/logo.jpg", client: xx.xx.xx.xx, server: localhost, request: "GET /https://s3.amazonaws.com//mybucket/logo.jpg HTTP/1.1", host: "54.164.92.206"
  • 1
    You should fix your links instead of trying to mitigate the symptoms of the problem. – Tero Kilkanen Aug 15 '14 at 13:36
  • 1
    already done. but there are few older clients(desktop app users who haven't upgraded to latest version) who are still using the older URLs. this is done to give them a migration path. – CuriousMind Aug 15 '14 at 13:42
3

Edit: (1) Sorry, some typo here (2) I adjust regex so it matches one or more slash before string mybucket like your log above.

Well, maybe you mean something like this

location ~* ^/https://s3\.amazonaws\.com/+mybucket(.*) {
  ...
  proxy_pass http://s3.amazonaws.com/mybucket$1;
}
  • with this, I get invalid port in upstream error. – CuriousMind Aug 15 '14 at 16:38
  • @CuriousMind, check the updated answer – masegaloeh Aug 15 '14 at 22:06
  • This doesn't match the location, I get 404 – CuriousMind Aug 18 '14 at 12:02
  • 1
    I recently tested above configuration and it's working as expected. No 404 error or other nginx error messages. Probably you should reorder the location. In my case, the location direction was right below location / {. If reordering location doesn't help, please add the full version of your nginx config in the question. – masegaloeh Aug 18 '14 at 14:13
1

I hit this error and it was because I was setting the Host header wrong.

Here's the config that ended up working:

set $s3_bucket        's3-us-west-1.amazonaws.com';
...
proxy_set_header       Host $s3_bucket;

Note that the host header does not include http/https. That should be included in the proxy_pass line

      proxy_pass             http://$s3_bucket/whatever

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.