1

To start out, the nginx config:

server {
    listen 443 ssl;
    server_name example.com;
    ssl_certificate     /example/fullchain.pem;
    ssl_certificate_key /example/privkey.pem;

    client_max_body_size 5M;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    location @example {
        add_header X-Orig-Url $uri;
        proxy_pass http://example; # the nodes
    }

    location ~ "^/assets/uploads/(.*)" {
        add_header X-Static-Asset $1 always;
        root /usr/share/nginx/html/;
        try_files /uploads/$1 @example;
    }

    location / {
        # Offline handling
        proxy_pass http://example;
        proxy_redirect off;

        # Socket.io Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

And here's the directory structure

 /usr/share/nginx/html
   uploads
     system
       site-logo.png
       ~ 100 files
     _foo_bar
       site-logo.png
       ~ 86k files

The setup is an nginx server reverse proxying a Node.js express-powered server. Since Express is fairly slow at serving static assets, the location ~ "^/assets/uploads/(.*)" { is meant to intercept requests to upload assets and serve them directly from nginx instead.

The custom headers are just there for debugging purposes. Express adds an X-Powered-By: express header to requests it handles.

Here's where the weird stuff happens. I copied the file site-logo.png from /usr/share/nginx/html/uploads/system to /usr/share/nginx/html/uploads/_foo_bar. A request to example.com/assets/uploads/system/site-logo.png shows the X-Static-Asset header and not the X-Powered-By: express header. A request to example.com/assets/uploads/_foo_bar/site-logo.png instead shows the X-Orig-Url header and the X-Powered-By: express header, but not the X-Static-Asset header.

This tells me that all of the requests are being picked up by the location block, but nginx can't find the files in _foo_bar for some reason.

The directory at /usr/share/nginx/html/uploads is an NFS share, I'm not sure how relevant that is. My hypothesis is that nginx has trouble finding the file is such a large directory, but I'm not sure why that would be or how to fix it. I'm sure splitting the files into smaller directories would work, but I really don't want to do that because these file locations are already stored in the database in many different locations.

1

Make sure the nginx user owns the folders that you are serving from.

  • 1
    Or has execute permissions on the directories and read permissions on files in directory. – Tero Kilkanen Apr 28 '17 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.