To start out, the nginx config:
server {
listen 443 ssl;
server_name example.com;
ssl_certificate /example/fullchain.pem;
ssl_certificate_key /example/privkey.pem;
client_max_body_size 5M;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
location @example {
add_header X-Orig-Url $uri;
proxy_pass http://example; # the nodes
}
location ~ "^/assets/uploads/(.*)" {
add_header X-Static-Asset $1 always;
root /usr/share/nginx/html/;
try_files /uploads/$1 @example;
}
location / {
# Offline handling
proxy_pass http://example;
proxy_redirect off;
# Socket.io Support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
And here's the directory structure
/usr/share/nginx/html
uploads
system
site-logo.png
~ 100 files
_foo_bar
site-logo.png
~ 86k files
The setup is an nginx server reverse proxying a Node.js express-powered server. Since Express is fairly slow at serving static assets, the location ~ "^/assets/uploads/(.*)" { is meant to intercept requests to upload assets and serve them directly from nginx instead.
The custom headers are just there for debugging purposes. Express adds an X-Powered-By: express header to requests it handles.
Here's where the weird stuff happens. I copied the file site-logo.png from /usr/share/nginx/html/uploads/system to /usr/share/nginx/html/uploads/_foo_bar. A request to example.com/assets/uploads/system/site-logo.png shows the X-Static-Asset header and not the X-Powered-By: express header. A request to example.com/assets/uploads/_foo_bar/site-logo.png instead shows the X-Orig-Url header and the X-Powered-By: express header, but not the X-Static-Asset header.
This tells me that all of the requests are being picked up by the location block, but nginx can't find the files in _foo_bar for some reason.
The directory at /usr/share/nginx/html/uploads is an NFS share, I'm not sure how relevant that is. My hypothesis is that nginx has trouble finding the file is such a large directory, but I'm not sure why that would be or how to fix it. I'm sure splitting the files into smaller directories would work, but I really don't want to do that because these file locations are already stored in the database in many different locations.
