0

Context

I have a Rails app with a nginx server and Passenger.

The app is dynamically generating pages based from the request url: if the url exists in the database the app renders the corresponding page. Or if the url does not exist in the database the app renders a 404 page.

Problem

Many crawlers are trying to find vulnerabilities and request lots of urls (.git, admin/config.php, wp-login.php etc...)

Each of those requests are reaching the Rails app, which is generating hits in the database.

Solution

I am looking for a way to do this:

  1. first time a "non existent" url if requested it goes through the Rails app, which responds with a 404
  2. nginx caches and remember this url
  3. next time the same url is requested, nginx directly respond with 404 status without going through the Rails app

Also when the Rails app is restarted (through Passenger) this cache should be purged.

Tries

  • I tried to add fastcgi_cache_valid 404 10m; in the server block, it's not working.
  • Also tried proxy_cache_valid 404 10m;

As you may guess I'm new to nginx. Thanks for your help.

Nginx config

server {
  listen ...;

  server_name ...;

  root /path/to/rails/app;

  error_page 404 /404;
  error_page 500 502 503 504 /500;

  # First I tried this, no success so I removed it
  fastcgi_cache_valid 404 10m;

  # Then I tried this, no success so I removed it also
  proxy_cache_valid 404 10m;

  location / {
    gzip_static on;
    etag off;
    charset utf-8;
    add_header Cache-Control "max-age=0, private, must-revalidate";
    add_header Referrer-Policy strict-origin-when-cross-origin;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options deny;
    add_header X-XSS-Protection "1; mode=block";

    location = / {
      try_files /cached/index.html @rails;
    }

    location / {
      try_files /cached$uri.html /cached$uri $uri @rails;
    }
  }

  location @rails {
    passenger_enabled on;
    passenger_ruby /path/to/ruby;
    passenger_app_env production;
  }
}
  • Please show your nginx caching configuration. – Michael Hampton Apr 13 at 14:53
  • @MichaelHampton added – Benj Apr 13 at 17:34
  • You do not appear to have configured nginx to do any caching. Rather it appears to be serving static files generated by your application. – Michael Hampton Apr 13 at 17:57
  • That's correct. The app is generating the pages once and generate static html, then nxing serves the static html file. Would you point me to the right direction on how to configure caching? – Benj Apr 14 at 3:38
  • Is there a reason you aren't using Nginx as a caching reverse proxy to your Rails server? If such a solution is acceptable to you, I can write one up. – Brian Bauman Apr 16 at 22:14
0
+50

I'm most familiar with caching in a reverse proxy environment, so that is the approach I'd take. Thankfully, Nginx is able to proxy for itself fairly easily:

# define your cache
proxy_cache_path /path/to/cache levels=1:2 keys_zone=cacheName:[metaDataSize] max_size=[maxCacheSize] inactive=60m use_temp_path=off;

http {
  server {
    # Any TLS, caching, or gzipping on this virtual server
    listen ...;
    server_name [Actual Domain];

    location / {
      proxy_pass http://127.0.0.1:80;
      proxy_set_header Host [domain.passenger];

      # Activate and configure caching here
      proxy_cache cacheName;
      proxy_cache_valid 404 10m;
      ...any other proxy settings you want.

      # Forward original request info
      proxy_set_header X-Original-Host $http_host;
      proxy_set_header X-Original-Scheme $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;

      # Gzip if you want
      gzip on;
      gzip_proxied any;

      ...etc
    }
  }

  server {
    # Any Rails/Passenger configuration on this virtual server
    listen 80;
    server_name [domain.passenger]; 

    # Don't log requests twice
    access_log off; 

    # Only allow local requests
    allow 127.0.0.1;
    deny all;

    location / {
      passenger_enabled on;
      passenger_ruby /path/to/ruby;
      passenger_app_env production;
    }
  }
}

Purging the cache just requires running rm -rf /path/to/cache/*, so you could script that into your Rails restart procedures in whatever way pleases you best.

  • 1
    The 404 caching and proxying to the Rails app is working well.However clearing the cache is not as easy as it sound. The best way I found to do it without reloading nginx is to run find /path/to/cache/ -type f -delete. But now I have user rights access problem I need to solve. – Benj May 2 at 6:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.