1

I have following configuration used in server block and as the number of locations grew, the performance has degraded substantially even if there is only one request (e.g. https://example.com/loc1/MyContext/index.html). Direct access (without proxy) is very quick. By degraded I mean it takes minutes to render the page via proxy vs. seconds for direct access.

The purpose of multiple redirects for same location is due to the following original requirement.

https://example.com/loc1 -> should go to https://example.com/loc1/MyContext/index.html
https://example.com/loc1/ -> should go to https://example.com/loc1/MyContext/index.html
https://example.com/loc1/myadmin -> should go to https://example.com/loc1/MyContext/index.html
https://example.com/loc1/admin -> should go to https://example.com/loc1/MyContext/admin.html
https://example.com/loc1/reporter -> should go to https://example.com/loc1/reporter/index.html

User can always type in entire URL (e.g. via bookmark). The keepalive_timeout is set to 0 because the proxied server doesn't support keepalive. The common headers are included from a common file included via include directive (but removed from this question for simpicity).

For each proxied server there are 6 location blocks to support above mentioned url scheme. The default config file size is now more than 250KB.

Questions/issues: 1. Why has the performance degraded to such levels that even first location in config file is taking few minutes (and even timing out at times). The proxied server is very reponsive for direct access. 2. How can the location config refactored to improve the response time

The main config file (nginx.conf) is as below

user  nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;
worker_rlimit_nofile 50000;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    underscores_in_headers on;
    proxy_buffering off;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    error_log   /var/log/nginx/error.log  debug;

    sendfile        on;
    #tcp_nopush     on;
    server_tokens   off;
    keepalive_timeout  0;
    #gzip  on;

    include /etc/nginx/conf.d/default.conf;
}

The default.conf is as below:

server {
  listen 80;
  return 301 https://$host$request_uri;
}
server {
    listen 443 ssl;
    server_name example.com;
    ssl on;
    ssl_certificate       /etc/nginx/security/server.pem;
    ssl_certificate_key   /etc/nginx/security/key.pem;
    ssl_session_cache shared:SSL:5m;
    ssl_protocols TLSv1.2;
    server_name_in_redirect on;
    client_max_body_size 100m;

    location = /loc1 {
        try_files $uri $uri/ =301 /loc1/myadmin;
    }
    location = /loc1/myadmin {
        return 301 https://example.com/loc1/MyContext/index.html;
    }
    location = /loc1/admin {
        return 301 https://example.com/loc1/MyContext/admin.html;
    }
    location = /loc1/reporter {
        return 301 https://example.com/loc1/reporter/index.html;
    }

    location = /loc1/ {
        return 301 https://example.com/loc1/MyContext/index.html;
    }

    location /loc1/ {
        proxy_pass          https://Sevice_1_IP_Address:443/;
        proxy_set_header        Host <Service_1_IP_Address>;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_set_header        Upgrade $http_upgrade;
        proxy_set_header        Connection "Upgrade";
        proxy_buffering         off;
        proxy_http_version      1.1;
        proxy_set_header        Origin '';
    }   

    location = /loc2 {
        try_files $uri $uri/ =301 /loc2/myadmin;
    }
    location = /loc2/myadmin {
        return 301 https://example.com/loc2/MyContext/index.html;
    }
    location = /loc2/admin {
        return 301 https://example.com/loc2/MyContext/admin.html;
    }
    location = /loc2/reporter {
        return 301 https://example.com/loc2/reporter/index.html;
    }

    location = /loc2/ {
        return 301 https://example.com/loc2/MyContext/index.html;
    }

    location /loc2/ {
        proxy_pass          https://Sevice_2_IP_Address:443/;
        proxy_set_header        Host <Service_2_IP_Address>;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_set_header        Upgrade $http_upgrade;
        proxy_set_header        Connection "Upgrade";
        proxy_buffering         off;
        proxy_http_version      1.1;
        proxy_set_header        Origin '';
    }

}

I have searched through SO and this forum but didn't come across anything which points to my issue. I will be happy to provide any additional information.

  • I don't know what could be causing this but there are a few things you can check for leads. Are you sure the redirects are working like you intend? You could use cURL like so: curl -IL https://exmaple.com/loc1/admin to check. Also, does the request get to the backend server immediately or hang up in the proxy? Watch the access logs of both the proxy and the backend server in real-time without file buffer or check timestamps. And finally: is the backend server getting the request you're expecting it to. Use access logging on the backend server and make sure it matches your tests. – mjb2kmn Oct 23 '18 at 17:14
  • I'm not clear what your question is. You mention two things - Nginx got slower when the config file size increased, and access via the proxy is slow. You will probably get more relevant help if you can narrow down the problem yourself, and make sure your question / problem is clearly stated. For example, on your test server only put the Nginx config in required to test this specific problem, and if it's ok then Nginx config size may be the issue. If you have two problems you can ask two questions. – Tim Oct 23 '18 at 19:28
  • The backend servers serving the actual requests are responding immediately. I can access them directly without proxy and they are quick to respond. If the urls are accessed via proxy, then the responses are coming at an alarmingly slow rate. If the config size is small, then the responses via proxy is quick - this lead me to the conclusion that with bigger config file, the proxy is very slow to react. I would appreciate if you can provide any pointers to debug the issue. I thought the current location configuration might be inefficient and hence the second question about possible refactoring. – Nitb Oct 24 '18 at 7:21
  • If that is all of your config file I can tell you that the file is not big. So the file size is not the problem. – Christopher Perrin Oct 24 '18 at 8:59
  • @ChristopherPerrin the config is trimmed to remove the other 248 x 6 location blocks for similar urls for different proxied servers (e.g. loc1, loc2 ... loc 250). The file size is 250kb. – Nitb Oct 24 '18 at 9:01
0

Apparently, the issue got resolved after the k8s pod got restarted. Prior to that I tried to check all the metrics related to CPU, Memory, Disk etc. and there was nothing to worry about.

So I don't know what caused the fix so I'll consider it as one off incidence.

I would still like to pursue refactoring of configuration in a separate question. Thanks all for spending your time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.